text:
Do neurons of a neural network model a linear relationship?
I'm certain that this is a very naive question, but I am just beginning to look more deeply at neural networks, having only used decision tree approaches in the past.  Also, my formal mathematics training is more than 30 years in the past, so please be kind. :)
As I'm reading François Chollet's book on Deep Learning with Python, I'm struck that it appears that we are effectively treating the weights (kernel and biases) as terms in the standard linear equation ($y=mx+b$). At page 72 of the book, the author writes
output = dot(W, input) + b
output = (output < 0 ? 0 : output)

Am I reading too much into this, or is this correct (and so fundamental I shouldn't be asking about it)?

In a neural network (NN), a neuron can act as a linear operator, but it usually acts as a non-linear one. The usual equation of a neuron $i$ in layer $l$ of an NN is
$$o_i^l = \sigma(\mathbf{x}_i^l \cdot  \mathbf{w}_i^l + b_i^l),$$
where $\sigma$ is a so-called activation function, which is usually a non-linearity, but it can also be the identity function, $\mathbf{x}_i^l$ and $\mathbf{w}_i^l$ are the vectors that respectively contain the inputs and the weights for neuron $i$ in layer $l$, and $b_i^l \in \mathbb{R}$ is a bias. Similarly, the output of a layer of a feed-forward neural network (FFNN) is computed as
$$\mathbf{o}^l = \sigma(\mathbf{X}^l \mathbf{W}^l + \mathbf{b}^l).$$
In your specific example, you set the new weight to $0$, if the output of the linear combination is less than $0$, else you use the output of the linear combination. This is the definition of the ReLU activation function, which is a non-linear function.

Yes, I understand that... And I realize I was using ReLU rather than sigmoid...  but am I wrong to see a clear relationship between how we are currently training networks and the classic function that defines a line?

Oh, and a follow-up... From my reading over the last week, I had the impression that sigmoid was out of favor in current thought.  Is that not true?  I realize it was the standard just a few years ago.

@DavidHoelzer In the equation of a line, $\sigma$ is always the identity function, while in the case of NNs, $\sigma$ is almost never the identity function. This is the main difference.

@DavidHoelzer It seems to me that ReLU (or variations) is used more often (than sigmoids), but I think you can find several discussions on the web related to this topic.

@DuttaA Yes, but I think the equations have just been transcribed incorrectly. Maybe we can ask for clarifications regarding the passage of the book where these equations are shown.

The code does not appear as written, and I really don't know how to properly mark up a dot product in the SO Markdown..  I'm paraphrasing the discussion and function representation of the "What are activation functions" in the box on page 72.  Also, I suspect Mr. Chollet would not be pleased to find that someone has posted his book publicly with DRM removed.

Almost never. The sum of linear functions is another linear function, so if neurons were only linear transformations there would be basically no point to having more than one neuron per layer. Instead, every neuron applies some kind of nonlinear function to its input. There are lots of different variations, but in the end the combination of the nonlinear activation function at each layer with the linear matrix multiplication connecting the outputs of each layer to the inputs of the next, creates something that has much more intricate behavior while still being reasonably efficient to compute.

I know that the sum of a set of linear functions is linear...  Perhaps I mislead you with my question.  I was simply asking about the connection between how kernel and bias values are applied to produce weights;  Obviously, we apply non-linear functions to the data during transformations.  I appreciate your thoughts thought.

Taking the question from comments on nbro's answer.

Am I wrong to see a clear relationship between how we are currently training networks and the classic function that defines a line?

You are right about it. This is an intuitive way to understand neural networks. You can create a neural network that only does simple linear regression, by using linear activations functions in all the layers, such as the neural network (model) output is a linear combination of the inputs. And, this seems like a great way to introduce neural networks to students.
But, one must also look at the fact that neural networks provide the flexibility to model many kinds of non-linear relationships.

A list of activation functions.

You're quite right in your interpretation, but I'll answer in two parts in order to avoid confusion with respect to activation functions.
Part 1. (TLDR: a neurons weights is the normal vector of a hyperplane that divides input space in two parts. The neuron's preactivation is proportional to the distance of the input point to the plane.) Every artificial neuron learns a linear relationship between its inputs. The most recalled equation of a line is $y=m \cdot x+b$, but that's actually a very specific form that allows us going through values of X of the line and seeing to what values of Y it corresponds. A most general form would be $0=n \cdot y + m \cdot x + b$. This tells us that the line is formed by the points (X,Y) whose values make that series of calculations be zero. We can explore different values of (X,Y) and see that most of them give non-zero values, and that they give positive values at one side of the line and negative values at the other side. Only if you land just on the line it will give you a zero. This is a very important interpretation, because it's what allows neurons to find divisions of the input space (into a positive side and a negative side). Of course it probably won't be a 2d space, so it will be a hyperplane instead of a line, but I hope you get the idea.
Part 2. However, if we only use linear transformations we couldn't learn non-linear functions. Here's where the activation function plays a very important role: it distorts the neuron's preactivation value (which is linear) in a non-linear way (what makes it a non-linear function). Activation functions have lots of bells and whistles, which are too much to write here, but you can start thinking about them as distortions applied to that distance of the input point to the neuron's hyperplane. The one you saw is called ReLU, and it basically truncates the negative values, thus focusing only on the positive side of the hyperplane (it may be interpreted as measuring how far the point has crossed a frontier).


text:
Which unsupervised learning algorithm can be used for peaks detection?
So, I have a dataset that has around 1388 unique products and I have to do unsupervised learning on them in order to find anomalies (high/low peaks).
The data below just represents one product. The ContextID is the product number, and the StepID indicates different stages in the making of the product.
    ContextID   BacksGas_Flow_sccm  StepID  Time_ms
427 7290057 1.7578125   1   09:20:15.273
428 7290057 1.7578125   1   09:20:15.513
429 7290057 1.953125    2   09:20:15.744
430 7290057 1.85546875  2   09:20:16.814
431 7290057 1.7578125   2   09:20:17.833
432 7290057 1.7578125   2   09:20:18.852
433 7290057 1.7578125   2   09:20:19.872
434 7290057 1.7578125   2   09:20:20.892
435 7290057 1.7578125   2   09:20:22.42
436 7290057 16.9921875  5   09:20:23.82
437 7290057 46.19140625 5   09:20:24.102
438 7290057 46.19140625 5   09:20:25.122
439 7290057 46.6796875  5   09:20:26.142
440 7290057 46.6796875  5   09:20:27.162
441 7290057 46.6796875  5   09:20:28.181
442 7290057 46.6796875  5   09:20:29.232
443 7290057 46.6796875  5   09:20:30.361
444 7290057 46.6796875  5   09:20:31.381
445 7290057 46.6796875  5   09:20:32.401
446 7290057 46.6796875  5   09:20:33.431
447 7290057 46.6796875  5   09:20:34.545
448 7290057 46.6796875  5   09:20:34.761
449 7290057 46.6796875  5   09:20:34.972
450 7290057 46.6796875  5   09:20:36.50
451 7290057 46.6796875  5   09:20:37.120
452 7290057 46.6796875  7   09:20:38.171
453 7290057 46.6796875  7   09:20:39.261
454 7290057 46.6796875  7   09:20:40.280
455 7290057 46.6796875  12  09:20:41.429
456 7290057 46.6796875  12  09:20:42.449
457 7290057 46.6796875  12  09:20:43.469
458 7290057 46.6796875  12  09:20:44.499
459 7290057 46.6796875  12  09:20:45.559
460 7290057 46.6796875  12  09:20:45.689
461 7290057 47.16796875 12  09:20:46.710
462 7290057 46.6796875  12  09:20:47.749
463 7290057 46.6796875  15  09:20:48.868
464 7290057 46.6796875  15  09:20:49.889
465 7290057 46.6796875  16  09:20:50.910
466 7290057 46.6796875  16  09:20:51.938
467 7290057 24.21875    19  09:20:52.999
468 7290057 38.76953125 19  09:20:54.27
469 7290057 80.46875    19  09:20:55.68
470 7290057 72.75390625 19  09:20:56.128
471 7290057 59.5703125  19  09:20:57.247
472 7290057 63.671875   19  09:20:58.278
473 7290057 70.5078125  19  09:20:59.308
474 7290057 71.875  19  09:21:00.337
475 7290057 69.82421875 19  09:21:01.358
476 7290057 69.23828125 19  09:21:02.408
477 7290057 69.23828125 19  09:21:03.548
478 7290057 72.4609375  19  09:21:04.597
479 7290057 73.4375 19  09:21:05.615
480 7290057 73.4375 19  09:21:06.647
481 7290057 73.4375 19  09:21:07.675
482 7290057 73.4375 19  09:21:08.697
483 7290057 73.4375 19  09:21:09.727
484 7290057 74.21875    19  09:21:10.796
485 7290057 75.1953125  19  09:21:11.827
486 7290057 75.1953125  19  09:21:12.846
487 7290057 75.1953125  19  09:21:13.865
488 7290057 75.1953125  19  09:21:14.886
489 7290057 75.1953125  19  09:21:15.907
490 7290057 75.9765625  19  09:21:16.936
491 7290057 75.9765625  19  09:21:17.975
492 7290057 75.9765625  19  09:21:18.997
493 7290057 75.9765625  19  09:21:20.27
494 7290057 75.9765625  19  09:21:21.55
495 7290057 75.9765625  19  09:21:22.75
496 7290057 75.9765625  19  09:21:23.95
497 7290057 76.85546875 19  09:21:24.204
498 7290057 76.85546875 19  09:21:25.225
499 7290057 76.85546875 19  09:21:25.957
500 7290057 76.85546875 19  09:21:26.984
501 7290057 75.9765625  19  09:21:27.995
502 7290057 75.9765625  19  09:21:29.2
503 7290057 76.7578125  19  09:21:30.13
504 7290057 76.7578125  19  09:21:31.33
505 7290057 76.7578125  19  09:21:32.59
506 7290057 76.7578125  19  09:21:33.142
507 7290057 76.7578125  19  09:21:34.153
508 7290057 75.87890625 19  09:21:34.986
509 7290057 75.87890625 19  09:21:35.131
510 7290057 75.87890625 19  09:21:35.272
511 7290057 75.87890625 19  09:21:35.451
512 7290057 76.7578125  19  09:21:36.524
513 7290057 76.7578125  19  09:21:37.651
514 7290057 76.7578125  19  09:21:38.695
515 7290057 76.7578125  19  09:21:39.724
516 7290057 76.7578125  19  09:21:40.760
517 7290057 76.7578125  19  09:21:41.783
518 7290057 76.7578125  19  09:21:42.802
519 7290057 76.7578125  19  09:21:43.822
520 7290057 76.7578125  19  09:21:44.862
521 7290057 76.7578125  19  09:21:45.884
522 7290057 76.7578125  19  09:21:46.912
523 7290057 76.7578125  19  09:21:47.933
524 7290057 76.7578125  19  09:21:48.952
525 7290057 76.7578125  19  09:21:49.972
526 7290057 76.7578125  19  09:21:51.72
527 7290057 77.5390625  19  09:21:52.290
528 7290057 77.5390625  19  09:21:52.92
529 7290057 77.5390625  19  09:21:53.361
530 7290057 77.5390625  19  09:21:54.435
531 7290057 76.66015625 19  09:21:55.602
532 7290057 76.66015625 19  09:21:56.621
533 7290057 72.94921875 22  09:21:57.652
534 7290057 3.90625 24  09:21:58.749
535 7290057 2.5390625   24  09:21:59.801
536 7290057 2.1484375   24  09:22:00.882
537 7290057 2.05078125  24  09:22:01.259
538 7290057 2.1484375   24  09:22:01.53
539 7290057 1.953125    24  09:22:02.281
540 7290057 1.953125    24  09:22:03.311
541 7290057 2.1484375   24  09:22:04.331
542 7290057 2.1484375   24  09:22:05.351
543 7290057 1.953125    24  09:22:06.432
544 7290057 1.85546875  24  09:22:07.519
545 7290057 1.7578125   24  09:22:08.549
546 7290057 1.85546875  24  09:22:09.710
547 7290057 1.7578125   24  09:22:10.738
548 7290057 1.85546875  24  09:22:11.798
549 7290057 1.953125    24  09:22:12.820
550 7290057 1.85546875  1   09:22:13.610
551 7290057 1.85546875  1   09:22:14.629
552 7290057 1.953125    1   09:22:15.649
553 7290057 1.85546875  2   09:22:16.679
554 7290057 1.85546875  2   09:22:17.709
555 7290057 1.85546875  2   09:22:18.729
556 7290057 1.953125    2   09:22:19.748
557 7290057 1.85546875  2   09:22:20.768
558 7290057 1.7578125   3   09:22:21.788
559 7290057 1.7578125   3   09:22:22.808
560 7290057 1.85546875  3   09:22:23.829
561 7290057 1.953125    3   09:22:24.848
562 7290057 1.85546875  3   09:22:25.898
563 7290057 1.953125    3   09:22:27.39
564 7290057 1.953125    3   09:22:28.66
565 7290057 1.7578125   3   09:22:29.87
566 7290057 1.85546875  3   09:22:30.108
567 7290057 1.7578125   3   09:22:31.129
568 7290057 1.953125    3   09:22:32.147
569 7290057 1.85546875  3   09:22:33.187

I use the following code to plot a graph.
Code:
lineplot = X.loc[X['ContextID'] == 7290057]
x_axis = lineplot.values[:,3]
y_axis = lineplot.values[:,1]

plt.figure(1)
plt.plot(x_axis, y_axis)

and the graph:

In this graph, the peaks (marked in red circles) are the anomalies that need to be detected.
And when I have a graph like this: No anomalies must be caught since there are no undesirable peaks.

I tried using OneClassSVM, but I am somehow not satisfied with the results.
I would like to know which unsupervised learning algorithm can be used for such a task at hand.

If your anomalies are simply peaks, why should you be using machine learning methods? You could use peak detection algorithms for the purpose.
If you still insist on ML, isolation forest is a good try.

For completeness and for people not familiar with them, maybe you can suggest 1-2 concrete "peak detection algorithms" that could be used to solve the problem.

Sure ! One simple method is finding difference between consecutive samples and if the difference suddenly increases compared to the previous set (say 5 samples) of differences you detect a peak. The other more robust method is perform the Discrete Wavelet transform of the data, square the results and find values that are twice higher than the median.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Which unsupervised learning technique can be used for anomaly detection in a time series?
I've started working on anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor-making machines.
My dataset looks like this:
ContextID   Time_ms Ar_Flow_sccm    BacksGas_Flow_sccm
7289973 09:12:48.502    49.56054688 1.953125
7289973 09:12:48.603    49.56054688 2.05078125
7289973 09:12:48.934    99.85351563 2.05078125
7289973 09:12:49.924    351.3183594 2.05078125
7289973 09:12:50.924    382.8125    1.953125
7289973 09:12:51.924    382.8125    1.7578125
7289973 09:12:52.934    382.8125    1.7578125
7289999 09:15:36.434    50.04882813 1.7578125
7289999 09:15:36.654    50.04882813 1.7578125
7289999 09:15:36.820    50.04882813 1.66015625
7289999 09:15:37.904    333.2519531 1.85546875
7289999 09:15:38.924    377.1972656 1.953125
7289999 09:15:39.994    377.1972656 1.7578125
7289999 09:15:41.94     388.671875  1.85546875
7289999 09:15:42.136    388.671875  1.85546875
7290025 09:18:00.429    381.5917969 1.85546875
7290025 09:18:01.448    381.5917969 1.85546875
7290025 09:18:02.488    381.5917969 1.953125
7290025 09:18:03.549    381.5917969 14.453125
7290025 09:18:04.589    381.5917969 46.77734375

What I have to do is to apply some unsupervised learning technique on each and every parameter column individually and find any anomalies that might exist in there. The ContextID is more like a product number.
I would like to know which unsupervised learning techniques can be used for this kind of task at hand since the problem is a bit unique:

It has temporal values.
Since it has temporal values, each product will have many (similar or different) values as can be seen in the dataset above.

So if I understood correctly:
You have data from 2 sensors in time: Ar flow and BackGas Flow (SCCM, what is that?)
You have that data for multiple products.
1 - Since it is relatively low dimensional, you may try using raw data with K-Means or Self Organizing Maps.
2 - If you searching for anomalies in time, you might try using feature engineering with things like flow change:

i) Take 2 points in time and "derivate" (Point A - Point B)/(Time A - Time B)
ii) Take the result for A and repeat the process as a second derivative, try that for many levels of "derivative"
iii) Machine learning application for sensor failure detection in polymerization process. In: Simpósio Brasileiro de Telecomunicações e Processamento de Sinais.

3 - You might want to check other Time-Series related research, even on regression and classification, since they may give you ideas on relevant features or approaches:

i) An enseble approach to time dependent classification (10.1109/ICMLA.2018.00164)
ii) Multivariate Time Series for Data-driven Endpoint Prediction in the Blast Oxygen Furnace (DOI: 10.1109/ICMLA.2018.00231)

4 - Note that anomalies are outliers, so analysing data distant from the mean of any features you might engineer from the time. An model based on RBF kernels might pick up some information.
5 - Since you don't know the amount of clusters you can try hierarchical clustering.
6 - Don't forget to talk to people in your field about what to expect from this.
If you can elaborate on the type of anomalies you're looking for I may come with better ideas. This is pretty general tips.
I may have misunderstood what you wanted to cluster:
7 - If you want to detect anomalies in:

i) products behavior tips above were mostly for it and you might use all 23 features in short time intervals and cluster like this:

    For each product in dataset {

      samples = {} # empty set

      For each time interval in product {

        features in interval add to samples

      }

      perform clustering method using samples

    }

This algorithm would help cluster periods of time where the product behavied weirdly

ii) sensors behavior then for something like that you might try tip 8

iii) products itself create a feature vector with all avaiable information for each product, then cluster that. Since it is composed of variable time interval and 23 features per product (plus time) you might want to use a bit of dimensionality reduction (PCA should work fine, since it gives you the direction of greater variance and you're look for that). Using a regressor to model products behavior and cluster that could all be usefull (create a regressor in time for each product to model its behavior, then cluster the weights of the regressors as the product representation)

8 - Let's say you want to identify sensors behavior, it might be helpful to model the sensors output as a function f(time, extravariables) and do regression over it (try linear, then try non-linear). Points in time that the regressor had bad performance predicting the answer might indicate that that behavior is anomalous

Hey, Pedro. Thank you for your valuable tips. I shall try the things suggested by you, but I've put here only 2 parameters (ArFlow and BacksGas). In my dataset, I've about 23 different parameters. So do you think KMeans can still be applied for every single parameter separately? And sccm- is Standard Cubic Centimeter Flow

I'll add 2 more itens to the answers

if anyone can help me to make this algorithm readable I would thank hahaha

important to notice that tip 7 will find timeintervals where a product presented anomalous behavior to itself, if you want to detect it on other products that should have the same effect create a set using similar products to cluster

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Deep Q-Learning poor convergence on Stochastic Environment
I'm trying to implement a Deep Q-network in Keras/TF that learns to play Minesweeper (our stochastic environment). I have noticed that the agent learns to play the game pretty well with both small and large board sizes. However, it only converges/learns when the layout of the mines is the same for each game. That is, if I randomize the mine distribution from game to game, the agent learns nothing - or near to it. I tried using various network architectures and hyperparameters but to no avail.
I tried a lot of network architectures including:

The input to the network is the entire board matrix, with the individual cells having values of -1 if unrevealed, or 0 to 8 if revealed.
The output of the network is also the entire board representing the desirability of clicking each cell.
Tried fully connected hidden layers (both wide and deep).
Tried Convolutional hidden layers (tried stacked them, using different kernel sizes, padding, etc.).
Tried adding Dropout after hidden layers too.

Is DQN applicable for environments that change every episode or have I approached this from the wrong side?
It seems no matter the network architecture, the agent won't learn. Any input is greatly appreciated. Please let me know if you require any code or further explanations.

The inputs that you describe seem like they should be sufficient for a DQN-based agent to learn a good strategy for playing Minesweeper, regardless of whether or not the starting layout changes. The inputs contain all information that is necessary.
However, the problem certainly becomes much easier (probably too easy) if the initial problem is always the same. The DQN algorithm will be very likely to "pick up" on this trend and "exploit" it. The inputs may be sufficient to learn a more general Minesweeper strategy, but if it is consistently faced with the same "level" every single time, it will be much easier for the DQN algorithm to just memorize exactly where the mines are and play perfectly based on that memory, rather than any actual strategy. Due to the way learning is implemented (based on gradient descent), the algorithm will generally tend to converge to such an easily-reachable "memorization" strategy rather than something that is actually "smart".
For that reason, I do think training will be much slower in the case where the layout is randomized. I'm not just thinking along the lines of e.g. twice as slow here, but would expect multiple orders of magnitude more experience required for successful learning. That's just my educated guess though, never tried training DQN for minesweeper specifically so can't tell for sure. It might also require more elaborate hyperparameter tuning, and maybe require a different network architecture (probably require a larger network).

Thanks for your input! I certainly agree that the agent is memorizing the game rather than actually learning strategies. I'll try running it with a relatively large network for a day and see how it does.
On a side-note: Do you recommend CNN to exploit spatial structure in the game or just flatten out the board and go with a fully connected network?

@Sanavesa Hmm my initial guess would be that... CNNs seem like they should be applicable here? I'm really not 100% confident though, might be missing something.

After 24 hours of training (~500k games), it is fluctuating unstably around 0-5% win rate. I don't think it is slow, I think the learning is non-existent lol.

@Sanavesa Hmmm, there do seem to be some others who have used DQN for Minesweeper. For example: https://github.com/jakejhansen/minesweeper_solver . Maybe there are important differences in terms of network architecture or reward structure or anything like that, between your implementation and theirs? In theory I really can't think of any reason why it shouldn't be able to ever work, but it's not going to be an easy problem. Deep RL isn't exactly known for its stability/reliability either, if things like hyperparams aren't just perfect it can break down quite easily.

Thank you very much. I'll take a look. Off the bat, the biggest difference is the network structure, in which the author uses a multiple channel one-hot encoding of the cells instead of a single channel. That could be the reason.

@Sanavesa Aaah yes I think that may be important, I should've thought of that. Actually I did, but it was when I wasn't behind my computer, and forgot about it again later :D If you just use numerical entries, neural networks will "think" that there is some level of... "continuity" in the inputs, and try to generalize across cells with "close" input numbers. I don't think that behaviour is desirable in the case of Minesweeper, where the best strategy is going to involve more "binary" choices


text:
How should I model all available actions of a chess game in deep Q-learning?
I just read about deep Q-learning, which is using a neural network for the value function instead of a table.
I saw the example here: Using Keras and Deep Q-Network to Play FlappyBird and he used a CNN to get the Q-value.
My confusion is on the last layer of his neural net. Neurons in the output layer each represent an action (flap, or not flap). I also see the other projects where the output layer also represents all available actions (move-left, stop, etc.)
How would you represent all the available actions of a chess game? Every pawn has a unique and available movement. We also need to choose how far it will move (rook can move more than one square). I've read Giraffe chess engine's paper and can't find how he represents the output layer (I'll read once again).
I hope somebody here can give a nice explanation about how to design NN architecture in Q-learning, I'm new in reinforcement learning.

To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can be found starting from the bottom of page 13.
Briefly, an action is described by picking a piece and then picking a move with it. The size of the board is 8 by 8 so there can be 8x8 possibilities for picking a piece. Then we can either pick linear movements (in 8 directions) and then pick the number of steps in that direction (maximum 7 steps) or we can make a knight movement (maximum 8 possibilities). So far that is 8x7 + 8. Furthermore, we also need to consider underpromotions (promoting a pawn into a non-queen piece). In this scenario we can have 3 types of pawn movements (forward, left diagonal or right diagonal capture) and 3 types of promotions (rook, knight, bishop) so that makes it 9. So the total dimension of the action space is 8x8x(8x7+8+9) and this will be the number of neuron outputs you will need to use.
Note that this action space representation covers every possible scenario and for example at the start of the game the action of picking the tile E4 and promoting it to a bishop doesn't make sense (there are no pieces on tile E4 at the beginning of the game). Or if we pick a tile where there is a rook we cannot make a knight movement with it. Therefore you will also need to implement a function that can return you the set of possible actions in a given state and ignore all neural network outputs that is not contained in this set.
Obviously this action representation is not set into stone so if you can come up with something better or more compact you can use that one too. You can also make restrictions to your game for example by not allowing underpromotions.

I just read this question: https://stackoverflow.com/questions/27340967/questions-about-q-learning-using-neural-networks and he uses action as input, and only one value as output (Q(s,a)), I think it's more efficient isn't?

@malioboro: It might be more efficient in some cases. However, you do need to run this network forward once per allowed action, whilst for an "all actions" network you run it forward once, then filter to allowed actions. The extra cost for calculating extra unused moves with the larger output may not be as much as running the smaller network many times. Especially with deep networks, a one-shot-then filter could be more efficient than a filter then many-shot.

Thank you @NeilSlater and HaiNguyen it is clear to me now :)

The accepted answer severely overcounts the actual action space because of the assumption that any piece can move a maximum of 7 squares in any direction from any square on the board. The calculation 8x8x(8x7+8+9) = 4672 is even much more than the most naive estimate of 64x63 = 4032.
The actual action space to specify any possible legal move is 1924. The difference is not negligible as this number is less than half of 4672. So, if using a dense final layer, this would save about 60% of the work for that layer.
Here is the Python script using chess I wrote to confirm it:
import chess

# Each chess piece moves like either a queen or a knight from one square to another square.
# All possible moves except pawn promotions can be specified by the "from" square and the "to" square
# (even castling, ex. e1->g1, or e1->c1).
action_space = 0
b = chess.BaseBoard.empty()
for square in range(64):
    # Place queen and see where it attacks
    b.set_piece_at(square, chess.Piece.from_symbol('Q'))
    q_moves = b.attacks(square)

    # Place knight and see where it attacks
    b.set_piece_at(square, chess.Piece.from_symbol('N'))
    n_moves = b.attacks(square)

    # Logical or to combine bitmaps (ex. 1100 | 0101 = 1101)
    all_moves = q_moves | n_moves

    # Convert bitmap to list of bools, so the sum
    # is exactly the # possible moves from this square
    action_space += sum(all_moves.tolist())

    b.remove_piece_at(square)

# Count underpromotions manually:
# 8 forward promotions, 7 right-capture promotions, 7 left-capture promotions
# which can all promote to 4 pieces (but 1 is already counted above) for 2 colors
action_space += (8+7+7) * (4-1) * 2

print('actual action space:', action_space)
print('naive action space:', 64 * 63)
print('accepted answer\'s action space:', 8*8*(8*7 + 8 + 9))

# actual action space: 1924
# naive action space: 4032
# accepted answer's action space: 4672

Note: This assumes that the same output neuron will be used to represent for example the moves Ra7a8 and a8=Q, as well as the moves Re1g1 and O-O (by white). If this is not desirable, the action space that counts queen promotions and castling moves as distinct actions has size 1972. You can see this by changing (4-1) to 4 in the final step that accounts for underpromotions, then adding 4 for the 2 castling directions by both colors.

Having written the code below, I find 1968 unique moves.
This thread is old but maybe this helps someone.
import chess

# Create an empty chess benter preformatted text hereoard
board = chess.Board()

# Initialize an empty list to store all possible moves
action_space = []

# Iterate over all squares and all piece types
for square in chess.SQUARES:
    for piece_type in chess.PIECE_TYPES:
        for color in chess.COLORS:
            # Set player to color
            board.turn = color
            # Place piece on given square
            board.set_piece_at(square, chess.Piece(piece_type, color))

            # If the piece is a pawn, place other pawns around it to include attacks
            if piece_type == chess.PAWN:
                other_color = chess.WHITE if color == chess.BLACK else chess.BLACK
                for square_offset in [-7, -9, 7, 9]:
                    if square + square_offset in chess.SQUARES:
                        board.set_piece_at(square + square_offset, chess.Piece(piece_type, other_color))

            for move in board.legal_moves:
                # If the move is not already in the action space, add it
                if move not in action_space:
                    action_space.append(move)

            # Clear the board
            board.clear()
action_space = sorted(action_space, key=lambda x: x.uci())

for i in range(len(action_space) - 1):
    if action_space[i] == action_space[i+1]:
        print("Duplicate found: ", action_space[i].uci())
        exit()
print("No duplicates found")
print("Number of moves: ", len(action_space))


text:
Facial recognition and classifying unknowns with neural networks
As far as I understand, neural networks aren't good at classifying 'unknowns', i.e. objects that do not belong to a learned class. But how do face detection/recognition approaches usually determine that no face is detected/recognised in a region? Is the predicted probability somehow thresholded?
I'm asking because my application will involve identifying unknown objects. In fact, most of the input objects are unknown and only a fraction is known.

Summary
It is true that neural networks are inherently not good at classifying 'unknowns' because they tend to overfit to the data that they have been trained on, if the underlying structure of the neural network is complex enough. However, there are multiple ways to go about reducing the affects of overfitting. For example, one technique that is used for this is called dropout. Another example can be batch normalization. Despite these techniques, the best way to reduce the affects of overfitting is to use more data.
For the facial recognition example that you have given above, it is common that the models that have been trained have 'seen' a huge amount of data. This means that there are very few 'unknowns' and even if there are, the neural network has learned how to tell if there are facial features present or not. This is because certain structures of neural networks are really good at telling if there is a pattern of features present in the input data. This helps the neural networks to learn if the image that is being input has certain features/patterns in it or not. If the these features are found then the input data is classified as face otherwise it is not.
What can you do in your case?
Let us assume that you are going to train your neural network to recognize if an input image is a cat or not. You will use a Convolutional Neural Network (CNN) and train it to recognize if the input is cat or not. The not part means that you have to include a lot of examples in your training data that are not cat. In the perfect case you will be able to show it everything that is not a cat and classify it as such. Also you show it multiple images of what a cat is. CNNs are really great for this application. You might want to research regarding this and see what kind of CNN best suits your application. If you don't have gazillion samples of what a cat is not then you can use regularization techniques like dropout and batch normalization.
PS: For more details please mention what strategies you have used up till now. Also it would be better if you can share what your desired task is.

I would like to highlight an import step for face recognition which is features extraction. Based on my experience, you can evaluated robust feature extraction methods like, Scale-Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF) using several matching approaches such as Brute Force Matcher, K-Nearest Neighbor (KNN), Best-Bin-First (BBF) and RANdom SAmple Consensus (RANSAC). The purpose is to identify the method(s) that is/are most appropriate to be used in your application. Then comes your machine learning model, which you need to test several model options like mentioned in the previous message.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
CNNs: What happens from one neuron volume to the next?
I've gone through several descriptions of CNNs online and they all leave out a crucial part as if it were trivial.
A "volume" of neurons consists of several parallel layers ("feature maps"), each the result of convolving with a different kernel.
Between volumes, there is usually a step where layers are pooled and subsampled.
The next volume has a different number of parallel layers.
How do the feature maps from one volume connect to the feature maps of the next volume? Is it one-to-many? Many-to-many? Do N kernels apply to each of the M feature maps in the first volume, yielding N*M feature maps in the second volume? Are these N kernels the same for each feature map in the first volume, or do different kernels apply to each one?
Or, is the number of maps in the second volume not necessarily a multiple of the number in the first volume? If so, do maps in the first volume get cross-synthesized somehow? Or, maybe different numbers of maps in the second volume follow from each one in the first?
Or, is it some other of umpteen trillion possibilities?

Short answer: One to many
Long answer:
The point is that you use a 3D convolution in a CNN. Each kernel has the size of n*m*C (C is the number of feature maps) and every feature map has its own kernel(=weights) and bias.
An example:
The size of layer 2 is 9x9x10 (stride 1, no padding), the kernel size is 3x3x10.
The dimension of the next layer would be 3x3xn (n are the number of kernels that layer 2 has).
Here I found a very clear explanation.
I hope this helps.

How can there be no stride? It seems stride needs to be at least 1. Also, if the layer is 9x9x10 and the kernel size is 3x3x10, how can such a convolution result in multiple feature maps? Isn't the depth of the next layer 10-D+1, with kernel depth D? It seems like the setup you describe yields a depth of 1 for the next layer.

Yes you are right. I meant stride 1. I edited my answer

the depth of the  feature map n+1 is equal to the numbers of kernels you used in the feature map n. so if you just use 1 kernel the depth of the next feature map is 1. - each feature map has its own kernel - this is one of the main advantages of the CNN over the full connected NN: there are much less parameters to fit.

If you really want to understand what goes on in a CNN and visualize it, then you can give a read to this.
This course needs some prerequisites like having a certain level of idea of the forward and backward propagation and knowing how the network learns on itself.
CNN is just a way of moving from fully connected layers (every input connects to every output) to partially connected layers.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Triplet Loss- Three forward pass and one backward pass(Propagation)
I am trying to build a CNN model based on the concepts of Contrastive Learning. In specific based on Triplet loss.
I have 5 different class labels and I create triplets such that in a triplet, two images are from the same class and the third one is from another class.
I have a CNN model which takes one input from a triplet at a time and generates its corresponding embedding in 128 dimensions.
All three embedding embeddings from a triplet are used for calculating loss. The loss is based on the Triplet loss.
Further, the loss is backpropagated and training is carried out stochastically.
The idea is to use the trained model to generate one embedding for an input image which can be further used for multi-class classification problems.
My question is, is this method of 3 forward passes and 1 backward pass valid in Tensorflow?
Here is a fragment of my code that I am using for training:
def cnn():
    model_input = layers.Input(shape=(112, 112, 3))
    x = layers.Conv2D(filters=16, kernel_size=3, padding='same', name='Conv1')(model_input)
    x = layers.MaxPool2D()(x)
    x = layers.BatchNormalization()(x)
    x = layers.ReLU()(x)

    x = layers.Conv2D(filters=32, kernel_size=3, padding='same', name='Conv2')(x)
    x = layers.MaxPool2D()(x)
    x = layers.BatchNormalization()(x)
    x = layers.ReLU()(x)

    x = layers.Conv2D(filters=64, kernel_size=3, padding='same', name='Conv3')(x)
    x = layers.MaxPool2D()(x)
    x = layers.BatchNormalization()(x)
    x = layers.ReLU()(x)

    x = layers.Conv2D(filters=128, kernel_size=3, padding='same', name='Conv4')(x)
    x = layers.MaxPool2D()(x)
    x = layers.BatchNormalization()(x)
    x = layers.ReLU()(x)

    x = layers.Conv2D(filters=256, kernel_size=3, padding='same', name='Conv5')(x)
    x = layers.MaxPool2D()(x)
    x = layers.BatchNormalization()(x)
    x = layers.ReLU()(x)

    x = layers.GlobalAvgPool2D(name='GAP')(x)
    output = layers.Dense(128, activation='tanh', name='Dense1')(x)
    origin = tf.zeros_like(output, dtype=float)
    unit_vector = tf.divide(output, tf.sqrt(tf.reduce_sum(tf.square(output-origin)))) # Normalize vector(L2_Norm)
    shared_model = Model(inputs=model_input, outputs=unit_vector)
    shared_model.summary()
    return shared_model

def triplets_loss(anchor_sample, positive_sample, negative_sample, alpha=0.2):
    anchor_pos_dist = tf.sqrt(tf.reduce_sum(tf.square(anchor_sample - positive_sample))) # distance between positive pairs
    anchor_neg_dist = tf.sqrt(tf.reduce_sum(tf.square(anchor_sample - negative_sample)))# distance between negative pairs
    triplet_loss = tf.maximum(((anchor_pos_dist - anchor_neg_dist) + alpha), 0.000001) # triplet loss
    return triplet_loss

def train(train_data_dir, training_batch=4, lr=1e-4, epochs=100,margin=0.2):
    model = cnn()
    ### creating triplet data loader object ###
    train_data_util_instance = TripletFormulator(data_path_dictionary=train_data_dir,
    batch=training_batch)
    train_data_array_dict, data_count = train_data_util_instance.data_loader()
    majority_class = max(data_count, key=data_count.get)
    ######
    optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
    for epoch in range(epochs):
        total_train_loss = 0
        start_time = time.time()
        batch_no = 0
        for majority_class_batch in train_data_array_dict[majority_class]:
            batch_loss = 0
            train_batch_dict = {}
            train_batch_dict['A'] = next(iter(train_data_array_dict['A']))
            train_batch_dict['B'] = next(iter(train_data_array_dict['B']))
            train_batch_dict['C'] = majority_class_batch
            train_batch_dict['D'] = next(iter(train_data_array_dict['D']))
            train_batch_dict['E'] = next(iter(train_data_array_dict['E']))
            train_triplets = train_data_util_instance.triplet_generator(train_batch_dict)
            for triplets in train_triplets:
                with tf.GradientTape() as tape:
                     anchor = model(tf.reshape(triplets[0], [-1, 112, 112, 3]))
                     positive = model(tf.reshape(triplets[1], [-1, 112, 112, 3]))
                     negative = model(tf.reshape(triplets[2], [-1, 112, 112, 3]))
                     if np.isnan(anchor).any() or np.isnan(positive).any() or np.isnan(negative).any():
                         print('NAN FOUND')
                     else:
                         loss = triplets_loss(anchor_sample=anchor, positive_sample=positive,negative_sample=negative, alpha=self.alpha)
                     total_train_loss += loss
                     batch_loss += loss
                grads = tape.gradient(loss, model.trainable_weights)
                optimizer.apply_gradients(zip(grads, model.trainable_weights))
             print(epoch, batch_no, batch_loss)
             batch_no += 1
        end_time = time.time()
        print('Training_loss: ', total_train_loss, 'Time_taken: ', end_time - start_time)

When I start training, I can see the training loss converging. But the overall idea is confusing about the number of forward passes per backward pass. Also, is this method a concept of weight sharing?
I would be very eager to discuss this topic as I do not see many problems similar to this.
In most cases, the CNN model takes multiple input and generate multiple outputs and further, the embeddings are used for binary classification that is if the inputs are the same or not.
I would be waiting for your comments and suggestions.

In triplet loss you basically have one encoder and the loss is minimised according to
In this sense, the weights are shared between the encoding calculated on the negative, anchor and positive parts of the sample.
I'd recommend however to not hard pick negative and positive samples for every anchor, but to let your code pick a convenient triplet instead: a negative and positive maximising the distance.
Tensorflow has this implemented directly:
https://www.tensorflow.org/addons/api_docs/python/tfa/losses/TripletSemiHardLoss

I'm not exactly sure what triplet loss is and not sure if I follow your explanation.
If you have a single loss function (i.e. a single scalar number), you have one forward pass and one backward pass. It doesn't matter if there are certain layers that are used multiple times (3 times, presumably) in the forward pass, that just means that layer will also be used 3 times in the backward pass.

Hello Taw, Thanks for the reply.
Actually there is one single model which I am using three times to get three embeddings. and then I calculate loss on the basis of these embeddings.
If you look at the code, in the function ´def train´, under the section of ´with tf.GradientTape() as tape: ´, I have tried to implement this concept.
Triplet loss - https://en.wikipedia.org/wiki/Triplet_loss

Correct me if wrong. I believe that while backpropogating, in my code, unwantedly i have created 3 different models and the model associated with negative samples is only getting updated.


text:
Comparing a large/general CNN to a smaller more specialized one?
I am still somewhat a novice in the ML world, but I had a strange idea about CNNs and wanted to ask if this would be a valid way to check the robustness of a general CNN that classifies certain images.
Let's say that I make a CNN that takes in many different images of sports players performing a certain action (basketball shot, football kick, freestyle in swimming, flip in gymnastics, etc). Firstly, would it be possible for such a CNN to distinguish between such varied images and classify them accurately? And if so, can it be a good idea to compare this "larger" CNN to multiple "smaller" more specialized ones that take in images from one particular sport?
In other words, I want to know that if I have a "larger" CNN that gives me an output like "football being kicked", is there a way to then double-check that output with a smaller CNN that only focuses on football moves? In essence, could we create a  system where once you obtain an output from a general CNN, it automatically classifies the same image through a more specialized CNN, and then if the results are of similar accuracy, you know for sure that CNN works?
Kind of like having a smaller CNN as a "ground-truth" for the bigger one? In my head it kind of goes like this:
large_net_output = 'Football kick identified with 95.56% confidence'

for sport in large_net:
    if sport == 'football':
        access = small_net_for_football
        return small_net_for_football_output

    elif sport == 'swimming':
        access = small_net_for_swimming
        return small_net_for_swimming_output

    elif sport == 'baseball':
        access = small_net_for_baseball
        return small_net_for_baseball_output

# and so on....
>>> small_net_for_football_output = 'Football kick identified with 97.32% confidence'

robustness_check = large_net_output - small_net_for_football_output
print(robustness_check)

>>> 'Your system is accurate within a good range of 1.76%'

I hope this makes sense, and that this question does not cause any of your to cringe. Would appreciate any feedback on this!

Cross-posted: https://ai.stackexchange.com/q/23772/1794, https://stats.stackexchange.com/q/489205/2921.  Please do not post the same question on multiple sites.

After reading your question I can relate it to the Representation Learning papers such as SimCLR and SwAV. These models use a "Big Task agnostic CNN" to obtain smaller representations of the images and then they train another CNN for classification. I suggest you read Big Self-Supervised Models are Strong Semi-Supervised Learners by Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi and Geoffrey Hinton. The code for the following can be found here. But I feel that training such a model would take up a lot of computational resources.


text:
Should binary feature be in one or two columns in deep neural networks?
Let's assume I have a simple feedforward neural network whose input contains binary 0/1 features and output is also binary two classes.
Is it better, worse, or maybe totally indifferent, for every such binary feature to be in just one column or maybe it would be better to split one feature into two columns in a way that the second column will have the opposite value, like that:
feature_x (one column scenario)

[0]

[1]

[0]

feature_x (two columns scenario)

[0, 1]

[1, 0]

[0, 1]

I know this might seem a bit weird and probably it is not necessary, but I have a feeling like there might be a difference for a network especially for its inner workings and how neurons in the next layers see such data. Has anyone ever researched that aspect?

You're simply adding a redundant feature by having it as two: $X_2 = 1 - X_1$. It would be equally useful to duplicate the first column. At best this will not improve your model, at worst it will decrease accuracy.


text:
Are there deep neural networks that have inputs connected with deeper hidden layers?
Are there any architectures of deep neural networks that connect input neurons not only with the first hidden layer but also with deeper ones (red lines on the picture)?

If so could you give some names or links to research papers?

This seems to be a duplicate of https://ai.stackexchange.com/q/17190/2444. Well, at least, I answer your question there too.

@nbro Well, I don't think this question is a duplicate, but yes there is an answer to my question in the linked one. Thanks!

This type of connections are called skip or residual connections. There are numerous works which employs this type of mechanism, for example: ResNet, SkipRNN. In addition here you can find a paper that empirically explores the skip connections for sequential tagging, or this one for speech enhancement.


text:
What activation functions are currently popular?
I am not asking what activation function is better. I want to know what activation functions are more used in research or deployment. Also, are they used in combination? E.g., ReLU, ELUs, etc. I'd appreciate any statistics or insight on this.

Currently, both ReLU and ELUs are the most popular activation functions (AF) used in neural nets (NNs). This is because they eliminate the vanishing gradient problem that causes major problems in the training process and degrades the accuracy and performance of NN models.
Also, these AFs, more specifically ReLU, are very fast learning AF which makes them even more useful in research.
However, depending on the type of NN you working on, it's always a good practice to pay some attention to new studies.

Maybe you should provide some references that show that ReLU and ELU are the most widely used AFs. If there are some statistics, that would even be better.

@nbro I will have a look and update any answer I provide over time. I haven't found any good source to provide stat about it. Please let me know if you know any. Thanks for the suggestion and also for keeping this community well.


text:
Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?
In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix:

The reason why is stated that this constrains the distribution of the weights of the output to have a standard deviation of 1.
Quoted from Transformer model for language understanding | TensorFlow:

For example, consider that $Q$ and $K$ have a mean of 0 and variance of 1. Their matrix multiplication will have a mean of 0 and variance of $d_k$. Hence, square root of $d_k$ is used for scaling (and not any other number) because the matmul of $Q$ and $K$ should have a mean of 0 and variance of 1, and you get a gentler softmax.

Why does this multiplication have a variance of $d_k$?
If I understand this, I will then understand why dividing by $\sqrt({d_k})$ would normalize to 1.
Trying this experiment on 2x2 arrays I get an output of 1.6 variance:

Even though here programming questions are generally off-topic, in case the code is necessary, it's preferable to provide it rather than an actual screenshot. So, please, if you can still provide the actual code, edit your post to include it.

In statistics, if $X$ and $Y$ are independent and randomly distributed variables:
$\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\
Var(X + Y) = Var(X) + Var(Y) \\
\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y] \\
Var(XY) = (Var(X) + \mathbb{E}[X]^2)(Var(Y) + \mathbb{E}[Y]^2) - \mathbb{E}[X]^2\mathbb{E}[Y]^2$
Let $Q$ and $K$ be random $d_k$ x $d_k$ matrices, where each entry is a some random distribution with $0$ mean and $1$ variance. Every entry is independent from each other.
Since each entry of $Q$ and $K$ have identical distribution, we can focus only on the top-left-most element of $QK$ without loss of generality. The same applies to every other element.
The top-left-most element of $QK$ is $\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1}$.
Since $Q$ and $K$ are independent:
$\mathbb{E}[Q_{1, i} K_{i, 1}] = \mathbb{E}[Q_{1, i}] \mathbb{E}[K_{i, 1}] = 0 \\
Var(Q_{1, i} K_{i, 1}) = (Var(Q_{1, i}) + \mathbb{E}[Q_{1, i}]^2)(Var(K_{i, 1}) + \mathbb{E}[K_{i, 1}]^2) - \mathbb{E}[Q_{1, i}]^2\mathbb{E}[K_{i, 1}]^2 = 1$
And so summing up $d_k$ of them:
$\mathbb{E} \left[\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1} \right] = \displaystyle \sum_{i=0}^{d_k} \mathbb{E} \left[ Q_{1,i} K_{i, 1}  \right] = 0 \\
Var\left(\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1} \right) = \displaystyle \sum_{i=0}^{d_k} Var\left( Q_{1,i} K_{i, 1}  \right) = d_k$
For your code block, you are computing the dot product of matrices $a$ and $b$, when you should be doing a matrix multiplication (the attention function multiplies $Q$ by $K$ after all, which is the vectorized form of dot-product -- it doesn't actually do dot product). It should work out to unit variance.
edit: the last paragraph is incorrect, as dot is the same as matrix multiply in the above case

Awesome, thank you this is a great answer. I am still confused on the code part though looking at the docs for numpy the np.dot function should be equivalent to np.matmul in 2 dimensions.

Ah, you are right! dot is the same as matmul. I'm not sure how to explain the code part in that case. Maybe dimension=2 is too small? The sample variance will itself have a variance that reduces with larger dimension size.

Hmmm I see I will try with larger shapes, in any case your answer provided great insight to my initial question so thanks for taking the time!

I appreciate it if you answer this related question https://ai.stackexchange.com/questions/40244/in-the-attention-mechanism-why-dont-we-normalize-after-multiplying-values

Assume that the query embeddings $Q$ and key embeddings $K$ have zero mean and unit std. Then for the variance of the attention score between any query and key we get:
$$ \alpha = q_i k_j^T = \sum_{n=1}^{d_k} q_{in} k_{jn} $$
$$ \text{Var}(\alpha) = d_k $$
$$ \text{std}(\alpha) = \sqrt{d_k} $$
But is it safe to assume that our query embeddings and key embeddings have unit variance?
The answer is yes. The embeddings are computed by multiplying the input with the query and key matrices ($W_Q$ and $W_K$). We can assume that the input already has unit variance by using a normalizing layer (e.g. LayerNorm), and the weights of the $W_Q$ and $W_K$ matrices should be initialized so that variance is preserved.
But why do we want unit variance at all?
Applying softmax on the attention scores with such high variance will result in all of the weight being placed on one random element, while all the other elements will have a weight of zero. Thus, in order to have the attention scores with unit std, we scale by $\sqrt{d_k}$.
For more checkout this.

Very good answer! For anyone wondering, we can approximate unit variance for out weights matrix using Xavier-Glorot:

The mathematical explanation is already explained in the another answer. But I feel bothered why OP's code get an output of 1.6 variance, hence I run my own experiment to prove it:
import numpy as np
a = np.random.randn(1111,111)
print('variance of matrix A =', a.var())
b = np.random.randn(111,1111)
print('variance of matrix B =', b.var())
c = a @ b
print('variance of dot-product =', c.var())
d = c / np.sqrt(111)
print('variance of scaled (normalized) dot-product =', d.var())

The output:
variance of matrix A = 0.9998308190814476
variance of matrix B = 0.9957289239999045
variance of dot-product = 110.5603610976103
variance of scaled (normalized) dot-product = 0.9960392891676604

As shown above, the variance is becoming (close to) 1 after dividing by $\sqrt({d_k})$.

It might help to take two small matrices that match the assumptions (mean of zero and variance of one) and just do the matrix multiplication.  The dimensionality of K scales Q in the multiplication, scaling the variance simultaneously.

Hey, I did do this but it does not come out as expected. I have edited the post


text:
How to design a NLP algorithm to find a food item in menu card list?
I am new to NLP and AI in general. I am just expecting springboard information so that I can skip all the introduction to NLP websites.  I have just started studying NLP and want to know how to go about solving this problem. I am creating a chatbot that will take voice input from customers ordering food at restaurants. The customer input I am expecting as;

I want to order Chicken Biryani

Can I have a Veg Pizza, please

Coca-cola  etc

I want to write an algorithm that can separate the name of the food item from the user input and compare it with the list of food items in my menu card and come up with the right item.
I am new to NLP, I am studying it online for this particular project, I can do the required coding, I just need help with the overall algo or sort of flow chart. It will save my time tremendously. Thanks.

Since you want a shortcut use the spoonacular API.  Below is a test with your words.  You can see it had trouble with 'Coca' and 'veg'.

What you need is 'named-entity recognition' for food.  This is not a new thing but clearly not a solved problem.  The Foodie Favorites repository attempts to solve the problem from scratch.
If want to do some research and dig deeper see FoodBase corpus: a new resource of annotated food entities.  From the abstract:

It consists of 12,844 food entity annotations describing 2105 unique food entities. Additionally, we provided a weakly annotated corpus on an additional 21,790 recipes. It consists of 274,053 food entity annotations, 13,079 of which are unique.


text:
How to handle class imbalance when the actual data are that way
My supervised learning training data are obtained from actual data; and in real cases, there's one class that happens less often than other classes, just around 5% of all cases.
To be precise, the first 2 classes are in 95% of training data and the last one is in 5%. Training while keeping the data ratio intact will make accuracy reach 50% at the right first step and reaches 90%+ immediately that doesn't make sense.
Should I exclude some data of classes 1 and 2, to make the numbers of samples of 3 classes equal? But it's not a real-world ratio.

You can use stratified cross-validation combined with an imbalanced learning technique applied to the training data. Stratification ensures that when you split your data into train and test, the ratio of frequencies between the classes will stay the same, and therefore the test data will always be "realistic".
However, when training a model (using only the training data, of course), the imbalance may have a negative impact. Therefore, have a look at some of the imbalanced learning techniques that are out there to remedy this situation. For example, you could try these:

random undersampling: discard random examples from the majority classes until the ratios of class frequencies are close to 1
random oversampling: make random duplicates of minority class examples until the ratios of class frequencies are close to 1
SMOTE: like random oversampling, except that synthetic examples are created instead of random duplicates
balanced bagging: performs random undersampling but does so multiple times to create an ensemble of models trained on balanced subsets of the training data

etc.
You should also take care about the metrics you use to assess predictive performance on the test data. Accuracy could be misleading here, so you may instead find metrics like sensitivity and specificity (calculated for each class individually) more informative.


text:
How can Image Caption work?
I have two models and a file contains captions for images. The output of model 1 is .pkl files that contain the features of the images. Model 2 is the language model that will be trained with the captions. How can I link between two models to predict a caption for any image? The output of model 1 should be the input of model 2. But the features only are not enough so the input of model 2 will be .pkl files + caption file. Right?
If someone can help me in getting the link between the two models, I will appreciate it.

The Standard Image Captioning Pipeline is to train the model in a single batch(or mini-batch) i.e. get the features from the CNN Image encoder and then feed that into an RNN decoder (features + Real Captions) to produce output captions for the Image.
The training loop in PyTorch would look something like this:
# zero the parameter gradients
decoder.zero_grad()
encoder.zero_grad()

# Forward pass
features = encoder(image)
outputs = decoder(features, captions)

# Compute the Loss
loss = criterion(outputs.view(-1, vocab_size),
                         captions.view(-1))

# Backward pass.
loss.backward()

# Update the parameters in the optimizer.
optimizer.step()

I'd suggest you go through the paper Show and Tell: A Neural Image Caption Generator.
I also made this Kaggle Kernel implementing the paper from scratch. Should help clear up any other doubts.

thanks for replying , should I train the language model for the captions of my own dataset separately?
Will I need the class code separately to link between the two models?

Yes you should train the RNN decoder for your particular dataset separately again (for best performance). I'm not able to understand your second question, can you please clarify

i have the two models separately .. i didn't write the  code that makes a caption for the image yet. how can i start for the linking between them


text:
How to avoid being stuck local optima in q-learning and q-network
When using the Bellman equation to update q-table or train q-network to fit greedy max values, the q-values very often get to the local optima and get stuck although randomization rate ($\epsilon$) has already been applied since the start.
The sum of q-values of all very first steps (of different actions at the original location of the agent) increases gradually until a local optimum is reached. It gets stuck and this sum of q-values starts decreasing slowly a bit by a bit.
How to avoid being stuck in a local optimum? and how to know if the local optimum is already the global optimum? I may think of this but it's chaotic: Switch on randomization again for a while, worse values may come at first but maybe better in the future.

I found out the problem of why the optimization process got stuck and never moved closer to the global optimum. It's because of the rate between 'explore' or 'exploit'.
Basically, in RL, the agent explores by doing a random action and to find new solutions, exploits the existing so-called known max future rewards to do the max action.
Initially, I put the agent to explore when $random() < 1/(replay\_index+1)$, exploration rate reduces too quick (<10% after 10 iterations), and when the number of replays (number of times to play again from start) is not enough, the explore rate at the end of the loop is almost zero, and nothing new learned.
The solution opted is allowing 'explore' and 'exploit' to have the same rate (or lower exploration a bit is also ok), pseudo-code:
# Part 1 in a step: Choose action
if random() < 0.5: # 0.25 is also good, 25% for exploration
    action = random_action()
else:
    action = choose_best_known_action()

Explore rate can be reduced correctly this way:
if random() < 1-i/NUM_REPLAYS: # i is current train step index
    action = random_action()
else:
    ...

With the half-explore/half-exploit scheme above, the agent will learn to infinity, so, it is kinda sure that the global optimum would be reached. When knowing from practice the number of iterations that should be used, 'exploit' may be utilized more for faster convergence.
Note that the 'explore' and 'exploit' rates are put equal above, the but q-table or q-network is still better and better due to having another 'exploit'-kind when updating q-table or fitting q-network with Bellman equation, there's another 'exploit' here, the 'max' in Bellman equation:
Pseudo-code:
# Part 2 in a step: Update q-table or q-network
q[s][a] += learning_rate * (reward + max(q[sNext][aNext]) - q[s][a])

# Q-network
# target = r + max(...
```

what do you mean with "when the number of replays (number of times to play again from start) is not enough"?

it's the outer most interation, RL plays from square 1, again and again a 'number of replays'

not enough replays --> not enough iterations (the outer most loop)


text:
Is AI already being used in the drug industry to combat the COVID-19?
We all have heard about how beneficial AI can be in health. There are plenty of papers and research about confronting diseases, like cancer. However, in 2020 with COVID-19 be one of the most serious health problems that have caused thousands of deaths worldwide.
Is AI already being used in the drug industry to combat the COVID-19? If yes, can you, please, provide a reference?

I have changed your question so that this post is more useful. Note that I have already asked a similar question: https://ai.stackexchange.com/q/18701/2444, but your question now is slightly different.

A global race is underway to discover a vaccine, drug, or combination of treatments that can disrupt the SARS-CoV-2 virus.
The problem is, there are more than a billion such molecules. A researcher would conceivably want to test each one against the two dozen or so proteins in SARS-CoV-2 to see their effects. Such a project could use every wet lab in the world and still not be completed for centuries.
Computer modelling is a common approach used by academic researchers and pharmaceutical companies as a preliminary, filtering step in drug discovery. However, in this case, even every supercomputer on Earth could not test those billion molecules in a reasonable amount of time.
Folding@home is a distributed computing project run by Stanford University. The aim of the project is to examine how proteins fold and it does this using spare computing power. however, there is a lot of research in progress that are harnessing the potential of Artificial intelligence to develop the potential treatment to combat the COVID-19.
Check this recent article By Tyler Orton in biv focus on how artificial intelligence is used to accelerate the process of Drug Discovery:
Drug research turns to artificial intelligence in COVID-19 fight
Here are the list of some companies that are using AI-driven approach for Drug discovery

BlackThorn Therapeutics
xscientia
Insilico Medicine
Insitro
Notable Labs
Standigm
Recursion Pharmaceuticals

The Hong Kong-based company Insilico Medicine, a developer of comprehensive drug discovery and biomarker development platform GENTRL, and a pioneer in the application of generative adversarial networks (GANs) to drug discovery.
Insilico Medicine, Publishes a paper in September last year titled,
Deep learning enables rapid identification of potent DDR1 kinase inhibitors," in a most reputed journal Nature Biotechnology. The paper describes a timed challenge, where the new artificial intelligence system called Generative Tensorial Reinforcement Learning (GENTRL) designed six novel inhibitors of DDR1, a kinase target implicated in fibrosis and other diseases, in 21 days.
Four compounds were active in biochemical assays, and two were validated in cell-based assays. One lead candidate was tested and demonstrated favorable pharmacokinetics in mice.

I'm not sure if it is being used directly in the industry, but here is an interesting article on research being done by 3 UK universities using AI.


text:
What is the $\ell_{2, 1}$ norm?
I'm reading this paper and it says:

In this paper, we present a multi-class embedded feature selection method called as sparse optimal scoring with adjustment (SOSA), which is capable of addressing the data heterogeneity issue. We propose to perform feature selection on the adjusted data obtained by estimating and removing the unknown data heterogeneity from original data. Our feature selection is formulated as a sparse optimal scoring problem by imposing $\ell_{2, 1}$-norm regularization on the coefficient matrix which hence can be solved effectively by proximal gradient algorithm. This allows our method can well handle the multi-class feature selection and classification simultaneously for heterogenous data

What is the $\ell_{2, 1}$ norm regularization? Is it L1 regularization or L2 regularization?

It's a matrix norm. You have it here.

$\ell_{2,1}$ is a matrix norm, as stated in this paper.
For a certain matrix $A \in \mathbb{R}^{r\times c}$,
we have
$$\|A\|_{2,1} = \sum_{i=1}^r \sqrt{\sum_{j=1}^c A_{ij}^2}$$
You first apply $\ell_2$ norm along the columns to obtain a vector with r dimensions. Then, you apply $l_1$ norm to that vector to obtain a real number. You can generalize this notation to every norm $\ell_{p,q}$.


text:
Is NLP likely to be sufficiently solved in the next few years?
The reason I am asking this question is because I am about to start a PhD in NLP. So I am wondering if there would be as much job opportunities in research in industry as oppose to in academia in the future (~ 5 to 10 years) or would it be mostly a matter of using a library off the shelf. I have done some research and it seems NLP is AI-complete, which means it's probably a problem that will be "solved" only when AGI is solved , but still I would appreciate any input.

You should directly talk and ask advice to your possible advisors. They will probably give you a better overview of the state of affairs of the field. This question seems to be primarily opinion-based, so it should be closed here. Maybe you can reformulate it in such a way that it can be answered unambiguously.

@Ash a little side note: without a doubt, NLP will be a research topic in 5 years. granted you can get a lot of utility of taking something off the shelf but there are still many weaknesses to these models and unknown characterstics. It is far from solved

Apologies.  This is a valid question, just not well suited for this Stack.  I'm closing provisionally, but I think there is a valid question here: "IS NLP likely to be sufficiently solved in the next few years?".  (Feel free to revise and submit for re-opening.  By reformulating the question to be suitable for this stack, you should be able to get the info you're looking for.)


text:
Determine Frequency from Noisy Signal With Neural Networks (With Adeline Model)
I'm trying to determine the frequency from a signal with NN. I'm using the Adeline model for my project and I'm taking a few samples in each 0.1-volt step in a true signal and a noisy one.
First question: am I wrong?
Second question: my network works fine until the frequency of my sample for the test is equal to the frequency of my sample for the training. Otherwise, my network doesn't work and gives me the wrong answer.
What do I need to do for this model?
for solving this problem I must use nonlinear steps like logarithmic steps. but How to use logarithmic steps in MatLab?
Edition: I understand my problem is not Overfitting! I found that my samples step are linear and my samples are nonlinear so this is wrong
for solving this problem I must use nonlinear steps like logarithmic steps. but How to use logarithmic steps in MatLab?

Tanks for edit Glorfindel :D


text:
Is there any classifier that works best in general for NLP based projects?
I've written a program to analyse a given piece of text from a website and make conclusary classifications as to its validity. The code basically vectorizes the description (taken from the HTML of a given webpage in real time) and takes in a few inputs from that as features to make its decisions. There are some more features like the domain of the website and some keywords I've explicitly counted.
The highest accuracy I've been able to achieve is with a RandomForestClassifier, (>90%). I'm not sure what I can do to make this accuracy better except incorporating a more sophisticated model. I tried using an MLP but for no set of hyperparameters does it seem to exceed the previous accuracy. I have around 2000 datapoints available for training.
Is there any classifier that works best for such projects? Does anyone have any suggestions as to how I can bring about improvements? (If anything needs to be elaborated, I'll do so.)
Any suggestions on how I can improve on this project in general? Should I include the text on a webpage as well? How should I do so? I tried going through a few sites, but the next doesn't seem to be contained in any specific element whereas the description is easy to obtain from the HTML. Any help?
What else can I take as features? If anyone could suggest any creative ideas, I'd really appreciate it.

First of all, there are multiple factors on how well models will work. Amount of data, source of data, hyperparameters, model type, training time etc... All of these will affect the accuracy. However, no classifier will work best in general. It all depends on the different factors, and not one can satisfy all, at least for now.
For improving the accuracy, we need to first make those factors ideal so that the classification will have a higher accuracy.
First of all, how many data do you have? If you are using html webpage, you probably need at least 10000 data samples. If you have at least that amount of data you should be ok with overfitting. You also need to clean the data. One way to do it is to tokenize it. Tokenization of text data basically means to split the text into words and make a dictionary out of it. Then each word is encoded to a specific number where each same word have the same encoding. You are using the raw HTML as input, which have a lot of unnecessary information and tags and stuff, you can try removing those or completely remove all html tags if they are not required. The key to cleaning the data is to extract the pieces of information that is important and necessary for the model to work.
Then, you should explore the model. For a NLP (Natural Language Processing) task, your best bet is to choose a RNN (Recurrent Neural Network). This type of network have memory cells taht helps with text type data as text often have distant linkage in a paragraph, for example one sentence may use a "she" that refers to a person mentioned in two sentence before, and if you just feed every single encoding of words in a MLP, it would not have this memory for the network to learn long term connection between text. A RNN also is time dependent, meaning it processes each token one by one according to the direction of time. This makes the text more intuitive to the network as text is designed to be read forward, not all at once.
Your current method is to first vectorize the HTML code, then feed it into a random forest classifier. A random forest classifier works great, but it cannot scale when there is more data. The accuracy of a random forest classifier will stay mostly the same when data increase while in deep neural networks the accuracy will increase with the amount of data. However a deep neural network will require a large amount of data to start of with. If your amount of data is not too much (< 10000), this method should be your choice of method. However if you plan to add more data or if teh data is more, you should try a deep learning based method.
For deep learning based method, ULMFit is a great model to try. It uses a LSTM(Long Short Term Memory) network (which is a type of RNN) with a language model pretraining and many different method to increase the accuracy. You can try it with the fast.ai implementation. https://nlp.fast.ai/
If you wish to try a method that you can practically implement yourself, you could try to use a plain LSTM with one hot encoding as input. However, don't use word2vec to do preprocessing as your input data is html code. The word2vec model is for normal English text, not the html tags and stuff. Moreover a custom encodings will work better as in the training process you can train teh encoding as well.
Hope I can help you

What do you mean by custom encodings? Also, I'm only using a small part of the HTML. Actually, the program searches the HTML for the 'description' tag, and then it vectorizes any of the words that it can find in the GloVe vector space. If it can't find those words, I omit it. I am not sure how to utilise the HTML in a better way. Is there any way I could use the text on the page? I couldn't find any single tag that holds all the text, so it seemed complicated. Also, I'm using 2400 datapoints (2100 training and 300 testing). I'm still new to Deep Learning, so I just used a standard classifier.

By custom encoding I mean making a dictionary of all teh tokens and assigning each of them an individual number, and convert every single word in the input text to teh number, then feed the one hot encoding of the number to the network

But if you are using the description tag, you can use word2vec or GloVe but a custom encoding will work better as the encoding will be trained as well with teh gradients back propagated

https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html    try this for the custom embedding /encoding I mentioned. Hope I can help you.

What I am doing is taking all the words in the description, embedding them into vectors, and adding up all of the vectors to get a vector for the whole description. I am putting this description vector in my model as an input. Should I be doing anything differently?

How did you embed the words to vectors? Are you using word2vec?

Yes, using the GloVe word2vec embeddings to create 300 dimensional vectors. And then adding up all the words. The reason I did this is because my model needs to have a fixed shape input, so I couldn't think of any other way but to go ahead and just add them to get a fixed 300 length vector.

You can check out my answer to your new question. Hope I can help you.

The accuracy depends on various factors. Might not always be the algorithm. For example a cleaner data with a poor algorithm might still give better results and vice versa.
What are the preprocessing techniques you are using? This preprocessing techniques article is a good starting point for html data. And by vectorising I assume you mean word2vec, use a pre-trained word2vec model. Like google's word2vec model it's trained on a lot of data(about 100 billion words).
LSTM performs good whenever the intent of the sentence is important. Check out this.
Ram hit Vijay and Vijay hit Ram, might mean the same to most algorithms. Example, Naive Bayes.


text:
GAN model predictions before training is predictable
I have a dataset of 3000 8x8 images, and I would like to train a GAN for an image generation purpose.
I am planning to start with a simple GAN model and see if it overfits. Before training, I try to do a comparison of the discriminator model prediction using real image input against the whole GAN model prediction using random seed input. My thought process is that since this model is not trained yet, the output for real images and fake images by the discriminator should not be predictable.
However, the discriminator model prediction using real image input always returns a value very close to 1.0, and the whole GAN model prediction using random seed input always returns a value near 0.5 with a small deviation. I suspect that during training, the model would simply pull the 0.5 value near 0.0 and would never actually learn from the dataset.
I try to increase the training parameters and different initializers, but the output is still the same.
By ruling out the possibility a bad dataset, what could be the reason for this situation?
This is some sneak peek of the generator and discriminator model building: https://pastebin.com/ehMDP7k6

what do you mean by "a comparison of the discriminator model prediction ... against the whole GAN model prediction"? do you mean the output of your generator for fake and real images?

In other words, I feed real images array into the discriminator model, and it outputs values close to 1.0. Next, I feed fake images (the one that is generated by the generator) into the discriminator model, and it outputs values close to 0.5. These happens too often, and it is very predictable.

Did you try to generate fake images? Since the main goal of GANs to produce synthetic samples, the main matric will be the quality of generated images. Take a look at this answer. There is a plot of G loss and D loss of DCGAN. We can see that D  loss is near 0, whereas G loss has oscillations, but G was able to produce good samples.

Yes, and it only generates noises. Although, I do notice some preference of a certain pallete, and I think this is a good start. The D accuracy went constant at 0.5 after 20 epochs, and the G accuracy went up to 1.0, and both losses gradually went down. All metrics did oscillate in the middle of the training but in small amount of deviation. Did my model overfit at epoch 20?

I took a look at your model. It seems you have incorrect architecture. The Conv2D layers in your D should have following params: (n_filters, kernel=3, padding='same'), where n_filters is the number of filters and it usually should be doubled as per DCGAN architecture. You can also use strides but since your images are small it won't make any sense.
The D network also should not include UpSampling2D layer. This layer can be used in G network instead of Conv2DTranspose.
Your images should be normalized to the range of [-1, 1] and the activation function of G should be tanh.
You also included m.add(keras.layers.Reshape((8, 8, 3))) to your G network which is the final size of your data. Since you have the final size in the first layers you don't have to include upsampling Conv2DTranspose layers.
Your final model should look like
def make_discriminator_model():
    model = tf.keras.Sequential()
    model.add(layers.Conv2D(8, kernel=3, strides=1, padding='same',
                                     input_shape=[8, 8, 1]))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Conv2D(16, kernel=3, strides=1, padding='same'))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Flatten())
    model.add(layers.Dense(1))

    return model

def make_generator_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(8*8*16, use_bias=False, input_shape=(100,)))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Reshape((8, 8, 16)))

    model.add(layers.Conv2D(8, kernel=3, strides=1, padding='same'))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2D(1, kernel=3, strides=1, padding='same', activation='tanh'))

    return model

Note that I did not test the model, so you should adapt it to the shape of your data.
I encourage you to follow this tutorial.
Update regarding overfitting
GANs do not have overfitting problem in the classical sense. Instead, vanilla GAN have other problems like vanishing gradient. It happens when the D becomes overconfident regarding fake samples. In that case, D stops providing useful information to the G, since its error becomes 0. To find out if training fell into this problem, you should plot D and G loss. It will look as follows:

Another problem is the mode collapse. In that case, the G tricks the D by producing only one type of samples which looks realistic but G won't represent the real data distribution. Therefore, the generated samples will be homogeneous.
For more information, see Improved Techniques for Training GANs and Wasserstein GAN.

Thank you for the guidance. I certainly built my models with baseless knowledge, and most importantly, I forgot to normalise my image array before feeding into the discriminator, which would explain a lot at my initial problem. I accepted this answer for raising the normalisation problem, and upvoted for the extra guidance.

I forgot to mention the overfitting issue. I updated the answer accordingly


text:
Can a large discrete action space be represented using Gaussian distributions?
I have a large 1D action space, e.g. dim(A)=2000-10000. Can I use continuous action space where I could learn the mean and std of the Gaussian distributions that I would use to sample action from and round the value to the nearest integer? If yes, can I extend this idea to multi-dimensional large action space?

The answer is "it depends". Once you have arranged the actions into order, a key trait is whether the action value function has a simple enough shape that sampling from a Gaussian policy function would give consistent expected returns, enough that learning can occur. If the underlying "true" value function has a lot of high frequency noise then learning would be slow. In the worst case, if the action value $Q(s,a_{n})$ and $Q(s,a_{n+1})$ is not correlated for any $n$, then it will not be possible to learn with the approximation at all.
You may have some sense of how similar actions $a_{n}$ and $a_{n+1}$ are. If the actions represent different ordinal choices, such as selecting an integer number of items to perform some task with such as buy/sell or transport, then in many environments there will often be a strong correlation between outcome of choosing e.g. $a_{900}$ and $a_{901}$. If this holds in general, then that is a good indicator that you can treat $n$ as being continuous, and use learned parameters of a simple distribution function to find optimal policies (and in addition this could be far more efficient than using a discrete representation).
It might not matter if for a small fraction of cases the difference in outcomes between $a_{n}$ and $a_{n+1}$ is large, provided that successive approximations to the optimal policy can improve towards optimal through adjusting mean $\mu(s)$ and standard deviation $\sigma(s)$.
There may still be difficult cases that cannot be learned by the approximation - for instance if a specific action $a_n$ is optimal for a given state $s$, but $a_{n-1}$ and $a_{n+1}$ are a lot worse, then the training process of a typical policy gradient approach may never settle upon $\mu(s) = n$ and $\sigma(s) \approx 0$, because any intermediate values between the starting policy and the optimal one will perform badly.
For expanding into more dimensions, the same ideas apply to each dimension separately. You may want to use different distributions, or even have one dimension that uses a continuous model with a few parameters whilst another remains discrete with a free parameter for each choice.


text:
Can action be dominated by state features in actor-critic algorithms?
I have a case where my state consists of relatively large number of features, e.g. 50, whereas my action size is 1. I wonder whether my state features dominate the action in my critic network. I believe in theory eventually it shouldn't matter but considering the sequential nature of RL training I am afraid the state features outweigh the action and its effect will be negligible.
What I already tried is the following:

Here where the state output and action are combined I use tanh activation because my action is in [-1, 1]. This led to almost flat performance from the very beginning with no improvement at all. I understand this might be due to vanishing gradients caused by tanh. I also tried the linear activation instead of the tanh, this time average episode return was fluctuating around some value with no signs of learning.
What I am currently testing is stacking the action, say 50 times, to match the number of the state features.
Any other ideas on how to tackle this issue.

Hi @Mika! Just to clarify - what is the input to your critic network? I.e., are you using the action as an input at all? It seems that way based on the wording of this question.

Hey @DeepQZero. Yes I am using the action as an input along with the state. But like i said state has a dimension of 50 and action has only one. Let me know if that helped.

It helped, but I personally would need to understand more details before supplying an answer. It looks like there is an answer on this question already, and I think I'll pass on answering this question for now.

@DeepQZero, I added more information if that helps understand the problem better

It is certainly possible for the state features to dominate the action features in the critic.
There are several strategies you can use:

Replace the action features with a high dimensional learned embedding vector. This way you can scale up it's importance.

Introduce the action at a deeper stage of the network. This way, the action is being combined with the state after the state has been compressed by the earlier layers

Simply have many action features inputs that are all identical. This is similar to method 1 but easier.

I have used all 3 of these methods. They all worked well in different cases.

Hi @chessprogrammer. Thanks for the answer, I have also thought about similar methods. I tried method 2 already, i.e. learning the state embedding by reducing it to a size one representation and combining it with the action. This didn't work, basically my performance was fluctuating around some value and not improving at all. Regarding method 3 I believe you mean just to stack the copies of the action, say 50 times to match the number of the features in the state, right? I am already doing this after failing with method 2, waiting for the results, will post how it did.

Not sure about the method 1, though. You mean to pass the action with size of one through several layers, then combine the learned representation with the state features, right?

Hi. First off, you misunderstood method 2. I am not saying to compress the state space to one before combing. That is way too low. I am saying to make it smaller. Not that small. And method 1 I mean passing it through an embedding layer. Look up torch.nn.embedding

I added a figure of what I did that I thought is similar to method 2; now I am confused. My understanding of torch.nn.embedding is that it's just a trainable look-up table; moreover, my action is continuous, not sure how I'd encode my action space.


text:
Are actions deterministic during testing in continuous action space PPO?
In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $\mu$ and $\sigma$. This results in an inherent exploration. However, during testing, when we no longer need to explore but exploit, the action should be deterministic, i.e. just $\mu$, right?

Which specific RL algorithm are you referring to? Please, edit your post to say this explicitly and provide a link to the paper. Moreover, please, put your specific question in the title. "Action selection during testing in continuous action space" is not a question but a topic/problem.

Proximal Policy Optimization would be one example. So, some algorithm where the policy network generates action values instead of state-value estimates or the like.

@DanielB. Ok, let's see what the OP is referring to. I think that a little bit more of context is needed to answer this question properly.

@nbro, I meant all algorithms that sample their actions using the given $\mu$ and $\sigma$, i.e. Reinforce, A2C, A3C, TRPO, PPO, SAC unlike DDPG and TD3 which return just $\mu$ as their action. But for clarity I edited the my question and made it for PPO. And thank you for improving my question.

Btw, a thorough explanation of the procedure for sampling actions from a distribution parameterized by a network's predictions/outputs is provided in Section 6 of this paper. Might be that this paper also contains the answer to your question. Haven't read it to the end yet, though. So, I can't tell.


text:
How to parallelize multi-agent DDPG (MADDPG)
I am experimenting with MADDPG algorithm implemented in this repo. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action prediction, etc. are done in a for loop. I want to increase the number of agents, say 10 or 30, and perform parallelization of the above-mentioned steps for all agents, i.e. I want to avoid for loops like this
for agent_idx in range(n_agents):
    ...
    ...

I tried Python Multiprocessing module with pool.map method but I am getting 'AttributeError: Can't pickle local object ...". Below is code I am running to get a joint action prediction but it results in the error above.
def get_ind_action(i, obs_i):
    return actor_critic[i].act(obs_i) # returns an individual action for a given observation for ith agent

def get_joint_action(obs):
    pool = Pool()
    args_list = [[i, obs[i]] for i in range(n_agents)]
    joint_action = pool.map(get_ind_action, args_list)
    return joint_action

Here actor_critic is a list of neural networks of all agents, obs is the joint state observed by the centralized critic but each actor only sees its own state. The algorithm has the following architecture.

Hey, maybe to help someone unfamiliar with MADDPG (like me) to quickly grasp the difficulty of parallelizing it can you clarify that (a) are the agents gonna share info about their parameters at some point? (b) and if any other information is going to be shared among process in the learning phase? (c) are you perhaps planning to parallelize on single node or multi nodes? (d) to answer the error perhaps share short reproducible parts of the code that you add to the code you linked

(a) they have identical architecture, i.e. the same number of layers and units in each layer but share no parameters. (b) no information is shared among agents' actor networks. (c) single node. (d) added above

You have to pass the agent as the argument to the function, because the subprocess do not have the agent in the memory. You might want to pass the actor's state_dict instead of the agent and reinstantiate the agent in the subprocess->load the state_dict -> and predict the action -> return the action

Yes, I think actor’s state_dict should be passed, I’ve just tried to pass the agent and I got the same error. Let me give a try. Thanks.

if you're on linux try other start method (spawn, fork, forkserver) and see the effect on the running time, or even start all processes first and send the state_dict and state to Queue to prevent instantiating processes on every step.

@Mika were you able to solve this problem? Any help would be much appreciated


text:
Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?
I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after some time). So, it is getting difficult for me to continue training in this environment.
The continuous state-space is $\mathcal{S}  \subseteq \mathbb{R}^{10}$ and the continuous action-space $\mathcal{A} \subseteq \mathbb{R}^{2}$.
What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?
Or I should go with the behavior cloning technique to train the actor-network only, so that it converges rapidly?
I just want to get the work done first and then I can think of exploring the environment.

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?

You certainly can, that is indeed one of the advantages of off-policy learning algorithms; they're still "correct", regardless of which policy generated the data that you're learning from (and a human expert providing the experience to learn from can also be viewed as such a "policy").
There are potential issues to be aware of though. For example, if you just put some expert-generated data in there and don't allow your agent to explore by itself, the experiences that you can learn from may be quite limited in the parts of the state-action space that they explore. So if your expert does not sufficiently explore the entire space, you cannot expect the agent to learn how to act if for whatever reason it ever ends up in some unexplored space. This is no different from what would happen if you trained with an agent that had too little exploration (like a greedy agent).

Or I should go with the behavior cloning technique to train the actor network only, so that it converges rapidly?

I cannot confidently say which approach would work better, so I cannot really answer this... I imagine the answer may also be different for specific different problem domains. But the basic principle of learning from expert data with an off-policy algorithm is not inherently wrong.

Thank you for the answer. You are right when you say that I should disable exploration when I am putting expert data in replay buffer. When I first ran the agent with this setup, I forgot to disable the exploration part and the agent slowly drifted to bad policy from a policy which was similar to expert.