text:
How fast are autoencoders?
I was exploring image/video compression using Machine Learning. In there I discovered that autoencoders are used very frequently for this sort of thing. So I wanted to enquire:-

How fast are autoencoders? I need something to compress an image in milliseconds?
How much resources do they take? I am not talking about the training part but rather the deployment part. Could it work fast enough to compress a video on a Mi phone (like note8 maybe)?

Do you know of any particularly new and interesting research in AI that has enabled a technique to this fast and efficiently?

Actually it depends on the size of your AE, if you use a small AE with just 500.000 to 1M weigths, the inferencing can be stunningly fast. But even large networks can run very fast, using Tensorflow lite for example, models are compressed and optimized to run faster on Edge-devices (Handys for example, end-user devices). You can find a lot of videos on Youtube, where people test inferencing large networks like Resnet-51 or Resnet-101 on a raspberrypi, or other SOC Chips. Handys are comparable to that, but maybe not that optimized.
For example,I have an Jetson Nano (SOC of Nvidia costs arround 100 euro) and i tried to inference a large Resnet with arround 30 million parameters over my fullHD Webcam. Stable 30 FPS, so speaking in milliseconds its around 33 ms per image.
To answer your question, yes Autoencoders can be fast, also very fast in combination with an optimized model and hardware. Autoencoder structures are quite easy, check out this medium,keras example

Well, I will be testing it out. But my assumption is that the encoder is able to compress at least a single image in milliseconds, so as to process a continuous stream of video. Is that idea viable, or too much to hope for? Maybe the autoencoder might take more time for mobile devices....

It depends on your image size and the size of the compression you want! Usually deep learning algorithms are not so fast as why they run on GPU, and we have highly optimized frameworks like TensorFlow! Something I can say for sure is:

Compressing video using autoencoders means compressing each frame one by one! However, video compressions usually contain the calculation of the deference of every frame with the previous frame. This means the compressing video is much more time consuming than compressing just a single image.

The encoder is half part of the autoencoder, so the compression is faster than training the whole autoencoder.

Use GPU! It really makes much different!

Try Google Colab! You can choose between CPU and GPU and then make a decision.


text:
Can I use an autoencoder with high latent representational space?
I am trying to use a neural network to predict the next state output given the current state and action pairs. Both input and outputs are continuous variables. Due to the high dimensionality of each input, ( ~50 dimensional input ) and 48 dimensional output, I am not able to achieve an achieve a satisfiable enough accuracy.
I am thinking of using an auto-encoder to learn a latent representation of the state. Would a latent representation from an auto-encoder help to improve the prediction accuracy ? and can the latent representation have a higher dimensional space compared to the original state ?

i feel , i can answer this question, based the web source that i found and i read. that can use an autoencoder with hight latent representational space. example use LSTM autoencoder, LTSM autoencoder used for sequence or time series data. on the source, they use CNN autoencoder to denoise some syntetic noised data they have generated. but they have asked what is the meaning of this latent representation space? for what they have done before. on that answer. they said"your input data is noisy sinewave data. your are not supposed to use Convolutional autoencoder for sequence data".
https://stackoverflow.com/questions/59438488/cnn-autoencoder-latent-space-representation-meaning
addition other source for answer this question
https://towardsdatascience.com/understanding-latent-space-in-machine-learning-de5a7c687d8d
https://towardsdatascience.com/deep-inside-autoencoders-7e41f319999f
hopefully help answer :))

hey thanks for your answer. I found out what I was looking for. I was looking more at a sparse autoencoder, where the activations in the hidden layer ( > input dimens ) is penalised to prevent the overfitting of the identity function

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Using DCGAN on a (very small) dataset of art
I am developing a DCGAN using the this tutorial in PyCharm. As my usage of this tutorial suggests, I am quite new to DCGANs as I've previously only had a few experiences with machine learning algorithms on classifying problems. My goal is to feed my DCGAN a dataset of paintings of a specific painter, and get 'new' paintings in return. Needless to say, a painter does not paint thousands of paintings in his life, leaving me with a dataset of around 60 paintings. One of the smallest datasets I have ever worked with. I have two, related, questions:

Is it realistic to properly train a DCGAN on this type of dataset? If not, would there be any alternative you would suggest?
What would be a good set of parameters to start of from to properly train this DCGAN?

Thanks in advance!

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How does optical computing work and deal with nonlinearity?
This article states that:

One of the algorithms that photonics is very good at implementing is matrix multiplication

But how are parameters stored and updated(in backpropagation)?
One more serious problem is that there are nonlinear operations in neural networks, then how does the photonic neural network deal with activation functions?


text:
Why are Dueling Q Networks not used more often to approximate Q-values in reinforcement learning algorithms?
I've just learned about Dueling Network Architectures to estimate $Q$-values and am wondering why this architecture is not used more often in deep RL algorithms? DDPG and TD3 estimate the $Q$-function using Double Q Learning instead of the empirically better Dueling Approach.


text:
How to improve neural network training against a large data set of points with varying magnitude
I am currently using TensorFlow and have simply been trying to train a neural network directly against a large continuous data set, e.g. $y = [0.014, 1.545, 10.232, 0.948, ...]$ corresponding to different points in time. The loss function in the fully connected neural network (input layer: 3 nodes, 8 inner layers: 20 nodes each, output layer: 1 node) is just the squared error between my prediction and the actual continuous data. It appears the neural network is able to learn the high magnitude data points relatively well (e.g. Figure 1 at time = 0.4422). But the smaller magnitude data points (e.g. Figure 2 at time = 1.1256) are quite poorly learned without any sharpness and I want to improve this. I've tried experimenting with different optimizers (e.g. mini-batch with Adam, full batch with L-BFGS), compared reduce_mean and reduce_sum, normalized the data in different ways (e.g. median, subtract the sample mean and divide by the standard deviation, divide the squared loss term by the actual data), and attempted to simply make the neural network deeper and train for a very long period of time (e.g. 7+ days). But after approximately 24 hours of training and the aforementioned tricks, I am not seeing any significant improvements in predicted outputs especially for the small magnitude data points.

Figure 1

Figure 2

Therefore, do you have any recommendations on how to improve training particularly when there are different data points of varying magnitude I am trying to learn? I believe this is a related question, but any explicit examples of implementations or techniques to handle varying orders of magnitude within a single large data set would be greatly appreciated.

One option is normalizing your data. In particular, min-max feature scaling to bring all values into the range [0,1] is particularly useful with gradient descent.

It might be that the large labels dominate the loss value, so the model pays more attention to them. You could use an L1 loss rather than an L2 loss, such that the predictions get pulled towards each label equally rather than being pulled more strongly when the label is further away from the prediction.
There may also be a data imbalance, i.e. you train on more large labels than small labels. This would also cause the model to pay more attention to becoming good at predicting the large labels. If this is the case, you can either collect more small labels or train more often on the small labels you have.
Another possibility is that your labels are not evenly distributed. For example, the gaps between large data points may be larger than between small data points, so the small data points all look similar to the model. You can plot a histogram of your labels to find out and then transform your labels (e.g. using log) to make space them out more evenly.
The general rules for training artificial neural networks apply. For example, normalize your inputs and outputs by subtracting the mean and divide them by their standard deviation, estimated across the training set. It also seem that your network is likely too deep and and not wide enough. I'd try 4x100 rather than 8x20 and add a small amount of weight decay when you see overfitting.

I think there is no special method for training the neural network in large datasets. But I can add some suggestion for you:
1) Use the convolutional neural network for this dataset.
2) You can use huber loss instead of squared loss and see what happens.
3) See if you have enough small magnitude training data.
Also, please define your problem in more detail (like what those images represent and what you want to predict, what are the features etc).

Probably you could:

Normalize your dataset
Add Regularization

Hi @Prateek! If possible, please add some supporting information to this answer to help other readers understand why the answer is correct. Thanks!


text:
Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity
I'm working on a RL problem with the following properties:

The rewards are extremely sparse i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering as that would lead to a different optimization problem.
Actions are continuous. Discretization should not be used.
The amount of stochasticity in the environment is very high i.e. for a fixed deterministic policy the variance of returns is very high.

More specifically, the RL agent represents the investor, the terminal reward represents the utility of the terminal wealth (hence the sparsity), actions represent portfolio positions (hence the continuity) and the environment represents the financial market (hence the high stochasticity).
I've been trying to use DDPG with a set of "commonly used" hyperparameters (as I have no idea have to tune them besides experimenting which lasts too long) but so far (after 10000 episodes) it seems that nothing is happening.

My questions are the following:

Given the nature of the problem I'm trying to solve (sparse rewards, continuous actions, stochasticity) is there a particular (D)RL algorithm that would lend itself well to it?
How likely is it that DDPG simply won't converge to a reasonable solution (due to the peculiarities of the problem itself) no matter what set of hyperparameters I choose?

You seem to be using the right kind of algorithm (I would think that most policy gradient methods could at least be applied here, although some may perform better than others). Your problem may well be intractable though. Do you have any evidence that it is solvable given the state data you are using?

The optimization problem I'm trying to solve has been approximated analytically - the approximation to the optimal solution depends on three variables (t-remaining time, q-current inventory and s-current price). It is precisely those variables that I've included in my state space. Therefore, RL should in theory be able to learn this function, provided enough episodes and an NN function approximator with enough capacity.

Use Introduction Multi agent sysrtem

(1) You might want look into RND (Random network distillation) which allows usage of a curiosity-based exploration bonus for the agent as an intrinsic reward. You can use the intrinsic reward to complement the sparse extrinsic reward return by the environment.
The general idea is to have a randomly initialized fixed target network which encodes the next state & a predictor network is trained to predict the output of the target network. The prediction error is used to "quantify the novelty of new experience". Stronger novelty will be a good indication for the agent that it maybe worthwhile to explore more.
The authors of this (A) paper were able to achieve SOTA performance in Montezuma's Revenge, which is notorious for it's sparse reward.
In appendix A.1, It is mentioned that: "An exploration bonus can be used with any RL algorithm by modifying the rewards used to train the
model (i.e., rt = it + et)." It is also mentioned that the authors combined this exploration bonus with PPO (which also works in continuous action space). In A.2, a pseudo code is provided.
I would also recommend this (B) paper (see section 3) if you're interested in exploring the available Bonus-Based Exploration Methods out there which may help in tackling hard exploration games with sparse rewards.
With regards to high stochasticity & variance, I found an interesting remark (on page 3, under Figure 2) in this (C) paper:
"our investigation of DDPG on different network configurations shows that for the Hopper environment, DDPG is quite unstable no matter the network architecture. This can be attributed partially to the high variance of DDPG itself, but also to the increased stochasticity of the Hopper
task."
The remark was made in the context where the authors were trying to "tune DDPG to reproduce results from other works even when using their reported hyper-parameter settings".
Have a look here for a different benchmark on how DDPG fair against other algorithms.
(2) From the information provided, I can't conclusively provide you a quantitative assessment on DDPG performance for your specific problem. However, I would recommend the following:
(a) I will encourage you to try different RL algorithms when face with a difficult problem so that you can benchmark & find out which is more suitable. Also in (A), the authors mentioned, "PPO is a policy gradient method that we have found to require little tuning for good performance."
(b) Try different sets of hyperparameters. There are many ways to tune them systematically but discussion about this will be out of scope for this question.


text:
Subtracting the entropy from our policy gradient will prevent our agent from being stuck in the local minimum?

In the information theory, the entropy is a measure of uncertainty in
  some system. Being applied to agent policy, entropy shows how much the
  agent is uncertain about which action to make. In math notation,
  entropy of the policy is defined  as : $$H(\pi) = -\sum \pi(a|s) \log
 \pi(a|s)$$ The value of entropy is always greater than zero and has a
  single maximum when the policy is uniform. In other words, all actions
  have the same probability. Entropy becomes minimal when our policy has
  1 for some action and 0 for all others, which means that the agent is
  absolutely sure what to do. To prevent our agent from being stuck in
  the local minimum, we are subtracting the entropy from the loss
  function, punishing the agent for being too certain about the action
  to take.

The above excerpt is from Maxim Lapan in the book Deep Reinforcement Learning Hands-on page 254.
In code, it might look like :
 optimizer.zero_grad()
 logits= PG_network(batch_states_ts)
 log_prob = F.log_softmax(logits, dim=1)
 log_prob_actions = batch_scales_ts * log_prob[range(params["batch_size"]), batch_actions_ts]
 loss_policy = -log_prob_actions.mean()

 prob = F.softmax(logits, dim=1)
 entropy = -(prob * log_prob).sum(dim=1).mean()
 entropy_loss = params["entropy_beta"] * entropy
 loss = loss_policy - entropy_loss

I know that a disadvantage of using policy gradient is our agent can be stuck at a local minimum. Can you explain mathematically why subtracting the entropy from our policy will  prevent our agent from being stuck in the local minimum ?

In other words, what author wanted to say is that entropy helps exploration. If you don't explore you might get suboptimal policy. Local minimum is an (overused) term to signify something suboptimal, in this case policy. Using entropy term will not prevent getting suboptimal policies but it might help getting better policies because you're exploring more.

@Brale_ It says "subtracting the entropy from our policy will prevent our agent from being stuck in the local minimum". So after the training we should always get the optimal policy or close. This is what I understand

Well no, if it was that easy to always get optimal policy just by adding entropy reinforcement learning would be solved. You're taking it too literally, it just means you're helping exploration by adding entropy.


text:
Do I need to maintain a separate population in each distributed environment when implementing PBT in a MARL context?
I have questions regarding on how to implement PBT as described in Algorithm 1 (on page 5) in the paper, Population Based Training of Neural Networks to train agents in a MARL (multi-agent reinforcement learning) environment.
In a single agent RL environment, the environments can be distributed & agents trained in parallel & there will be a need to maintain some sort of centralized population of weights & hyperparameters. (Please correct me if I'm wrong.)
In a MARL context, do I need to also maintain a centralized population for all agents in all environments or do I need to maintain a separate population for agents in each distributed environment? Which is a correct or more effective approach?
Any pointers would be appreciated. Thank you.


text:
Are there any novel quantum machine learning algorithms that are fundamentally different from "classical" ones?
Generally, if one googles "quantum machine learning" or anything similar the general gist of the results is that quantum computing will greatly speed up the learning process of our "classical" machine learning algorithms. However, "speed up" itself does not seem very appealing to me as the current leaps made in AI/ML are generally due to novel architectures or methods, not faster training.
Are there any quantum machine learning methods in development that are fundamentally different from "classical" methods? By this I mean that these methods are (almost*) impossible to perform on "classical" computers.
*except for simulation of the quantum computer of course

just a note: any quantum algorithm that is not "fundamentally different" from classical ones is totally useless. Quantum computing doesn't provide speed-ups due to faster processing, it does (if and when it does) due to its use of completely different ways of processing information. This said, I don't think there is any clear-cut answer to this question as of yet. You can find some links that might be of interest here https://quantumcomputing.stackexchange.com/a/1268/55

this question is also related: https://ai.stackexchange.com/q/36/4314

Generally, if one googles "quantum machine learning" or anything similar the general gist of the results is that quantum computing will greatly speed up the learning process of our "classical" machine learning algorithms.

This is correct. A lot of machine learning methods involve linear algebra, and it often takes far fewer quantum operations to do things in linear algebra than the number of classical operations that would be needed. To be more specific, for a matrix of size $N\times N$, if a classical computer needs $f(N)$ operations to do some linear algebra operation, such as diagonalization which can take $f(N)=\mathcal{O}(N^3)$ operations on a classical computer, a quantum computer would often need only $\log_2 f(N)$ operations, which inn (in and only in) the language of computational complexity theory, means exponential speed-up. The "only in" part is there because we have made here an assumption that "fewer quantum operations" means "speed-up", which for now is something we only know to be true in the world of "computational complexity theory".

However, "speed-up" itself does not seem very appealing to me as the current leaps made in AI/ML are generally due to novel architectures or methods, not faster training.

I disagree. Take vanilla deep learning for example (without GANs or any of the other things that came up in the last decade). Hinton and Bengio had been working on deep learning for decades, so why did interest in deep learning suddenly start growing so much from 2011-2014 after a roughly monotonic curve from 1988-2010? Not that this rise started before newer advances such as GANs and DenseNet were developed:

Notice also the similarity between the above graph and these ones:

These days pretty much everyone doing deep learning uses GPUs if they have access to GPUs, and what is possible to accomplish is extremely tied to what computing power a group has. I don't want to undermine the importance of new methods and new algorithms, but GPUs did play a big role for at least some areas of machine learning, such as deep learning.

Are there any quantum machine learning methods in development that are fundamentally different from "classical" methods?

I think you mean: "Most quantum machine learning algorithms are simply based on classical machine learning algorithms, but with some sub-routine sped-up by the QPU instead of a GPU -- are there any quantum algorithms that are not based on classical machine learning algorithms, and are entirely different".
The answer is yes, and more experts might be able to tell you more here.
One thing you might consider looking at is Quantum Boltzmann Machines.
Another things I'll mention is that a child prodigy named Ewin Tang who began university at age 14, discovered at around the age of 17 some classical algorithms that were inspired by quantum algorithms rather than the other way around, and
the comments on the Stack Exchange question Quantum machine learning after Ewin Tang might give you more insight on that. This is related to something called dequantization of quantum algorithms.

By this I mean that these methods are (almost*) impossible to perform on "classical" computers. *except for simulation of the quantum computer of course

Unfortunately quantum computers can't do anything that's impossible for classical computers to do, apart from the fact that they might be able to do some things faster. Classical computation is Turing complete, meaning anything that can be computed can be computed on a big enough classical computer.

@DuttaA That's a fair point. The huge amount of data is certainly important, and smartphones/gadgets did help with that, though internet and smartphones did exist in the 90s (PalmPilot) and 2000s (BlackBerry) too. Basically everyone doing serious deep learning is doing it on GPUs though. Why is that? GPUs were indeed a game changer for deep learning, and GPU development skyrocketed last decade. Also about big data: Deep Learning was a topic of Hinton's research since the 80s, and even without smartphones we still had enough data to do deep learning. But more data in the last decade did help!

Nice answer, but I disagree with your statement "Unfortunately quantum computers can't do anything that's impossible for classical computers to do". Quantum computers can do operations that are simply impossible for a classical computer to do (they can only be simulated). Turing complete unfortunately does not apply when comparing quantum and classical computers. So to rephrase my question: I'm interested if there is a (complete) quantum machine learning algorithm that can only be simulated by classical computers.

Unfortunately comments aren't saved the way answers are, so when my screen accidentally refreshed when I was almost done writing my ~400 character comment, what I wrote got permanently lost. In any case, what I wrote would have led to you asking further questions, and this might go for awhile, so I suggest you ping me at: https://chat.stackexchange.com/rooms/117114/hpqc-high-performance-quantum-computing where we can chat about this very intetesting question more smoothly.


text:
Algorithm which learns to select from proposed options
My goal is to write a program that automatically selects a routing out of multiple proposed options.
The data consists out of the multiple proposed options with each the attributes time, costs and if there is a transhipment and also which of the options was selected.
Example of data:

My idea at the moment is that I have to apply so type of inference to learn which attribute (time, costs, transhipment) has the highest impact on how to choose the best option. But I don't know exactly where to start with this.
Is there a "best" ML algorithm for this? Or how should I approach this?
The dataset currently consists out of 1000 samples in case if this is important.
Thanks in advance for your responses.

Some clarification needed: 1) Are the choices sequential i.e. choose one option leads to another state where the decision needs to be made again? Or is each choice a one-off, independent one? 2) Is the goal to select the best of 4 choices at any one time? If so, do you have a single consistent measure of "best" e.g. is it the lowest cost, or some combination of time, costs and "transshipment"? Or are you trying to modify something else e.g. you can control time, costs and transshipment and want to make your item the chosen one?

@NeilSlater 1) The choice is independent 2) It probably is a combination of all three, but some attributes might be weighted more than others. And no, I am not trying to modify time, costs and transshipments. Those attributes are given for each option. Please let me know if my question is still unclear. Thank you

Thanks for the clarification. If you are trying to discover the weightings, then you must have some other measure of overall success? If you compare two different results after selecting an option, how do you know which is better? Is your data a list of the best choice out of four for each set that happened in the past, or is it something else?


text:
How come a detection works after global average pooling 2D?
I use an off-the-shelf convolutional neural network, where at the end of the convolutional part, the depth of the last convolutional layer is expanded and then its 2D average is computed (such that for a tensor of say 8x8x512, you get its 2D average, which is of 1x512). It is a commonly used operation in deep networks, called Global Average Pooling 2D.
The only tensor that is input to the fully-connected part is that 2D averaged 1x512 tensor, i.e., a tensor that should not preserve the 2D information. Yet, my fully-connected last layer neurons, which have been trained to predict the 2D location of objects, work very well.
I thought about it for a long time and couldn't find any convincing explanation how come the network preserved the 2D information in the averaged tensor.
Any idea?

a good explanation can be found at https://stats.stackexchange.com/questions/257321/what-is-global-max-pooling-layer-and-what-is-its-advantage-over-maxpooling-layer.

@GerryP The explanation on your link explains how global-avgpool2d works, yet it doesn't answer how after applying this layer you can still perform 2D detection. They say the opposite actually ("more robust to translations")


text:
Does net with ReLU not learn when output < 0?
The derivative of ReLU is 0 if its output is lower than 0 -  $d ReLU(x)/dReLU$ is $0$ if $x < 0$. Let's denote some net's output by $Out$, so if this net's last layer is ReLU then we get that $dOut/dReLU$ is $0$ if $Out < 0$. Subsequently, for every parameter $p$ in the net we would get that $dOut/dp$ is $0$. Does that mean that for every sample $x$ such that $Out(x) < 0$ the net doesn't learn at all from that sample since the derivative for each parameter is $0$?

I have never seen the ReLU being used as the activation function of the last layer.

There is no benefit to using ReLU as the output activation of a neural network. As you said, the network will ignore training labels below zero and it will train on labels above zero as if no output activation were present. However, the problem you're describing can also occur for individual units of hidden layers, where ReLU activations are common. This is known as the dead ReLU problem. In practice, it is rarely a problem but it can be avoided with smooth rectifiers like ELU and Swish. Another interesting ideas is CRelu, which concatenates both positive and negative parts of the pre-activation, resulting in twice as many outputs, half of which always receive a non-zero gradient.


text:
How to work on different models for a given problem?
I am working on the MNIST data on my own. The idea is to use different values for the number of hidden layers, number of nodes in a given layer, etc. How do you organize these things while you are working on creating a model for a problem? DO you do everything in one code file or you use different code files for choosing the best?

Hi and welcome to AI SE! What do you mean by "code file"? Is this just a programming question? If it's just a programming question, it's off-topic here. See https://ai.stackexchange.com/help/on-topic for more details. Please, edit your post to clarify this.

it seems to me that you are talking about hyperparameter tuning and effect of hyperparameters on the network in general. If you are working with tensorflow, I recommend you to look into tensorboard.
Hands-on TensorBoard can be a good starting point.

I wanted to know how do you work with different models for a given problem. Say you have a model with 10 hidden layers and another with 7 hidden layers. How do you organize and set up different models that are meant for a project in your computer or cloud?

You can write function where you pass input parameter as number of layers and other hyperparameters and run that model.Then you can save that model which can be reused. It also depends what do you want to do with the model. What do you do with different models for same problem?

For a given classification problem, you check out a different number of hidden layers or you check out for a different number of nodes in each layer. I am using Jupyter Notebooks and it gets hard to do everything in just one .ipynb file. What can I do to organize my code for doing the above mentioned without making everything cumbersome. Thanks.


text:
Using AI to find the correct set of object/numbers based on previous data
There are 11 objects of which 4 are "Bad" objects. So there are 7 "Good" objects.
You have to choose as many Good objects before proceeding to another set of objects of a different sequence.
How would you train an AI that predicts the position of the Good objects based of previous sets of data?


text:
Class of functional equations that backpropagation can solve
There is a theorem that states that basically a neural network can approximate any function whatsoever. However, this does not mean that it can solve any equation. I have some notes where it states that backpropagation allows us to solve problems of the following kind
$$ F(x_i, t) = y_i $$
Can someone point me to what exactly this means?


text:
Building a spell check model
I have customer review texts. The data consists of the the raw and manually corrected texts of the reviews. I have aligned these pairs by using similarity algorithms and matched the words on them. Since there are some mis-matched words pairs, I have eliminated the pairs under a threshold value for their counts.
Now there are raw and corrected word pairs. What kind of machine learning model can I build for spellcheck by using the data mentioned as well?


text:
How do I approach this problem?
Let's say I have a dataset with multiple types of multiple ingredients (salt1,salt2, etc). Each n-th variation of each ingredient vs flavor may be represented by an n×k matrix that where an ingredient corresponds with a particular value of "flavor".
A recipe consists of a 1×n vector (where n is the number of ingredients) where each value corresponds to the quantity of ingredient in the recipe.
A particular combination of ingredients, with particular weights, with some transformation, would result in a particular 1×k "flavor" profile, in this simple model.
One approach could be to formulate this as a Probabilistic Matrix Factorization problem (I think), with k being the number of flavor parameters. And combining the recipe vector with the flavor matrix might do the trick.
But the problem is, the flavor value of each ingredient (and each variation of the ingredient) in the ingredient-flavor matrix would be very very limited. The recipe flavor profile might have a corresponding flavor vector, that too would be limited, and would not be available, at the beginning. So in order to capture the relationship between the ingredients and the flavor, the system would be dependent on user-submitted data on recipe/ingredient flavors.
Is there a way I could create clusters of recipes based on user flavor ratings and extrapolate these to the constituent ingredients or vice versa? Could this be done via some unsupervised learning algorithm?
I am quite new to this, I would appreciate some help or some pointers to which mathematical approaches I should be looking at to model this problem.


text:
Is it a good idea to overfit on a small part of your data for faster model convergence?
I working on a classification problem that needs to detect patterns on a time serie. Basically, there's a catch-all class that means "no pattern detected", the other are for the specific patterns. The data is imbalanced (ratio 1/10 at least), but I adapted the class weights.
I'm able to overfit successfully on a few days of data, but when I train on 2 years of data, the model seems stuck on class1 "no pattern detected" for a veeeery long time. I've tried several  learning rates, but it doesn't make the convergence happen significatively faster.
Is it a better starting point for my training to use the overfitting model's weight as a starting point? Could this allow the model to converge faster?

The first thing which you first have to understand is that does your trained model is working efficiently with both the training and testing data. If yes then its not overfitting. There is only one case where over-fitting doesn't matter and that is when your testing data is same as training data and dealing with is very important or it could be disastrous. Therefore, to avoid overfitting you can try several methods and they are followings:

Optimization Algorithms like sgd, Adam, RMSprop, Adagard, are few gradient descents.
Loss function like Hinge loss could be useful.
Parameter initialization greatly influence the optimization process.

Overall, Large datasets are better for building ML models but there are some methods there is an interesting article that i would recommend you to read: Article Link
I Hope this Helps!!!

Thx for your answer, but I think that you didn't understand the question. My problem here is not to avoid overfitting, I'm searching for a way to make my model train faster.

@Sebastien If you are really looking for a faster training model you have to include the coding, datasets, and modules. Looking forward to help!

I'm not willing to include everything (this may become more complicated than my simple question. I just want to know if sometime, doing 1) train model on small data until overfit  then 2) train model on all data can be faster than just 2)

@Sebastien Droping small pieces of data until overfitting will converge faster in training data. And you really focusing on convergence then check article too link


text:
Automating browser actions using AI
I am at a very initial stage of my research so I will try to describe what I am trying to achieve:
I want to create an AI model which learns how to navigate the browser's component like clicking or creating new favorites and tabs, or navigating browser action menu, or bookmarking the website etc. In short, automating the testing for browser using selenium and an AI model so that over time the model learns itself to navigate the browser and test different functionality itself and eventually it test the functionalities that are not seen by the model before. For example: if I feed the AI model how browser is closed when "x" is clicked and minimize when "-" is clicked, next it can learn itself how to maximize the browser.
The initial input could be to record some videos of navigating the browser using selenium and then feed it to the model and with time it learns itself to go to different section of the browser which the model does not know and still test it.
Is it even possible to combine AI and selenium together to create something like this? If yes, how can I achieve it and what is the best approach to develop such model.
Thanks in advance.


text:
What are the best datasets available for music information retrieval?
I am interested in doing some work in classification problems in music information retrieval. I know that there are some formats of datasets (such as MIDI, Spectrogram, Piano-roll, MusicXML, etc.) for this work but have been unable to find any nice large datasets for this. What are the best free datasets for such work? I am mainly looking into western classical music.


text:
Backpropagation of neural nets with shared weight
I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. However, I am stuck at some point.
Consider a simple fully connected neural network with two hidden layers. For simplicity, choose linear activation function (${f(x) = x}$) at all layer. Now consider that this neural network takes two $n$-dimensional inputs $X^{1}$ and $X^{2}$. However, the first hidden layer only takes $X^1$ as the input and produces the output of $H^1$. The second hidden layer takes $H^{1} $and $X^2$ as the input and produces the output $H^{2}$. The output layer takes $H^{2}$ as the input and produces the output $\hat{Y}$. For simplicity, assume, we do not have any bias.
So, we can write that, $H^1 = W^{x1}X^{1}$
$H^2 = W^{h}H1 + W^{x2}X^{2} = W^{h}W^{x1}X^{1} + W^{x2}X^{2}$ [substituting the value of $H^1$]
$\hat{Y} = W^{y}H^2$
Here, $W^{x1}$, $W^{x2}$, $W^{h}$ and $W^{y}$ are the weight matrix. Now, to make it more interesting, consider a sharing weight matrix $W^{x} = W^{x1} = W^{x2}$, which leads, $H^1 = W^{x}X^{1}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$
I do not have any problem to do forward propagation by my hand; however, the problem arises when I tried to make backward propagation and update the $W^{x}$.
$\frac{\partial loss}{\partial W^{x}} = \frac{\partial loss}{\partial H^{2}} . \frac{\partial H^{2}}{\partial W^{x}}$
Substituting, $\frac{\partial loss}{\partial H^{2}} = \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$
$\frac{\partial loss}{\partial W^{x}}= \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y} . \frac{\partial}{\partial W^{x}} (W^{h}W^{x}X^{1} + W^{x}X^{2})$
Here I understand that, $\frac{\partial Y}{\partial H^{2}} = (W^y)^T$ and $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$ and we can also calculate $\frac{\partial Y}{\partial H^{2}}$, if we know the loss function. But how do we calculate $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$?

If we write $ H^2 = W^{h}H1 + W^{x}X^{2} $ then it will be better to understand the backward propagation step.
Now,
$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$ can be written as:
$\frac{\partial H^2}{\partial H^1}\frac{\partial H^1}{\partial W^{x}} $
$\frac{\partial H^2}{\partial H^1} =  (W^h)^T$ and
$\frac{\partial H^1}{\partial W^{x}} = (X^{1})^T $
Therefore,
$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = (W^h)^T(X^{1})^T  $
I hope it has solved your problem.

So, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$?

Can you please tell me, in this expression $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$, why have you taken transpose of $(X^2)^T$  as derivative. Why is it not only $(X^2)$

I think your notations are unclear, but I can give an answer based on what you probably meant. For example, $\frac{\partial{L}}{\partial{W^x}}$ should be replaced by $(\nabla_{W^x_{j:}}L)_{j=1, ...,n}$ (assuming everything stays in $\mathbb{R}^n$). Also your expression for $\frac{\partial{L}}{\partial{W^x}}$ is wrong, even accounting for the notation.
Since $W^x_{j:}$ affects the loss through $H_{1,j}$ and $H_{2,j}$, it would be better to treat the math in this way:
$$\nabla_{W^x_{j:}}L=\frac{\partial{L}}{\partial{H_{1,j}}}\nabla_{W^x_{j:}}H_{1,j}+\frac{\partial{L}}{\partial{H_{2,j}}}\nabla_{W^x_{j:}}H_{2,j}$$
Now, $H_{1, j}$ affects the loss though $H_{2,k}\ \forall\ k=1,...,n.$ So,
$$\frac{\partial{L}}{\partial{H_{1,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{H_{2,k}}}W^x_{kj}$$
And,
$$\frac{\partial{L}}{\partial{H_{2,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{Y_{k}}}W^y_{kj}$$
Similarly, $\nabla_YL$ can be computed.

The product rule of partial derivative:
$\frac{\partial}{\partial x} f g = g \frac{\partial}{\partial x} f + f \frac{\partial}{\partial x} g$
According to this:  $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = W^{h}X^{1}$, because derivative of other term with respect to $W^{x}$ is zero. (I am not considering the transpose notation as it depends on how you organize your data.)
However, Your assumption of giving $H^{1}$ and $X^{2}$ as input to second hidden layer is not valid(they are called hidden layer for that reason). The output of first hidden layer ($H^{1}$)  will be fed to the input of second hidden layer.
Your output of second hidden layer would be $H^{2} = W^{h}  * H^{1}$.
You have to fed your input $X^{1} X^{2}$ to your network at once by means of looping or vectorization.

Firstly, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$?
Secondly, think about an Elman RNN network. you can think $X^1$ as input at time t=0 and $X^2$ as input at time t=1. Then the $\hat(Y)$ is the output from the time step t=1

My ans is saying what will be derivative of your asked partial derivative equation. The dimension mismatch is happening because the equation you are getting from forward pass is probably wrong. $H^{2} = W^{h} W^{x} X^{1} + W^{x} X^{2}$ is not right i guess.
I don't know much about Elman RNN.  My ans is based on simple fully connected N.N as you have mentioned in the question.
** adding a diagram would make people better understand your query. Thanks.


text:
How do i start building an autoclick bot for pubg mobile?
I want to make a bot which clicks the fire button on the mobile screen upon seeing an enemies head.
In pubg mobile which is an android game you have to control the fire button and the aim along with many other controls to kill an enemy or other players. I want to automate the fire button and everything else would be controlled by me, when I'll aim on a player's head the bot should click the fire button instantly.
So there are a few problems in this ,first one is that what if it shoots upon seeing my teammates head and other is what if there are  two players at once ,which will it shoot, for that it needs to shoot when my aim or the red crosshair is on the player's head.
I don't know how to get started , I need to make an image recognition app and an autoclicker and combine them both. How do I get started? Assuming that I only know basic python.

The problem is score kinda thing. Rest you can rely on a very high performance computer with Deep RL

I seen on some videos, people will sellotape a cross hair to the middle of there screen and use this instead as the raycast / the ejecting bullet always comes from the center in some games, for this game I am not sure, but I think a good initial way to go first would be narrowing down the problem to its simplest form, how can you make shooting more effective and efficient as a human player? So think in the game, when you see someones head on the screen, are all players the same height for example thus same Y Axis?


text:
Adversarial Q Learning should use the same Q Table?
I'm creating a RF Q-Learning agent for a two player fully-observable board game and wondered, if I was to train the Q Table using adversarial training, should I let both 'players' use, and update, the same Q Table? Or would this lead to issues?

should I let both 'players' use, and update, the same Q Table?

Yes this works well for zero-sum games, when player 1 wants to maximise a result (often just +1 for "player 1 wins") and player 2 wants to minimise a result (score -1 for "player 2 wins"). That alters algorithms such as Q-learning because the greedy choice switches beween min and max functions over action values - player 1's TD target becomes $R_{t+1} + \gamma \text{min}_{a'} [Q(S_{t+1}, a')]$ because the greedy choice in the next state is taken by player 2.
Alternatively, if the states never overlap between players, then you could learn a Q function that always returns the future expected return according the current player. This can be forced if necessary by making part of the state record whose turn it is. With this you need a way to convert between player 1 and player 2 scores in order to use Q learning updates. For zero-sum games, then the Q value from a player 2 state is negative of player 1's value for that same state, and vice versa, so the TD target changes for both players to $R_{t+1} - \gamma \text{max}_{a'} [Q(S_{t+1}, a')]$
The first option can result in slightly less complexity in the learned function, that might be an issue if you are using function approximation and learning a Q function e.g. using neural networks instead of a Q table. That may result in faster learning and generalisation, although it will depend on details of the game.

Or would this lead to issues?

No major issues. I am performing this kind of training - a single Q function estimating a global score which P1 maximises and P2 minimises - for the Kaggle Connect X competition, and it works well.
I can think of a couple of minor things:

You may still want to have the ability for each player to be using a different version of the table or learned Q function. This would allow you to have different versions of your agent (e.g. at different stages of learning) compete, to evaluate different agents against each other. To do this, you have to write code that allows for multiple tables or functions in any case.
You need to keep track of how both players express and achieve their opposing goals when using the table, as you can already see by the modified TD targets above. This becomes more important when adding look-ahead planning, which is a common addition and can significantly improve the performance of an agent - during look-ahead you must switch between player views on what the best action choice is. It is possible to make off-by-one errors in some part of the code but not others and have the agent learn inconsistently.


text:
What if the rewards induced by an environment are related to the policy too?
Assume we have a policy $\pi_{\theta}$ in a classic reinforcement learning setting, and a reward function $R^{\pi}(s,a)$ that changes as long as $\pi$ changes i.e. not only is it predefined by the environment itself, how can we model the popular algorithms (e.g. SAC) according to this change?

That would be an odd formulation. Could you give a motivation, or an example of how immediate reward would interact with the policy? What version of policy are you considering $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ or $\pi(a|s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$? Is this potentially a game theory question about modelling adversarial or multi-agent environments?


text:
How to use 'Canny/Watershed' algorithm's output as an input for Image Classification Model
I have a very silly problem in hand. I have implemented 2 methods which give me the mask to separate the objects from the background. What I get from one method is the object encapsulated in the red Contour or boundary and the other one makes the background Red.
I am using Keras to classify Trash. I wanted to use the output or the EXTRACTED object as an input to the CNN model. Now I do not see any difference in the images and output. All there is extra boundary around the object and I fail to understand how it can help my model.
I could add an extra alpha channel to second method to make the background transparent but Keras' ImageDataGenerator do not work with RGBA images. What shoul dI do to improve the model?

By 'classify trash', do you mean a binary classifier which gives a boolean output for a given image (trash in image: true/false), or are you attempting to segment the trash in the image (a binary mask indicating which pixels represent trash in the image)

I want to extract the object from the image of garbage and then predict what kind of waste it is (paper,metal,glass,cardboard etc) using a Keras model.  There is just one object per image.

First and foremost, I have to say that this could (and likely will) be a very hard task. Neural networks (NNs) have excelled at computer vision tasks identifying everything from textures to complex objects but what you are trying to do goes beyond that. We (humans) identify trash using the context as much as the object. An object on a table and the same object in a trash bin can look identical but the context tells us which one is trash. Also, anything can be trash. Trash is not an object, its a state.
Having said all that, it sounds like an interesting project and it would be a very useful model so I'll do my best to explain how you would go about trying this. As per your comment, you have outlined the two steps you need.

Identify and extract the trash object from the image.
Classify the type of waste

This could be accomplished using a single model but I'll explain how to do it with two models to make clear what is being done.
Identify and extract trash
The objective of the first model is to identify and extract the region of interest (the part of the image containing the trash). This is an image segmentation task typically accomplished using an R-CNN - some guides explaining how they work can be found here, here or here. These methods use supervised learning which requires masks delineating the object of interest to be used as the ground truth. A mask is a binary image the same size as your input image where each pixel in the image representing your positive class (trash) is set to 1 and all others are set to 0.
The output of your Canny/Watershed algorithms could be used as the masks to train the model, however your model will only ever be as good as your masks. Therefore, you might as well use your Canny/Watershed algorithms to accomplish task 1. If the masks generated by your algorithm are not of sufficient accuracy you will need to find another way to generate your masks - maybe even doing it manually.
An approximate rule of thumb for object detection with NNs is that you need 1,000 representative images per class, where class implies a specific object. In this case, unless you are attempting to identify very specific items of trash, you would likely need 100s of thousands of images to obtain a high degree of accuracy.
Classify the type of waste
This task should be easier, making the assumption that the segmented image from the first step is always trash. By assigning a label to each segmented image (paper, metal, glass, cardboard, etc) it becomes a normal multi-class classification task which are well documented online with lots of explainations and tutorials.
Single model
These tasks could be combined into a single model by modifying the masks in step 1. Instead of using a single binary mask, you could create a multichannel mask of shape m x n x o where m x n is your input image size and o is the number of kinds of waste you are attempting to identify. Each channel is a binary mask for a given type of waste. Therefore, it becomes a matter of not just identifying and segmenting trash but of segmenting each type of trash separately. Needless to say the complexity of the model would be a lot higher and so would the process to create the masks.


text:
How can transition models in RL be trained adversarially?
To give a little background, I've been reading the COBRA paper, and I've reached the section that talks about the exploration policy, in particular. We figure that a uniformly random policy won't do us any good, since the action space is sparsely populated with objects that the agent must act upon - and a random action is likely to result in no change (an object here occupies only about 1.7% of the space of the screen). Hence we need our agent to learn in the exploration phase a policy that clicks on and moves objects more frequently.
I get that a random policy won't work, but I've difficulty understanding how and why the transition model is trained adversarially. Following is the extract which talks about the same, and I've highlighted parts that I don't completely understand -

"Our approach is to train the transition model adversarially with an exploration policy that learns to take actions on which the transition model has a high error. Such difficult-to-predict actions should be those that move objects (given that others would leave the scene unchanged). In this way the exploration policy and the transition model learn together in a virtuous cycle. This approach is a form of curiosity-driven exploration, as previously described in both the psychology (Gopnik et al., 1999) and the reinforcement learning literature (Pathak et al., 2017; Schmidhuber, 1990a,b)."

How does it help to take actions on which the transition model has a
high error?
I don't exactly see how a virtuous cycle is in action

Could someone please explain? Thanks a lot!


text:
How can DDPG handle the discrete action space?
I am wondering how can DDPG or DPG handle the discrete action space. There are some papers saying that use Gumbel softmax with DDPG can make the discrete action problem be solved. However, will the Gumbel softmax make the deterministic policy be the stochastic one? If not, how can that be achieved?


text:
MNIST Classification code performing with 88%-90% whereas other codes online perform 95% on first epoch
I have been trying to write code to implement plain neural net without convolution from scratch. I took some help online here and added my code to my github account.
I don't understand why the prediction made by my code is only 88%-90% accurate after the 1st epoch, whereas his code is 95% accurate after 1st epoch with the same parameters (Same Xavier initialization for weights, biases are not initialized, same hidden layer neurons). While his architecture uses 2 hidden layers, my code performed worse with 2 hidden layers. For 1 hidden layer, his code performs similar (~96%).

Different starting weights or ordering of the training data might make a difference in the first epoch. Do the two programs perform similarly after they've been given a chance to converge? And do you see similar differences in performance when you rerun your code with a different random seed?

No my program only has around 90-92% accuracy even after 30 epochs (with 200 neurons in hidden layer), and performs similarly on different seeds. It only reaches 93% on first epoch when i increase hidden layer neurons to 1000.

Ok I found the mistake. The weights and biases of the input to hidden layer were not getting updated.


text:
Does Q Learning learn from an opponent playing random moves?
I've created a Q Learning algorithm to play Connect Four against an opponent who just chooses a random free column. My Q Agent is currently only winning about 0.49 games on average (30,000 episodes). Will my Q Agent actually learn from these episodes, seeing as its opponent isn't 'trying' to beat it, as there's no strategy behind its random choices? Or should this not matter – if the Q Agent is playing enough games, it doesn't matter how good/bad its opponent is?

Is it by any chance for this competition: https://www.kaggle.com/c/connectx ? If not, then you may find quite a lot of advice on the forums and notebooks there.

It's not for that but I'll take a look there. Thanks.

It should be possible to train an agent using some variant of DQN to beat a random agent around 100% of the time within a few thousand games.
It may require one or two more advanced techniques to get the learning time down to a low number of thousands. However, if your agent is winning ~50% of games against a random agent, something has gone wrong, since that is the performance you would expect of another random agent. Even simple policies, such as always play in same column, will beat a random agent a significant fraction of the time.
First thing to consider is that there are too many states in Connect 4 to use tabular Q learning. You have to use some variant of DQN. As a grid-based board game where winning patterns can repeat, some form of convolutional neural network (CNN) for the Q function is probably a good start.
I think for a first step, you should double-check that you have implemented DQN correctly. Check the TD target formula is correct, and that you have implemented experience replay. Ideally you will also have a delayed-update target network for calculating the TD targets.
As a second step, try some variations of hyper-parameters. The learning rate, exploration rate, size of replay table, number of games to play before starting learning etc. A discount factor $\gamma$ slightly below 1 can help (despite this being an episodic problem) - it makes the agent forget more of the initial bias for early time steps.

Or should this not matter – if the Q Agent is playing enough games, it doesn't matter how good/bad its opponent is?

Up to a point this is true. It is hard to learn against a perfect agent in Connect 4, because it always wins as player one, which means all policies are equally good and there is nothing to learn. Other than that, if there is a way to win, eventually a Q learning agent with exploration should find it.
Against a random agent, you should be seeing some improvement if your agent is correctly set up for the problem, after a few thousand games. As it happens I am currently training Connect 4 agents using variants of DQN for a Kaggle competition, and they consistently beat random agents with 100% measured success rate after 10,000 training games. I have added a few extras to my agents in order to achieve this - there are some discussions of approaches in the forums at https://www.kaggle.com/c/connectx

Thanks, Neil. I'm using 'vanilla' Q Learning and no NNs. After 100,000 episodes, my agent is winning 0.86 games on average – though my Q Table file is now over 150mb!

@mason7663: That's not too bad a result for a tabular approach. Yes the problem will be  that it is impossible to store all possible states, or even visit them in a single lifetime using a simple approach. I don't think you will hit 100% success rate against a random agent if you continue - although I guess you may beat 90% or even 95% before you start to run out of RAM for your table.


text:
Is Q-Learning suitable for time-dependent spaces?
Many Q-learning techniques have been developed to capture discrete state(observation), actions like a robot in a grid world, and even continuous (state or action) spaces. But I am wondering how we can model the states/space in a time-dependent environment. Please, consider the following example:
There is one smartphone (client) and five compute servers that are addressing/serving many clients (smartphones) at the same time. The smartphone transfers some raw data (e.g, sensor data) to one of those five servers (e.g., every t seconds) and gets the results. Suppose the server computes the stress-level of the client in real-time based on the collected data.
Now, a q-learning agent should be deployed to the smartphone to be able to select the best server with minimum response time (i.e., the goal is to minimize the execution/response time). Note that servers are serving different clients and their load is a function of time and varies from time to time.
So in the above scenario, I am wondering what would be our "states" and how we can model the "environment"?

Does the client that needs to make a decision know the current load on the server, or have state data that would allow it to approximate it? Is it allowed to have state such as response statistics for its last few requests, or some statistical data from the servers for past equivalent time periods?

Architecturally, in the real world, it is more normal to have server allocation managed inside a cluster by a load balancer server rather than by clients, since the service will be able to measure and compare load/responses across multiple services whilst the clients cannot do this. If you are trying to solve a real world problem to create a stabel and fast service, you should probably consider this approach first as opposed to AI at the client, but the AI issue is likely interesting and on-topic here regardless of that.

Well, the smartphone can log and store its own last few requests.

Regarding load balancer: yes that is correct. I just simplified the problem. I am working on an offloading technique for IoT, in which the and edge device is connected to a fog node, and the fog node is connected to the cloud (with several servers). Now the edge should decide to run the task itself, or offload the task to the fog nodes, or to the cloud.

According to the system model, three different sites are considered
as a feasible platform for offloading computation (1) the
mobile fog in close proximity of end-user devices, i.e. site L1 (2)
the adjacent mobile Fog (or distant mobile Fog) to handle mobility
and load balancing issues, i.e. site L2 (3) the remote public cloud to
manage huge traffic and computing requirements and archiving,
i.e. site L3. So the actions are: 1) run in the device,, offload to L1, offload to L2, or offload to L3.

it would be great if we could have multiple agents (collaborating with each other) e.g, one agent in the IoT device, one agent in each fog nodes, and one agent in the cloud. so we could offer more actions such as migration within L1, migration from L1 to L2, etc.

but let's focus on the simplest case which I explained in my first question.


text:
PPO algorithm converges on only one action
I have taken some reference implementations of PPO algorithm and am trying to create an agent which can play space invaders . Unfortunately from the 2nd trial onwards (after training the actor and critic N Networks for the first time) , the probability distribution of the actions converges on only action and the PPO loss and the critic loss converges on only one value.
Wanted to understand the probable reasons why this might occur . I really cant run the code in my cloud VMs without being sure that I am not missing anything as the VMs are very costly to use . I would appreciate any help or advice in this regarding .. if required I can post the code as well . Hyperparameters used are as follows :
clipping_val = 0.2 critic_discount = 0.5 entropy_beta = 0.01 gamma = 0.99 lambda = 0.95
code repo : github.com/superchiku/ReinforcementLearning .


text:
Backward pass of CNN like Resnet: how to manually compute flops during backprop?
I've been trying to figure out how to compute the number of Flops in backward pass of ResNet. For forward pass, it seems straightforward: apply the conv filters to the input for each layer. But how does one do the Flops counts for gradient computation and update of all weights during the backward pass?
Specifically,

how to compute Flops in gradient computations for each layer?
what all gradients need to be computed so Flops for each of those can be counted?
How many Flops in computation of gradient for Pool, BatchNorm, and Relu layers?

I understand the chain rule for gradient computation, but having a hard time formulating how it'd apply to weight filters in conv layers of ResNet and how many Flops each of those would take. It'd be very useful to get any comments about method to compute total Flops for Backward pass. Thanks

Hi and welcome to AI SE! Please, ask only one question. You're asking multiple questions. I suggest you edit your post to ask a single question and please focus on the smallest problem you need to solve before going to the next one.

It's actually just 1 question, how to get total op count without coding in backprop for a layer. I didn't get any answer so I thought I should at least lay the whole issue out for anyone to comment.


text:
How to evaluate the performance of an autoencoder trained on image data?
I am training an autoencoder on (general) image data.
I use binary crossentropy loss function, but it is not very informative when I want to evaluate the performance of my autoencoder.
An obvious performance metric would be pixel-wise MSE, but it has its own downsides, shown on some toy examples in an image from paper from Pihlgren et al.

In the same paper, the authors suggest using perceptual loss, but it seems complicated and not well-studied.
I found some other instances of this question, but there doesn't seem to be a concensus.
I understand that it depends on the application, but I want to know if there are some general guidelines as to which performance metric to use when training autoencoders on image data.

I will answer my own question to try and provide some insights.
My research supervisor suggested that I should use the SSIM metric or some other well-known image processing metric (see the book "Modern Image Quality Assessment" by Wang and Bovik) for assessing the visual similarity of an images.
Another way I evaluate the performance of an autoencoder is by simply visually comparing the input and output images taken from the test set. This is by no means very scientific, but it gives a good idea whether an autoencoder is able to reconstruct the input images. One thing I would add here is that even if an autoencoder can reconstruct images perfectly, it doesn't mean that the encoding it learned is useful. For example, when I wanted similar images to be mapped to similar encodings, the autoencoder that was able to do that better was outputting more blurred reconstructed images in comparison to the autoencoder that wasn't achieving this similarity preservation (but was outputting better reconstructions).


text:
State-of-the-art algorithms not working on a custom RL environment
I'm trying to train a RL agent on a custom, highly stochastic environment (MDP). In order to do so I'm using existing implementations of state-of-the-art RL algorithms as provided by Stable Baselines. However, no matter what algorithm I try out and despite extensive hyperparameter tuning, I'm failing to obtain any meaningful result. More precisely, a trivial "always perform the same action (0.7,0.7) each time" strategy works better than any of the obtained policies. The environment is highly stochastic (model of a financial market). How likely is it that the environment is simply "too stochastic" for any meaningful learning to take place? If interested, here's the environment code:
class environment1(gym.Env):
def __init__(self):
    self.t = 0.0 # initial time
    self.s = 100.0 # initial midprice value
    self.T = 1.0 # trading period length
    self.sigma = 2 # volatility constant
    self.dt = 0.005 # time step
    self.q = 0.0 # initial inventory
    self.oldq = 0 # initial old inventory
    self.x = 0 # initial wealth/cash
    self.gamma = 0.1 # risk aversion parameter
    self.k = 1.5 # intensity of arivals of orders
    self.A = 140 # constant
    self.done = False
    self.info = []
    high = np.array([np.finfo(np.float32).max,
                     np.finfo(np.float32).max,
                     np.finfo(np.float32).max],
                    dtype=np.float32)
    self.action_space = spaces.Discrete(100)
    self.observation_space = spaces.Box(-high, high, dtype=np.float32)
    self.seed()
    self.state = None

def seed(self, seed=None):
    self.np_random, seed = seeding.np_random(seed)
    return [seed]

def step(self, action):
    old_x, old_q, old_s = self.x, self.q, self.s # current becomes old
    self.t += 0.005 # time increment
    P1 = self.dt*self.A*np.exp(-self.k*(action//10)/10) # probabilities of execution
    P2 = self.dt*self.A*np.exp(-self.k*(action%10)/10)
    if random.random() < P1: # decrease inventory increase cash
        self.q -= 1
        self.x += self.s + (action//10)/10
    if random.random() < P2: # increase inventory decrease cash
        self.q += 1
        self.x -= self.s - (action%10)/10
    if random.random() < 0.5:
        self.s += np.sqrt(0.005)*self.sigma
    else:
        self.s -= np.sqrt(0.005)*self.sigma
    self.state = np.array([self.s-100,(self.q-34)/25,(self.t-0.5)/0.29011491975882037])
    reward = self.x+self.q*self.s-(self.oldx+self.oldq*self.olds)
    if np.isclose(self.t, self.T):
        self.done = True
    self.oldq = self.q
    self.oldx = self.x
    self.olds = self.s
    return self.state, reward, self.done, {}

def reset(self):
    self.t = 0.0 # initial time
    self.s = 100.0 # initial midprice value
    self.T = 1.0 # trading period length
    self.sigma = 2 # volatility constant
    self.dt = 0.005 # time step
    self.q = 0.0 # initial inventory
    self.oldq = 0.0
    self.oldx = 0.0
    self.olds = 100.0
    self.x = 0.0 # initial wealth/cash
    self.gamma = 0.1 # risk aversion parameter
    self.k = 1.5 # intensity of arivals of orders
    self.A = 140 # constant
    self.done = False
    self.info = []
    self.state = np.array([self.s-100,(self.q-34)/25,(self.t-0.5)/0.29011491975882037])
    return self.state

The state space is mostly normalized. The action space consists of 100 possible discrete actions (integers from 0 to 99 which are then transformed to (0.0,0.0),(0.0,0.1),...(1.0,1.0). The reward is simply given by the change in the portfolio value (cash+stock).
Note: I've also tried transforming the action space into a continuous one in order to use DDPG but all to no avail.

If the environment is too stochastic, then this is the best meaningful result you can get. Else, you may prefer changing the definition of state/action to emulate some particular environment the best in order to hope some luck.


text:
Q table not converging for an arbitrary experiment
This is an experiment in order to understand the working of Q table and Q learning.
I have the states as
states = [0,1,2,3]
I have an arbitrary value for each of these states as shown below (assume index-based mapping) -
arbitrary_values_for_states = [39.9,47.52,32.92,37.6]
I want to find the minimum of the state which will give me the minimum value.
So I have complimented the values to 50-arbitrary value.
inverse_values_for_states = [50-x for x in arbitrary_values_for_states]
Therefore, I defined reward function as -
def reward(s,a,s_dash):
    if inverse_values_for_states[s]<inverse_values_for_states[s_dash]:
        return 1
    elif inverse_values_for_states[s]>inverse_values_for_states[s_dash]:
        return -1
    else:
        return 0

Q table is initialized as -
Q = np.zeros((4,4)) (np is numpy)
The learning is carried out as -
episodes = 5
steps = 10
for episode in range(episodes):
    s = np.random.randint(0,4)
    alpha0 = 0.05
    decay = 0.005
    gamma = 0.6
    for step in range(steps):
        a = np.random.randint(0,4)
        action.append(a)
        s_dash = a
        alpha = alpha0/(1+step*decay)
        Q[s][a] = (1-alpha)*Q[s][a]+alpha*(reward(s,a,s_dash)+gamma*np.max(Q[s_dash]))

        s = s_dash

The problem is, the table doesn't converge.
Example. For the above scenario -
np.argmax(Q[0]) gives 3
np.argmax(Q[1]) gives 2
np.argmax(Q[2]) gives 2
np.argmax(Q[3]) gives 2

All of the states should give argmax as 2 (which is actually the index[state] of the minimum value).
Another example,
when I increase steps to 1000 and episodes to 50,
np.argmax(Q[0]) gives 3
np.argmax(Q[1]) gives 0
np.argmax(Q[2]) gives 1
np.argmax(Q[3]) gives 2

More, steps and episodes should assure convergence, but this is not visible.
I need help where I am going wrong.
PS: This little experiment is needed to make Q-learning applicable to a larger combinatorial problem. Unless I understand this, I don't think I will be able to do that right.
Also, there is no terminal state because this is an optimization problem. (And I have heard that Q-learning doesn't necessarily needs a terminal state)

for episode in range(episodes):
doesn't it should be for steps in range(episodes) ? Sorry, I'm a C++ guy, it's a bit daunting to me... :P

@abhas_RewCie range() is like a vector of integers. episodes specify the stop value for generating the integers. episode is like a candidate element (i) of that vector. So, it will loosely translate as for(int episode=0; episode<episodes.size();episode++)

if you don't mind writing the algorithm....

If your intention is to learn make the agent learn which has the min arbitrary value, then you would need to modify your rewards a bit.
The current reward structure provides the incentive to just move to a stage where it gets a reward.
For example, if it is at state 0, it gets the same reward to go to either state 2 or state 3, since both of them have a higher inverse value.
To make the agent learn to move to state 2, you would have to provide it with more incentives to go to state 2.
def reward(s,a,s_dash):
    if s_dash == 2:
        return 5
    elif inverse_values_for_states[s]<inverse_values_for_states[s_dash]:
        return 1
    elif inverse_values_for_states[s]>inverse_values_for_states[s_dash]:
        return -1
    else:
        return 0

I tried using this and it converges to 2. This is a hard-coded version, but I guess you get the idea.

It's worse than you suggest. The OP's reward structure encourages the smallest possible incremental improvement on each action, because then it is possible for it to receive maximum rewards before finishing. It is better to score $[1,1,1,-1,1,1,1,-1....]$ starting from the worst state than score $[1,0,0...]$ and this is shown by the OP's converged values.


text:
How can raw data from a motion sensor (like an IMU) reduced to the main points of the data
How can I reduce the caputured movement data of a person in a way, that I have filtered the main features of the movement. Or how can I detect pattern/ main features in that data set?
I captured some data with a device (Inertial sensor and motion capturing (x,y,z)) attached to human body. So I have a huge data set. I want to prepare the data set so I have no noise and no "unecessary" data.
In the end I just want to know for example that certain movement behaviour hints to a clueless guy or a person under stress.
I first thought approaches from association analysis could be useful but I think based on what I have read about the application of such algorithms they are not suitable for this data set.


text:
What is 3D face recognition? and how we can check liveness of a face image?
Actually what is mean by 3D face recognition? In normal cases we are extracting face encoding s from a 2D image,right?
Is 3D face recognition is used for liveness detection? how its possible?

This needs a reference. It could mean RGBD (where D is depth), or stereoscopic, or constructing 3D from muliple images (video) or something based around 3D model for facial features, or even just guessing an identify from other than simple frontal views. Or something else entirely. Where have you seen the mention of "3D face recognition"?

Is it an object or face or an image?

1. What is typically meant by 3D-face recognition? We are usually extracting the face encoding from 2D-images, right?
Yes. The goal is to reconstruct the three-dimensional shape, as well as the texture of a face from a single or multiple images of that person.
In recent years, "the performance of 2D face recognition algorithms has significantly increased with the use of deep neural networks and the use of large-scale labeled training data." Deep 3D Face Ifentification with the latter still being a notable problem because there are so few of them and they are usually needed to better measure and hence train the deep networks. (The referenced paper gives you a nice overview.) Even with enough data, occlusions, lens distortions or possible variations of expressions, etc. make the problem tough. So many other tricks and augmentation techniques are needed.
2. How can 3D face recognition used for liveness detection?
3D face recognition is used to differentiate 2D spoofs from actual present people. Using multiple time-delayed images one could possibly track if the face moves like a living person as well, or if it is static or stretching weird like a mask, puppet, or moving display.
Nonetheless, often even advanced 2D image-based techniques fall flat in real-world applications because of the used cameras. They might not have enough resolution to recognize if the input is a video of a face on a high-resolution display as explained here.


text:
How does heuristic work with multiple agents?
I have a question for heuristic search with multiple agents. I know how heuristic search works with one agent (ex. one Pacman) but I don't really understand it with multiple agents. Let's say we have this problem where Worm A has to get to its goal state A and Worm B to B, knowing that the agents can move only in vertical and horizontal way:

If we had only Worm B, the optimal cost from starting position to the goal position would be 9, since one action costs 1 and it'd follow the path RIGHT-RIGHT-RIGHT-RIGHT-RIGHT-RIGHT-UP-UP-UP.
My question is, if we have two worms, like in the picture, the optimal cost would be 9 + optimal cost for Worm A?
Also, strictly for this problem with 2 agents, if we use Manhattan distance as a heuristic for one agent, would it be admissible if we take the average of Worm A and B heuristics for a problem with two agents?
Another question, I know for a fact that sum of two admissible heuristics won't be admissible for one agent but would it be for the problem with two agents?
These two worms are dependent of each other. How? If one worm moves from position X to Y, the position X is marked as a wall and is not an available field to move in. So if one worm has been in a specific position, that position is no more free for moving in.
For example, if we have something like B^^^X^^^, where B is the Worm B, ^ is an available field and X is a wall, after one RIGHT action it'll look like XB^^X^^, after one more RIGHT: XXB^X^^ etc.

Your questions are interesting, but, please, *ask one question per post, even though they are somehow related. If you have $N$ questions, create $N$ posts. So, please, try to simplify this post and ask only one question.

@nbro Thank you for reaching this and commenting. I didn't want to look like a spam, that's why I asked these questions in one post. How ever, I'll have this in mind for asking questions in the future :)

Please, try to simplify this question is a little bit more. I couldn't even read it until the end because it contains too much information for my brain to process :)


text:
How to create vector representation of roadmap like scans
What would be the best way to create a vector representation of roadmap like scans? The goal I am trying to achieve is illustrated below. The left side represents the source image, the right side the output in the form of three vectors. The fussiness on the left is a simulation, not the actual source image:

The actual source image would look more like:

Currently I am looking at a combination of skeletonization and Hough transform. The result is rather messy though, and seems to warrant quite some extra engineering. Any other suggestions?

It turned out that my intuition was not far off. The skeletonization is a good step. The Hough transform though is not a good way to create a graph of the roadmap. It seems that the Ramer–Douglas–Peucker algorithm can help out here. This algorithm first takes all the skeleton pixels as input, and sees this as a starting graph. The algorithm then proceeds to remove intermediary pixels that do not add information to the shape of the graph.


text:
Which NLP model to use to handle long context?
I'm trying to process product data for an e-commerce platform. The goal is to understand products' size.
Just to show you some examples on how messy product dimension description is:
Overall Dimensions: 66 in W x 41 in D x 36 in H
Overall: 59 in W x 28.75 in D x 30.75 in H
92w 37d 32h",
86.6 in W x 33.9 in D x 24 in H
W: 95.75\" D: 36.5\" H: 28.75\"",
W: 96\" D: 39.25\" H: 32\"",
"118\"W x 35\"D x 33\"T.",
"28 L x 95 W x 41 H"
"95\" W x 26.5\" H x 34.75\" D"
"98\"W x 39\"D x 29\"H"
"28\" High x 80\" Wide x 32\" Deep"

Now assume that the product dimension description is short < 60 characters, I trained a two layer bidirectional LSTM, which can handle this task perfectly.
But the problem is, the above dimension is usually embedded in a long context (as a part of the product description). How can I extract the useful information from the long context and understand it? My LSTM can only accept context size of 60.
What language model is more suitable for this?

Why can you only train an LSTM with a maximum context of 60?

I would also use (in addition of LSTM) some symbolic AI approach, expert-system like (e.g. RefPerSys....)

In this problem, you can also use NER models in order to tag those numbers as Width, Height,... . You can also fine-tune a DistilBert model for your task.
if you want to train NER model by tok2vec you can use:
Spacy Library
for fine-tuning DistilBERT you can use this.
Hugging Face

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
Creating Text Features using word2vec
My task is to classify some texts. I have used word2vec to represent text words and I pass them to an LSTM as input. Taking into account that texts do not contain the same number of words, is it a good idea to create text features of fixed dimension using the word2vec word representations of the text and then classify the text using these features as an input of a neural network? And in general is it a good idea to create text features using this method?


text:
Designing a chatbot personal project with zero coding experience, using an existing platform
My girlfriend has a masters degree in linguistics and would like to create an AI chatbot personal project to show potential employers her linguistics skills since she is struggling to find a job.
Unfortunately she doesn't know how to program except for extremely basic Python skills. She has been searching for weeks for tools to create a chatbot without needing to program, refusing to ask on forums for help so I'm asking the StackExchange community. Sort of like a plugin/widget for Slack, Facebook Messenger or website that you can just install on your website and just concentrate on the workflow/data/conversational design in a similar way to programming in Scratch or Node-Red.
I know nothing about NLP, neural networks or anything like that and I can't understand for the life of me  what exactly it is a linguist needs to do in AI. Adversely, she doesn't have the computer knowledge to understand how an API works, or how to get some sort of service and a chat interface are needed to bootstrap her conversational designs with some code somewhere.
So my question is: Is there a way for a linguist to create a chatbot without knowing programming or going in too deep, all by themselves? We looked at tools like hubspot.com, where the chatbot design is either multiple choices questions with predefined answers or offers expensive paid solutions for companies. I'm sure there are free educational or community open-source platforms doing this.

Dialogflow is a good way to create and deploy an alexa skill/api using nlp, but that is again an api, but also has a GUI, you can just go through a medium/youtube tutorial to get a basic app, get some help (maybe hire a professional on fiverr/upwork for coding stuff) to create an app that showcases her linguistic skills

Thanks friend! For the past 5 hours, I've been searching for tutorials and online courses on DialogFlow to find which are best suited for her and some of them look promising. Also found lots of easy class for Python beginners. Unless someone else has a better solution, Dialogflow and Ptyhon could be the way to go.


text:
My CNN model performs bad on new (self-created) pictures, what are possible reasons?
I wanted to train a model that recognizes sign language. I have found a dataset for this and was able to create a model that would get 94% accuracy on the test set. I have trained models before and my main goal is not to have the best model (I know 94% could easiy be tuned up). However these models where always for class exercises and thus were never used on 'real' new data.
So I took a new picture of my hand that I know I wanted to be a certain letter (let's assume A).
Since my model was trained on 28x28 images, I needed to re-size my own image because it was larger. After that I fed this image to my model only to get a wrong classification.
https://i.sstatic.net/Rf1UK.jpg
These are my pictures (upper-left = my own image (expected class A), upper-right = an image of class A (that my model correctly classifies as A), bottom = picture of class Z (the class my image was classified as)).
You can clearly see that my own image looks for more like the image of class A (that I wanted my model to predict), than the model it did predict.
What could be reasons that my model does not work on real-life images? (If code is wanted I can provide it ofcourse but since I don't know where I go wrong, it seemed out of line to copy all the code).

This is not an uncommon situation. The data set your model is trained on represents a certain probability distribution. Your test set is most likely a good representation from that distribution so your test results will be good.
However when you use real world images they may or may not have a similar distribution. Typically if the training set is large and diverse it is a good representation of the distribution and when the model is used to classify a real world image it will do so correctly.
I think I know of the data set you are working with and if I recall it is fairly large.
So the problem may be that your model is not complex enough to fully capture the complexity of the data. You can test that fairly simply by using transfer learning with a model that is known to be effective for image classification. I recommend using the MobileNet model. It contains only about 4 million parameters but is about as accurate as larger models containing 10 times as many parameters. So MobileNet is not computationally expensive to train. Documentation can be found here.

I’m assuming that you used LeNet (our some other model with small number of parameters) since your training image size is 28x28. Note that LeNet doesn’t generalize well to new images. I think it performs fine (>90%) on MNIST but not good on CIFAR10 (>60%) albeit both datasets contain similar size image. (Just trying to remember the performance from PyTorch implementations). It’s more about if the model has capacity to learn the complexity of dataset. CIFAR10 is more complex and harder to model than MNIST.
LeNet is a small image classification model (in terms of capacity) so it cannot nicely learn the correlation between pixels of input images well and therefore doesn’t perform well on unseen images.
In your case it seems like your model has overfit to training examples. It might perform well on test images because both training and test subsets are sampled from same data generating distribution but real-world images it might experience in future might be different (like your own hand). If it doesn’t perform well on unseen images we say it has not generalized well, which looks the case in your situation. In this case you need a validation set to validate that your model generalizes to unseen images. If you have it then you should use it in early stopping regularization technique. You can also add other regularizers to your model (the simplest one is weight decay).
But instead of inventing your network architecture why don’t you use models like ResNet. Just fine-tune the pre-trained ResNet on your own dataset. I’d prefer to fine-tune personally in this situation because data distribution it was trained on (ImageNet) is pretty different from your hand sign dataset. In other case if your dataset contained nature and surrounding images I’d rather freeze the parameters of fixed-feature extractor layers and trained only the last few layers of ResNet (or similar model).
I hope this helps!

Are you sure the image quality in your test set and phone camera image similar?
I once trained a CNN model on poor quality images and with very good validation accuracy but when I tested on image from my camera it didn't work at all. I degraded the image quality by resizing the image from my camera to a small size then again back to required size and it worked perfectly.


text:
Simplification of expected reward under the limit in continuous tasks
I was reading the average reward setting for continuous tasks from rich sutton's book (page 202, 2nd edition). There he perform a simplification over the expected reward under the limit approaching to infinite. I mark this point in this picture:

The book does not clearly mention the steps to simplify the above expression. I search on the web to find the solution but there is no clear explanation on that. Can anyone explain the marked point?


text:
3d representation of a regression with two independent variables one of them is categorical and another is continuous
I have hopefully a fundamental question of Do I understand things right.
(Thank you in advance and sorry for my English which might be not so good)
1-Preambula 1:
I know that if we have 2 independent variables, both of a continuous type, it is ok to represent them as a 2d plane in a 3d space:

2-Preambula 2:
I have seen that many times when we have to deal with continuous and categorical variables(male\female for example), we represent them like this(note the lines are parallel):

3-Assumption:
In the beginning I assumed that it is 2d representation of this 3d case:

4-Discussion 1: But If my assumption above was right, why do they always "picture"  it with parallel lines? After all this is a very specific situation. In most of cases both regression lines will not be parallel, further more, they may have different slope direction (one negative and another positive)For example:

5-Discussion 2: On the other hand parallel models may be explained in such a way: if we will add a regression hyperplane which "somehow" fits both groups(male and female), we will get the parallel lines:

6-Finally My questions are quite simple.
question 5.1: Did I understand right the nature of parallel lines as I show it above (in discussion2)?
question 5.2: If I was right in 5.1, I assume that in such cases hyperplane regression is a quite a bad predictor. Am I right?


text:
What do the authors of this paper mean by the bias term in this picture of a neural network implementation?
I am reading a paper implementing a deep deterministic policy gradient algorithm for portfolio management. My question is about a specific neural network implementation they depict in this picture (paper, picture is on page 14).

The first three steps are convolutions. Once they have reduced the initial tensor into a vector, they add that little yellow square entry to the vector, called the cash bias, and then they do a softmax operation.
The paper does not go into any detail about what this bias term could be, they just say that they add this bias before the softmax. This makes me think that perhaps this is a standard step? But I don't know if this is a learnable parameter, or just a scalar constant they concatenate to the vector prior to the softmax.
I have two questions:
1) When they write softmax, is it safe to assume that this is just a softmax function, with no learnable parameters? Or is this meant to depict a fully connected linear layer, with a softmax activation?
2) If it's the latter, then I can interpret the cash bias as being a constant term they concatenate to the vector before the fully connected layer, just to add one more feature for the cash assets. However, if softmax means just a function, then what is this cash bias? It must be a constant that they implement, but I don't see what the use of that would be, how can you pick a constant scalar that you are confident will have the intended impact on the softmax output to bias the network to put some weight on that feature (cash)?
Any comments/interpretations are appreciated!

to the cash bias: I think this is simply the money that is still available at time t=50 and has not yet been invested.

Yes, the softmax is just a softmax, applied to the 12 values from the final linear layer.

As far as the "cash bias" goes — I'm not sure if the "cash" part has significance to the authors. But, the "bias" part is standard. The inputs to the last layer are 11-dimensional vectors. But the last layer has 12 parameters: one parameter for each input value, and one additive parameter called the bias.

That is, assuming the input vector is $(x_1, x_2, ..., x_{11})$, then output will be:
$$
y = w_0 + x_1w_1 + x_2w_2 + ... + x_{11}w_{11}
$$
The weight $w_0$ is called a bias. If you remove it, then you'll always map the input of all zeros, $(0, ... 0)$ to the output of $0$. This might not be desirable, and might prevent the layer from converging. Adding a bias $w_0$ allows the layer to map any input to any output.
Not that the $w_0$ is not a true constant. It's a trainable parameter like all others.

On page 8 they say:

In the experiments of the paper, the 11 most-volumed non-cash assets are preselected for
the portfolio. Together with the cash, Bitcoin, the size of the portfolio, m + 1, is 12. This
number is chosen by experience and can be adjusted in future experiments. For markets
with large volumes, like the foreign exchange market, m can be as big as the total number
of available assets.

They have 12 possible actions in the policy: buy one or more of the 11 non-cash assets, or sell them and hold more cash. It seems that the bias towards the cash position is a learnable parameter.
Although now that I read that quote a few times, it seems that they treat Bitcoin as cash?! At first I thought they meant a currency like USD. But in effect the outcome is the same. As you increase the cash bias, the softmax activation gets biased towards that policy.


text:
Reservoir of LSM vs. FF-NN or ELM
The reservoir of the Liquid State Machine is an array of random numbers connected to each other with a probability depending on the distance between each other. Because of this connection with each other it apparently has "recurrence". The reservoir is followed by a readout stage where the actual weight training is done.
The hidden layer of a FF-NN can be an array of random weights, which is exactly what an Extreme Learning Machine is. ELM has a closed form solution of the second-stage weight (Beta) calculation i.e. only the Beta needs to be trained.
So in both cases you have a second-stage layer or readout layer where weights are trained.
My question is, if the reservoir random weights are very much like the ELM random weights and both don't need to be trained, how are they any different than each other? In other words, both have a set of untrained random weights, so in LSM where is the recurrence exactly happening if the weights are just random? Can't the LSM be reduced to a FF-NN?


text:
Is there a parallelizable algorithm for training sparse neural networks?
Consider that we want to create a very big neural network. If we consider to use dense layers, we might face some challenges. Now consider that we use sparse layers instead of dense layers. When using a really sparse model, we would have much less parameters and could create much bigger neural networks. Is there an efficient algorithm to parallelize the training of such networks?


text:
How to manage the different pixel size for a CNN?
I have a convolutional neural network such as U-Net for a segmentation task and in input images with the same spatial resolution (256x256) but different pixel size due to the acquisition process. Specifically, every image has FoV 370x370mm 256/256 pixel, but a different zoom, for example an image might have 2.7/1.8 px/mm and another image 2.4/1.7 px/mm.
Considering the FoV the pixel size should be 370/256=1.44mm x pixel but with a zoom of 2.7/1.8 px/mm which is the pixel size in this case ? I thought 1.8/2.7= 0.67mm but I am not sure.
Why should I have the same in-plane resolution (pixel size) for each image when I train my CNN and not only the same spatial resolution (256x256 px) ?


text:
what will be the best loss function for unet to predict the each pixel values?

I'm predicting the used 9 pictures to predict the  last picture
so (40,40,9) -> unet -> (40,40,1)
but as you see the predict picture

It's not just a mask(0or 1) its float
so which loss function should I define to achieve the best Unet result? and why?