text:
Generation of 'new log probabilities' in continuous action space PPO
I have a conceptual question for you all that hopefully I can convey clearly. I am building an RL agent in Keras using continuous PPO to control a laser attached to a pan/tilt turret for target tracking. My question is how the new policy gets updated. My current implementation is as follows

Make observation (distance from laser to target in pan and tilt)
Pass observation to actor network which outputs a mean (std for now is fixed)
I sample from a gaussian with the mean output from step 2
Apply the command and observe the reward (1/L2 distance to target)
collect N steps of experience, compute advantage and old log probabilities,
train actor and critic

My question is this. I have my old log probabilities (probabilities of the actions taken given the means generated by the actor network), but I dont understand how the new probabilities are generated. At the onset of the very first minibatch my new policy is identical to my old policy as they are the same neural net. Given that in the model.fit function I am passing the same set of observations to generate 'y_pred' values, and I am passing in the actual actions taken as my 'y_true' values, the new policy should generate the exact same log probabilities as my old one. The only (slight) variation that makes the network update is from the entropy bonus, but my score
np.exp(new_log_probs-old.log_probs)  is nearly identically 1 because the policies are the same.
Should I be using a pair of networks similar to DDQN so there are some initial differences in the policies between the one used to generate the data and the one used for training?

Welcome to SE:AI!  (Thanks for specifying conceptual, which is on-topic. However, if you don't get an answer here, might try CS or Overflow:)

The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray too far away from the optimal point.
This means you repeat your step 6. epoch amount of times for the same batch of trajectories. Usually epoch is somewhere between 3 and 30 but it is a hyper-parameter you need to adjust. For the first repeat, the old and the new policy are the same, so their ratio should be 1. After the first update, the new probabilities will change due to the updated policy, whereas you will still need to use the old probabilities giving you a ratio different from 1. The old probabilities will stay the same during these epoch update steps, whereas your new probabilities will keep changing.

Thank you for your response. When the ratio is 1, how does the new policy change so the ratio isn't one the second time around? Doesn't a ratio of one mean the gradient is zero?

@Gideon When the ratio is 1, the update reduces to a simple policy gradient update, for example to one used in the advantage actor-critic. The gradient is not zero, because your parameterized policy $\pi_\theta$ depends on $\theta$ and this gradient will exist. The ratio is just a multiplier telling you how much you can scale the gradient $\partial (\log{\pi_\theta})/\partial \theta$.

If you'd like to "cheat" a bit, you could peek into this repo (https://github.com/seungeunrho/minimalRL/) that contains simple RL implementations in pytorch and compare for example the code for actor-critic and PPO.


text:
Which method of tree searching should be used for this board game?
Suppose the following properties of a board game:

High branching factor in the beginning of the game (~500) which slowly tends towards 0 at the end of the game

Evaluation of the any given board state isn't hard to create and can be quite accurate

And that we want to create an AI to play such board game.

What method of tree searching should be applied for the AI?

Considering the absurd branching factor (at least for most of the game), the Monte Carlo method of search is appealing. The problem is that from what I've seen usually monte carlo search methods are used on games with both high branching factor and no easy evaluation function. However that is not the case for this board game as previously stated.
I'm simply curious how this property of evaluation should influence my decision. For example: Should I replace simulations and playouts with an evaluation function? At that point, would alpha-beta pruning minimax work better? Is there some hybrid which would be optimal?

There are a bunch of "add-ons" to minimax that may work for you. Some of them are MTD(f), best node search, PVS, etc. I don't know which one would be the best for your game though.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
is it ok to take random actions while training a3c as in below code
i am trying to train an A3C algorithm but I am getting same output in the multinomial function.
can I train the A3C with random actions as in below code.
can someone expert comment.
while count<max_timesteps-1:
            value, action_values, (hx, cx) = model((Variable(state.unsqueeze(0)), (hx, cx)))
            prob = F.softmax(action_values,dim = -1)
            log_prob = F.log_softmax(action_values, dim=-1)
            print(log_prob.shape)
            print("log_prob: ",log_prob)
            entropy = -(log_prob * prob).sum(1, keepdim=True)
            entropies.append(entropy)
            actn = np.random.randn(3)
            action = actn.argmax()
            log_prob = log_prob[0,action]
            # print("log_prob ",log_prob)
            # print("action ",action)
            state, reward, done = env.step(action)
            done = (done or count == max_timesteps-2)
            reward = max(min(reward, 1), -1)

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What are examples of problems where neural networks have achieved human-level or higher performance?
What are examples of problems where neural networks have been used and have achieved human-level or higher performance?
Each answer can contain one or more examples. Please, provide links to research papers or reliable articles that validate your claims.

Here is an initial list of AI systems that used neural networks and have achieved human-level or superhuman performance. All of these systems are reinforcement learning systems that play videogames.

AlphaGo and AlphaGo Zero (an improved version of AlphaGo that does not use human knowledge but learns by playing against itself) have achieved superhuman performance in the game of go and, in the case of AlphaZero (a generalized version of AlphaGo Zero), also in the games of chess and shogi.
DQN has achieved human-level or superhuman performance in many Atari games
DeepStack has achieved human-level performance in poker
AlphaStar defeated a top professional player in the real-time strategy game StarCraft 2
OpenAI Five defeated world champions in the game of Dota

It's worth understanding that neither alphaStar nor OpenAI Five were able to actually visually understand the games they are playing. They are fed numeric information such as coordinates and unit type. This is significant as visual processing of the game is as important as anything else.

@FourierFlux Thanks for noticing this. To be honest, I am only familiar with the details of DQN and some details of AlphaGo, but not the details of the other two.

AlphaDogfight - from Defense Advanced Research Projects Agency (DARPA) a programme that pitted computers using F-16 flight simulators against one another and later went on to defeat Air Force’s top F-16 fighter pilots.
Check out this and this news and events by DARPA.


text:
How to avoid over-fitting using early stopping when using R cross validation package caret
I have a data set with  36 rows and 9 columns. I am trying to make a model to predict the 9th column
I have tried modeling the data using  a range of models using caret to perform cross-validation and hyper parameter tuning: 'lm', random forrest (ranger) and GLMnet, with range of different folds and hyper-parameter tuning, but the modeling has not been very successful.
Next I have tried to use some of the neural-network models. I tried the 'monmlp'. During hyper parameter tuning I could see that the RMSE drops to a level when using ~ 6 hidden units. The problem I observe using this model is

Prediction is almost equal to data
When doing a "manual" cross validation by removing a single datapoint and using the trained model to predict, it has no predictive power

I have tried to use a range of different hidden units, but i think the problem is that the model is overfitted despite using  caret cross validation feature.
There two feedbacks I would appreciate

Is there a way to prevent overfitting, by chosen optimal number of training iterations ( optimal RMSE on out of sample ). Can this by done using caret or some other package
Am I using the right model?

I am relatively unexperienced with ML and choosing a good model is tough: when you look at the available packages it is overwhelming:
https://topepo.github.io/caret/train-models-by-tag.html

You are handling a very small dataset. The only way to prevent overfitting then is to choose a very restrictive model search space. The simpler the better, and you should prefer models involving some regularization. Even tuning hyperparameters will be hard, so avoid families with many hyperparameters. Neural networks are definitely a no-go, IMHO.
Cross-validation is important in this case, a.o. for tuning the regularization hyperparameter. You were right to perform the extra leave-one-out cross-validation. But remember checking the confidence intervals, which will be very broad.
I would suggest to use some prior business knowledge to further reduce the number of candidate predictor variables to 2-4. Then you can use a linear regression or a ridge regression (i.e. linear regression with L2 regularization, amount to be tuned as hyperparameter) using only these variables. If you have reasons to believe that the relation is very non-linear, you'll need to find another family.
Another possibility is to perform a data-driven feature selection (see https://topepo.github.io/caret/feature-selection-overview.html), but it is harder to properly implement and cross-validate. I wouldn't recommend it to a beginner. Remember that this step can also introduce overfit or instability.

Thanks for your answer. I have tried to assess other models using cross validation and could see that the complex neural network models did not perform very well on out of sample data.. what tricked me was that i looked at the predicted data from the training on the full data set ( after optimization of hyper parameter tuning). This predicted data alwasys looked extremely well correlated with actual data. I did not look at the magnitude of the RMSE after hyperparamter tuning.Because. When I then did an extra "leave-one-out" validation i could get random results.
In the end I used caret's resampling method  which compares RMSE of different models.and found out that svmPoly (support vector machine) or random forrest had the best out of sample performance - better than lm and GLMnet.


text:
Is there a way to make my neural network discard inputs with bad results from learning?
What I want to achieve is this: If my desired outputs are [1, 2, 3, 4] I would rather have my network produce this output:
[0.99, 2.01, 999, 4.01]
than say this:
[0.94, 1.88, 3.12, 4.1]
So I'd rather have a few very accurate outputs and the rest completely off, than have them all be decent but no more than that. My question is, is there a known way to do this? If not, would it make sense to remove the inputs that produce poor outputs, and redo the learning phase?

I assume [1, 2, 3, 4] are the desired outputs for different examples for a regression task. Sound like you need a different loss function. From your description it seems you don't care how big the error is if it's bigger than some value. Try the Huber loss(in Pytorch and TensorFlow). Examples that are far from the expected value won't produce big gradients (:

Yes that's correct. Thank, I will check that out

Can I just make the gradient 0 if the error is too large? So I have 2 loss funtions, depending on the error. One is zero and the other one is quadratic or whatever else

If you implement the loss function to be a constant, when the error is too large, you’ll get this behavior. The best way to achieve it is using torch.where/tf.where. Note that this might not be optimal. If many examples have an error larger than the threshold, the model won’t train or it will overfit on a subset of the examples.
Can you provide more context?

I'm trying to predict stock prices so the idea is after an initial phase of learning, when no error is very large and the network has a decent grasp of the function, to ditch the worst outputs in order to get better accuracy on the good outputs

Use genetic algorithm. Run like 25 neural networks at once and choose the most successful  one. This method is similar to evolution, which is why it is very effective. I created a model like this with similar sized training data as yours and it reached an overall error rate of 0.06% in a second. Don’t get rid of nodes. Instead, eliminate the bad networks. However, this doesn’t produce extremely high error rates if that is what you want.

They don't have to be high, I just wanted to say that if error is larger than x it might as well be a gazillion, I don't care. But you're just saying to run the network many times, not necessarily at the same time?

Yes. Also consider increasing or decreasing your mutation amount (random increase to weights) based on how well the networks are performing.

This will help increase network accuracy, but doesn't address the specific question about prioritising a few correct outputs over punishing very incorrect outputs. Addressing this ultimately comes down to the loss function, as that determines how to handle bad outputs.

authors: هاني القباطي


text:
What are some programming related topics that can be solved using NLP?
I've been working on the Punctuation Restoration Problem for my Master's Thesis, however, me being primarily a programmer at heart, I wish I could use some of my NLP skills to solve issues related to programming in general.
I know Microsoft does lots of research in NLP and I think after they acquired Github, they have an immense dataset to work with for any problems related to programming they want to tackle. Most recently I think they did a great job on their new python suggestion extension on VSCode.
So, could you suggest to me some issues you think are interesting research topics? This is something that I would like to work with, but I have no idea where to start yet.

Look into the https://www.decoder-project.eu/ project

Also look into the RefPerSys project and consider using it for your Master's thesis and contributing to it. See also this answer


text:
Handling a Large Discrete Action Space in Deep Q Learning
I am attempting to solve a timetabling problem using deep Q learning. It could be thought of as a resource allocation problem to obtain some certificate of 'optimality'. However, how to define and access the action space is alluding me. Any help, thoughts, or direction towards the literature would be appreciated. Thanks!
The problem is entirely deterministic, the pair of the current state and action is isomorphic to the resulting state. The Q network is therefore being set up to approximate a Q value (a scalar) for the resulting state, i.e. for the current state and proposed action.
I have so far assumed that the action space should be randomly sampled during training to generate some approximation of the Q table. This seems highly inefficient.
I am open to reinterpretations of the action space. The problem involves a set of n individuals and at any given state a maximum of b can be 'active' and, of the remaining 'inactive' individuals, f can be made 'active' by an action. An action will need to involve making some reallocation to active individuals made up of those who are already active and the other f available people.
To give you a sense over the numbers that I will ultimately use, $n=17, b=7$, and $f$ will hover somewhere around 7-10 (but depends on the allocations). At first this sounds tractable, but a (very) rough approximation of the cardinality of the set of actions is 17 choose 7 = 19448.
Does anyone know a more efficient way to encode this action space? If not, is there a more sensible way to sample it (as is my current plan) than uniformly extracting actions from the space? Also when sampling the space is it valid to enforce some cap on the number of samples drawn (say 500). Please feel free to ask for further clarification.

I read a paper the other day that dealt with problems of this nature. Check out Deep Reinforcement Learning in Large Discrete Action Spaces


text:
Why does CNN forward pass take longer compared to MLP forward pass?
Let's take a 32 x 32 x 3 NumPy array and convolve with 10 filters of size 2 x 2 x 3 with stride 2 to produce feature maps of volume 16 x 16 x 10. The total number of operations - 16 * 16 * 10 * 2 * 2 * 2 * 3 = 61440 operations. Now, let's take an input array of length 3072 (flattening the 32 * 32 * 3 array) and dot it with a weight matrix of size 500 x 3072. The total number of operations - 500 * 3072 * 2 = 3072000 operations. The convolution takes 4-5 times longer than np.dot(w, x) even though number of operations is less.
Here's my code for the convolution operation:
for i in range(16):
    for j in range(16):
        for k in range(10):
            v[i, j, k] = np.sum(x[2 * i:2 * i + 2, 2 * j:2 * j + 2] * kernels[k])

Is np.dot(w, x) optimized or something? Or are my calculations wrong? Sorry if this is a silly question...

Programming questions are generally off-topic here. Please, ask this type of question on Stack Overflow. Here we focus on theoretical, philosophical and social aspects of AI. Please, read also our on-topic page https://ai.stackexchange.com/help/on-topic to know more about it.

From the NumPy Linear algebra documentation:

The NumPy linear algebra functions rely on BLAS and LAPACK to provide
efficient low level implementations of standard linear algebra
algorithms. Those libraries may be provided by NumPy itself using C
versions of a subset of their reference implementations but, when
possible, highly optimized libraries that take advantage of
specialized processor functionality are preferred.

In your case, np.dot() is a matrix-vector multiplication which internally calls  such a highly optimized BLAS routine. Those libraries are implemented in low-level languages like Fortan or C, they exist for many years now and are still unbeaten in terms of speed.
Your naive implementation of the convolution operation in python on the other hand is not optimized for execution speed and will be executed by the python interpreter (which is much slower than the compiled C functions).
You could try to replace your manual convolution by the SciPy's equivalent scipy.signal.convolve2d(), to make advantage of optimized libraries as well and get a speed up.


text:
What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?
I was watching a Youtube video in which the problem of trying to predict the last word in  a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk". The lecturer in this video stated that whilst sentences (the sequence) can be of varying lengths, if we take a really large fixed window we can model the whole sentence. In essence she said that we can convert any sentence into a fixed size vector and still preserve the order of the sentence (sequence). I was then wondering why do we need RNNs if we can just use FFNNs? Also does a fixed size vector really preserve sequential order information?
Thank You for any help!


text:
How to identify if 2 faces contain the same person?
I have got numerous frames and I've detected all the faces in all the frames using Retinaface. However I need to track the faces of people over frames.
For this purpose, I assumed I could try finding the landmarks from the face using libraries like dlib and maybe compare these landmarks to check if they are infact the face of the same person.
I would like to know if there are other methods or some useful resources I could refer for the same. Thanks a lot in advance.

The topic of your problem is person re-identification. You can check here.

You can try using something called as a Siamese network if you are willing to train the network on your own using something called as triplet loss(if you have lots of face images).
Another approach would we something called a one-shot using FaceNet(transfer learning approach)
FaceNet uses deep convolutional neural network (CNN). The network is trained such that the squared L2 distance between the embeddings correspond to face similarity. The images used for training are scaled, transformed and are tightly cropped around the face area.
Another important aspect of FaceNet is its loss function . It is already trained using triplet loss function.In this case you could just feed to face images and you would get a thershold score for the similarities.


text:
Neural Network for locating shifting resonant frequencies
I have multiple FFT's taken from a sample at different pressures, through different analysis I can see that the resonant frequencies are shifting in the spectrum for each FFT at a different pressure.
Using conventional peak tracking has been difficult as the peaks increase/decrease in magnitude within the FFT as well as shifting in the spectrum.
Is it possible for a neural network to 'detect'/'pick out' these frequency values?
Any help or guidance is appreciated :)
Thanks!


text:
Hyper-plane in logistic regression vs linear regression for same number of features
Geometric interpretation of Logistic Regression and Linear regression is considered here.
I was going through Logistic regression and Linear regression. In the optimization equation of both following term is used. $$W^{T}.X$$
W is a vector which holds weights of the hyper-plane.
I realized following about the dimensions of the fitted hyper-plane. Want to confirm it.
Let,
d = Number of features for both Logistic Regression and Linear Regression.
Logistic Regression case:
Fitted hyper-plane is d-dimensional.
Linear Regression case:
Fitted hyper-plane is (d + 1) dimensions.
Example
d = 2
feature 1 : weight,
feature 2 : height
Logistic Regression:
Its a 2 class classification.
y : {obsess, normal)
Linear Regression:
y: blood pressure (real value)
Here,

Logistic Regression will fit a 2-D line.
Linear Regression will fit a 3-D plane.

Please confirm if this understanding is correct and same happens even in higher dimensions.

Mm, I understand what you want to say, but I think you're slightly wrong with terminology. If you consider a d-dimensional vector space, people usually a hyper-plane (d - 1)-dimensional, because it's a subspace of lower dimensionality. So for your example d = 2 the logistic regression would fit a 1-dimentional line (which would separate 2 classes) and the linear regression would fit a 2-dimentional plane (for every combination of 2 features the prediction would lay on some plane in 3-D space). So the general rule would be logreg -> (d-1)-dimensional hyperplane in d-dimensional vector space, linear regression -> d-dimensional hyperplane in (d+1)-dimensional vector space.


text:
Is it feasible to train a DQN with thousands of input ports?
I designed a DQN architecture for some problem. The problem has a parameter $m$ as the number of clients. In my situation, $m$ is large, $m\in\{100,200,\ldots,1000\}$. For this situation, the number of input ports of the DQN is some few thousand, $\{1000, 2000, \ldots, 10000\}$. For some fixed $m$, I would like to see the performance of deep Q learning on the performance. So I have to train the DQN for every change that occurs on $m$ and this should handle thousands of inputs ports for each training. Is this situation familiar in DQN and if not how to solve this issue?

By input ports are you talking about the state space being an $m$ dimensional vector, or the action space?

Yes it is the state space, and it is a $O(m^2)$-dimensional vector.

that is...large. I've never experimented with a state space close to that size. I would imagine you'd first need to have a network capable of handling such large dimensional inputs and even then I'm not sure how feasible it would be to train an agent given the amount of exploration that would need to be done for a state space that large.


text:
Can we use NLP to understand/parse/compile programming code?
I wonder if we can use Natural Language Processing (NLP) to process programming code:
Given a piece of code, can we

Translate it to human language to understand what it does? The input could be a function definition（normally lack of documentation) in Python and the output could be the documentation for that function.
Compile or translate it to another programming language? Compile Python code to C or machine code, or translate C code to Python code?

Yes, and that is the ambition of the Decoder project (H2020 funded in Europe). Analyzing with NLP techniques the comments in e.g. C or C++ source code (of open source projects, and perhaps their git logs).
I even happen to be on the photo.
My dream is to try similar things, in a few years, in RefPerSys. You could join that project, BTW.

I just discovered that we are neighbors :)


text:
How to use a NN for seq2seq tasks?
I am trying to make a NN(probably with dense layers) to map a specific input to a specific output (or basically sequence2sequence). I want the model to learn the relation between the sequences and predict the output of any other input I give it.
I have 2 files - one with the inputs and another with all the corresponding outputs and would probably use a bunch of Dense Layers with word embeddings to vectorize it into higher dimensions. However, I cannot find any good resources out there for that.
Does anyone know how to accomplish such an NN? Which architectures are best for pattern matching? examples, links, and other resources would be very welcome. I was considering using RNN's but found them not very good in the pattern matching tasks so had ditched them. I would still consider them if someone can provide a plausible explanation...

Can you provide a sample of data?  The only clue you provided is word embeddings so I assume you have text.  Why do you speak of patterns?  Are you looking for specific phrases or sequences of words?  If so are you interested in semantics and not just syntax?

I think you meant pattern recognition instead of pattern matching in you question. Because pattern matching has nothing to do with NNs as far as I know. So RNNs is the easiest architecture you could use for this task. Have a look at this post. It's long, but it's very well written and explains why RNNs work well with sequential data.
In brief, RNNs accumulate information about what was fed to them previously and store it in the hidden state vector. That information is a summary of a sequence which helps it to predict the output sequence. RNNs can be applied to variable length input and catch dependencies among different parts of the input (as opposed to dense layers that can be applied to a sequence only point-wise).
Here's how exactly you can apply reccurent layers (2, 3).
There are more advanced architectures like LSTMs. Also you can try transformers (4) which use attention mechanism (5).

solotsky Can you give an explanation on why you find RNN's to be good for finding abstract patterns? Also, My input sequence is just a word of about 40 characters so I doubt it would be very useful...

Michael: NNs are used for pattern matching.

authors: Анна Поплавская


text:
Zero shot learning available labels in testing set
As we all know, zero shot learning involves a model predicting classes that it has not seen. But we are given all the attributes each class might have.
Is it fair to assume that we are "aware" of all the class labels a dataset might have ? (Including the test set)

Hello. Welcome to AI SE! It seems that you're asking for an opinion because you're asking "Is it fair to assume". Maybe you can formulate your question a little bit better so that an objective answer can be given.

The formal definition of zero-shot learning is that given labeled training instances $D_{tr}$ belonging to the seen classes $S$, the aim is to learn a classifier $f^u(·):X→U$ that can classify testing instances $X_{te}$ (i.e., to predict $Y_{te}$) belonging to the unseen classes $U$.
The general idea of zero-shot learning is to transfer the knowledge contained in the training instances $D_{tr}$ to the task of testing instance classification. So it is considered a transfer learning method, and, more specifically, heterogeneous transfer learning with different label spaces.
However, since no labeled instances belonging to the unseen classes are available,
to solve the zero-shot learning problem, some auxiliary information is necessary. Such auxiliary information should contain information about all of the unseen classes. This auxiliary information usually contains some semantic information about the unseen classes, and its representation belongs to a space that is often referred as $semantic$ $space$. In the semantic space, each class has a corresponding vector representation, which is referred to as the class prototype or $prototype$ for short.
So, you should be aware of classes of your problem, in order to construct the semantic space and their class prototypes. I also suggest you read the survey paper "A survey of zero-shot learning: settings, methods and applications".


text:
Are linear approximators better suited to some tasks compared to complex neural net functions?
Model based RL attempts to learn a function $f(s_{t+1}|s_t, a_t)$ representing the environment transitions, otherwise known as a model of the system. I see linear functions are still being used in model-based RL such as in robotic manipulation to learn system dynamics, and can work effectively well. (Here, I mean in learning the model, not as an optimization method for the controller selecting the best actions).
In model-based RL, are there situations where a learning a linear model such as using a Lyapunov function would be better suited than using a neural network, or are the examples of problems framed to use linear models when addressing them using model-based RL?

This might be helpful: "Linear approximations of non-linear physical systems prove accurate and are well accepted in many fields. For the comprehensive study, see Linear models of nonlinear systems"

This is just a case of supervised learning. You are trying to predict $s_{t+1}$ given $s_t$ and $a_t$, so the answer to your question depends on how complex your state dynamics are.
For example, if the state space is really complex, e.g. if your state space is an image and you want to predict the next image given the current image and an action, then linear methods are unlikely to work well.

Yes, state space will be key. This work uses a $32 * 32$ image on a linear model. Would that be termed as complex?

I'm not sure, I'm not an image expert. In this paper they learn state dynamics in a latent space -- the raw state space is an image -- and they use deep nets for this, so I would assume you need something more complex than a linear model to predict images.


text:
Back propagation approach to logistic regression: why is cost diverging but accuracy increasing?
Background I have tried to fit a logistic regression model - written using a forward / back propagation approach (as part of Andrew Ng's deep learning course) - to a very non-linear data set (see picture below). Of course, it totally fails; in Andrew Ng's course, the failure of logistic regression to fit to this motivates developing a neural net - which works quite nicely.  But my question concerns what my logistic model is doing and why.

The problem My logistic regression model's cost increases, even after massively reducing the learning rate. But at the same time my accuracy (slowly) increases. I simply cannot see why.
To confuse matters even more - if I resort to a negative learning rate (essentially trying to force the calibration to higher cost values) the cost then decreases for a time until the accuracy hits 50%. After this point, the cost then inexorably increases - but the accuracy stays equal to 50%. The solution so found is to set all points to either red or blue (a reasonable fit given logistic regression simply cannot work on this data).
My questions and thoughts on answers I have reproduced the Python code below - hopefully it's clear. My questions are:

Is there a mistake in the model that explains why negative learning rates seem to work better?
On the topic of why the cost increases even as accuracy asymptotes to 50%: is the issue that once the model has discovered the "all points equal to either red or blue" solution the parameters "w" and "b" just get larger and larger (in absolute terms) - driving all of the predictions closer to 1 (or conversely if it predicts all points are 0)?

To explain this second question a bit more: imagine red points are defined by y = 1. Suppose parameters w, b are chosen such that the probability for every point equals 0.9. Then the model predicts all points are red - which is correct for half the points. The model can then improve half the predictions by driving w and b up (so that sigmoid ( w*x + b) --> 1). But of course, this makes half the predictions (the blue points) more and more wrong - which causes the cost function for those points - log(1 - prob) - to diverge. I don't truly see why gradient descent would do this but it's all I can think of for the peculiar behaviour of the algorithm.
Hope this all makes sense. Hit me up if not.
import numpy as np
import matplotlib.pyplot as plt

# function to create a flower-like arrangement of 1s and 0s
def load_planar_dataset():
    np.random.seed(1)
    m = 400 # number of examples
    N = int(m/2) # number of points per class
    D = 2 # dimensionality / i.e. work in 2d plane - so X is a set of (x,y) coordinate points
    X = np.zeros((m,D)) # data matrix where each row is a single example
    Y = np.zeros((m,1), dtype='uint8') # labels vector (0 for red, 1 for blue)
    a = 4 # maximum ray of the flower

    for j in range(2):
        ix = range(N*j,N*(j+1))
        t = np.linspace(j*3.12,(j+1)*3.12,N) + np.random.randn(N)*0.2 # theta / random element mixes up some of the petals so you get mostly blue with some red petals and vice-versa
        r = a*np.sin(4*t) + np.random.randn(N)*0.2 # radius / again random element alters  shape of flower slightly
        X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
        Y[ix] = j

    X = X.T # transpose so columns = training example as per standard in lectures
    Y = Y.T

    return X, Y

# function to plot the above data plus a modelled decision boundary - works by applying model to grid of points and colouring accordingly
def plot_decision_boundary(model, X, y):
    # Set min and max values and give it some padding
    x_min, x_max = X[0, :].min() - 1, X[0, :].max() + 1
    y_min, y_max = X[1, :].min() - 1, X[1, :].max() + 1
    h = 0.01
    # Generate a grid of points with distance h between them
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    # Predict the function value for the whole grid
    Z = model(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    # Plot the contour and training examples
    plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
    plt.ylabel('x2')
    plt.xlabel('x1')
    plt.scatter(X[0, :], X[1, :], c=y, cmap=plt.cm.Spectral)

# sigmoid function as per sandard linear regression
def sigmoid(z):
    """
    Compute the sigmoid of z

    Arguments:
    z -- A scalar or numpy array of any size.

    Return:
    s -- sigmoid(z)
    """

    s = 1. / (1. + np.exp(-z))

    return s

#
def propagate(w, b, X, Y):
    """
    Implement the cost function and its gradient for the propagation explained above

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)

    Return:
    cost -- negative log-likelihood cost for logistic regression
    dw -- gradient of the loss with respect to w, thus same shape as w
    db -- gradient of the loss with respect to b, thus same shape as b
    """

    m = X.shape[1];

    # forward prop
    Z = np.dot(w.T, X) + b;
    A = sigmoid(Z); # activiation = the prediction of the model

    # compute cost
    cost =  - 1. / m * np.sum( (Y * np.log(A) + (1. - Y) * np.log(1. - A)  ) )

    #back prop for gradient descent

    da = - Y / A + (1. - Y) / (1. - A)
    dz = da * A * (1. - A) # = - Y (1-A) + (1. - Y) A =  A - Y
    dw = 1. / m * np.dot( X, dz.T )
    db = 1. / m * np.sum(dz)

    grads = {"dw": dw,
                "db": db}

    return grads, cost

def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
    """
    This function optimizes w and b by running a gradient descent algorithm

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of shape (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
    num_iterations -- number of iterations of the optimization loop
    learning_rate -- learning rate of the gradient descent update rule
    print_cost -- True to print the loss every 100 steps

    Returns:
    params -- dictionary containing the weights w and bias b
    grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.

    """
    costs = []

    for i in range(num_iterations):

        # cost /gradient calculation
        grads, cost = propagate(w, b, X, Y)

        #retrieve derivatives
        dw = grads["dw"]
        db = grads["db"]

        # update values according to gradient descent algorithm
        w = w - learning_rate * dw
        b = b - learning_rate * db

        # record the costs
        if i % 100 == 0:
            costs.append(cost)

            # Print the cost every 100 training iterations
            if print_cost:
                print("Cost after iteration %i: %f" %(i, cost))

    params = {  "w": w,
                "b": b}

    grads = {   "dw": dw,
                "db": db}

    return params, grads, costs

def predict(w, b, X):
    '''
    Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)

    Returns:
    Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X
    '''
    Z = np.dot(w.T, X) + b
    A = sigmoid(Z)

    Y_prediction = (A >= 0.5).astype(int)

    return Y_prediction

np.random.seed(1) # set a seed so that the results are consistent

X, Y = load_planar_dataset()

# Visualize the data:

plt.scatter(X[0, :], X[1, :], c=Y, s=40, cmap=plt.cm.Spectral); # s = size of points; cmap are nicer colours
plt.show()

shape_X = X.shape
shape_Y = Y.shape
m = shape_Y[1]  # training set size
n = shape_X[0] # number of features (2)

# initialise parameters
w = np.random.rand(n, 1)
b = 0

# print accuracy of initial parameters by comparing prediction to
print("train accuracy: {} %".format(100 - np.mean(np.abs(predict(w, b, X) - Y)) * 100))

# fit model and print out costs every 100 iterations of the forward / back prop
parameters, grads, costs = optimize(w, b, X, Y, num_iterations = 10000, learning_rate = 0.000005, print_cost = True)

# return the prediction
Y_prediction = predict(parameters["w"], parameters["b"], X)

# print accuracy of fitted model
print("train accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction - Y)) * 100))

# print parameters for interest
print( parameters["w"] , parameters["b"] )

# plot decision boundary
plot_decision_boundary(lambda x: predict(parameters["w"], parameters["b"], x.T), X, Y)
plt.show()

List item

Just realised the issue - super subtle (at least for a Python novice like me) - I implemented numerical gradient checking (as I should have done from the start) and saw the gradient descent was working incorrectly.
The dataset Y was created as a datatype "uint8". In the line of the back-propagation where I calculate the derivative with respect to "a" ("da") the statement "-Y" was returning 255 for those values of Y that were 1 - this appears to be a feature of these unsigned binary numbers.
By replacing "uint8" with "int8" in the definition of Y - or else forcing the correct behaviour by writing "-1.*Y" in the calculation of "da" (which I think casts "Y" as an integer rather than an unsigned integer) I managed to produce the correct behaviour.
This in turn means the gradient descent is doing what it should and the logistic regression converges to a stable value of the cost. Accuracy then tracks cost as well.


text:
Enforcing sparsity constraints that make use of spatial contiguity
I have a deep learning network that outputs grayscale image reconstructions. In addition to good reconstruction performance (measured through mean squared error or some other measure like psnr), I want to encourage these outputs to be sparse through a regularization term in the loss function.
One way to do this is to add an L1 regularization term that penalizes the sum of the absolute value of pixel intensities. While this is a good start, is there any penalization that take adjacency and spatial contiguity into account? It doesn't have to be a commonly used constraint/regularization term, but even potential concepts or papers that go in this direction would be extremely helpful. In natural images, sparse pixels tend to form regions or patches as opposed to being dispersed or scattered. Are there ways to encourage regions of contiguous pixels to be sparse as opposed to individual pixels?

@DuttaA Interesting idea! Do you have any examples or links that provide more info? Also, is this similar to total variation regularization (which minimizes differences in adjacent pixels)?

Does it matter which pixels (or regions) become zero or can they be in random positions? What are you trying to achieve ultimately, i.e. what kind of images are you trying to generate (or reconstruct)? Also, when you say "sparse", do you mean that the pixels will be zero (black)?


text:
What framework for a project with a custom environment?
I'm planning an RL project and I have to decide which RL framework do I use if any at all. The project has a highly custom environment, and testing different algorithms will be required to obtain optimal results. Furthermore, it will use a custom neural network, not implemented in the popular TensorFlow/PyTorch ML frameworks. Therefore, the framework should allow for customization with regard to approximation function (1) and the environment (2). The problem is that to my current knowledge, most of the framework allows only to work with a built-in environment. Does anybody know a framework that meets the two conditions (1) and (2)? Or anybody knows a review that contains information about framework in the context of those conditions?

Although I asked you to clarify your actual question, I don't think that asking "which library/framework is most suited for my task" is on-topic here, as our on-topic page (https://ai.stackexchange.com/help/on-topic, please, read our on-topic page) says. Given that this is related to reinforcement learning, I will leave it open for a while, but questions that ask for libraries, datasets or GPUs are probably more suited for Data Science SE, but I don't ensure you that they are on-topic there, given I am not familiar with the details of that site.

You can modify the environments and create custom models using Ray RLlib: https://docs.ray.io/en/latest/rllib.html


text:
Multiple Inertia sensors system based for gestures recognition
I am a newbie to Machine Learning field as I am engaging to a personal project that I am trying to use the 6 degree of freedom Inertial Measurement Units(IMUs) measuring the Acceleration acting on 3 axes(x-y-z) and the Angular velocity around the same 3 axis(x-y-z). One sensor generates a set of 6 raw variables of: Acc_x, Acc_y, Acc_z, Gyro_x, Gyro_y, Gyro_z.
Initially I have 2 of those sensors that used to be attached on to the arm (one to the part above the elbow and one to the part bellow the elbow) together they spit out a dataset of 12 raw variables that represent a specific movement of the arm, I save them as a the csv file. This is the point where I really get overwhelmed with a huge amount of data that I don't know how to process this kind of data and extract the features to differentiate the gestures.
My dataset of the first movement I recorded looks like this:

I denoted 1 for the first sensor above the elbow and 2 for the sensor below the elbow.
Looking forward to hearing the opinions from the experts and seniors on this.
Thank you in advanced.
Let me know if my question is inappropriate and lack of information as it is my first time.


text:
Would it be possible to use AI to measure pupil dilation diameters and fluctuation, on video films on a regular webcam?
I've been researching the topic of Cognitive Load Measurement through pupil dilation measurement. All solutions to pupil dilation measurement require some kind of special hardware setup. I was wondering if it would be possible to use AI on a regular webcam record and do those measurements later. If yes, I'd love some pointers to resources of what I need to know to be able to implement it.

The issue requring specialist kit may be a scale/zoom/focus issue from the raw image, as opposed to any smart processing in the specialist kit. A good rule of thumb is if a human could perform the analysis somehow from the image source you want to use, then so could an AI trained to perform the same task. Do you have any links/research that show how to measure pupil dilation from webcam images or similar - perhaps as some kind of post analysis from a video?

Yeah, that was my rationale. By all means, it seems like a task that could be accomplished by a human. From a conference paper:

(continuing)... "A Study on Algorithms of pupil diameter measurement" I saw you could combine algorithms that can perform such tasks in a picture. What I wanted to understand - and maybe it is a layman question - is if there would be ways to optimize those algorithms using machine learning techniques.


text:
Is it possible to have a negative output using only ReLU activation functions, but not in the final layer?
I know that if you use an ReLU activation function at a node in the neural network, the output of that node will be non-negative. I am wondering if it is possible to have a negative output in the final layer, provided that you do not use any activation functions in the final layer, and all the activation functions in the previous hidden layers are ReLU?

Yes, if there's no activation function in the last layer, the weights could simply be negative there, so the network would multiply a positive value with a negative weight, therefore outputting a negative value.
There is still an activation function, but it is the identity.

I guess you are using NN for Regresions.
In the most common aplication a scale of the outputs is implemented.
This is recommended. Specialy if you have more than one output with diferent scales.
Otherwise, you will remunerate the neural network for correcting the error of one variable over the other.
If you still want to avoid a scale of the outputs. Yes. You can use the identity function in the output layer or a linear function (tha same with different slope).
The weights and bias of some conections will become negative and the hidden neurons are going to work as always.


text:
How to implement or avoid masking for transformer?
When it comes to using Transformers for image captioning is there any reason to use masking?
I currently have a resnet101 encoder and am trying to use the features as the input for a transformer model in order to generate a caption for the image, is there any need to use masking? and what would I mask if I did need to?
Any help would be much appreciated
Thanks in advance.

If you're using a library such as Trax which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt trax.data.inputs.add_loss_weights() function and provide a value for the id_to_mask parameter.
Example Usage:
train_generator = trax.data.inputs.add_loss_weights(
data_generator(batch_size, x_train, y_train,vocab['<PAD>'], True),
id_to_mask=vocab['<PAD>'])

Here are some resources for building Transformers in Trax:

Named Entity Recognition with Reformer (Masking Implemented) and the associated blog post (contains an in-depth explanation of NER and Transformers)

Deep N-Gram Models (without masking but I recommend you go through these first to get a hang of the library)

The associated blog post link gives me a 404. Is there an up to date version?

Here's the updated link. Sorry I moved my blog from fastpages

@SauravMaheshkar It seems the link is still not working.

I keep changing blog platforms a lot so the links keep changing. I'll make it a point to add a permalink next time. For now, the link is this.  I apologize but the Markdown parser is still not fixed so the latex is going to be hard to read. If you necessarily need the latex rendered visit this alternative link


text:
Mapping given probabilities to empirical probabilities
Consider following problem statement:

You have given $n$ actions. You can perform any of them. Each action gives you success with some probability. The challenge is to perform given finite number of actions to get maximum successes.

Here, we can perform actions and slowly decide upon possible probabilities of each action for success. I have no doubts in this problem.
Now consider following variant of the problem:

You have given $n$ actions. You can perform any of them. Each action gives you success with some probability. Also you are given set of $n$ probabilities, but you are not told which probability is associated with which action. The challenge is to utilise this additional information to perform given finite number of actions to get maximum successes.

I have doubt in this problem that how we can map probabilities to actions? I can do some enough number of actions to gather empirical probability and them try to associate given probabilties with actions having closest empirical probabities. But, is there any algorithm for such problem in the literature?


text:
Do we need non-linear activation function in neural networks whose task isn't classification?
While researching why we need non linear activation functions, all the explanations revolve around neural network being able to separate values that aren't linearly separable.
So I wonder, if we have a neural network whose task is something else, say predicting an output value of a time series, is it still important to have an activation function that is non linear?

Yes, definitely.
In the simplest example, predicting an output value for a time series is classification. You take in the previous time steps and classify what is the most likely next value. You could do this with a RNN (Recurrent Neural Network) for example.
If the activation functions are all linear, the nerual network is just a glorified linear regression. Think of it like this: a neural network is trying to approximate a complicated function in $n$ dimensional space. It does this by combining operations on a series of known functions, to get a resultant function that hopefully mimics the desired function. The issue with combining linear functions is the only thing you'll ever get at the end is a linear function.
As a concrete example, try and approximate the function $y = x^3 + x^2 -x -1$ by adding a series of linear functions together. You'll find pretty quickly this is useless. However, if you use a non-linear function, such as a ReLU (Rectified Linear Unit) you can quite easily approximate this function. See this implementation on desmos.
If a problem has any sort of complexity to it, the function it follow is likely incredibly complicated, and futile to approximate using linear equations.

Thanks for you answer. I would just like to check if I'm getting this right, would the desmos implementation represent a neural network with 1 input (value of x), 6 hidden neurons with ReLU activation and 1 output neuron (predicted value of the funciton), where the output of the output neuron is just the sum of hidden neuron outputs?

Yep that's exactly it. And you can imagine the values inside the functions as weights and bias'. For example, for $n_1$ the weight would be -5 and the bias -7.7. Then from the hidden layer to the final layer these weights are either -1 or +1 as seen by the mix of positive and negative in the calculation of $o(x)$

I don't know if this belongs to a comment, but would it be possible to show the importance of the bias neuron in a similar fashion (on desmos)?

@user1477107 To see the importance, try matching $y=x^2$ using ReLU, but don't use a bias (as in, you can only multiply x by a value, but can never add to it). You'll notice pretty quick that the issue here is that you are stuck with a max(n, x) function, though perhaps scaled differently. You can never reach any sort of complexity. If you try again, but you are able to add a bias, you can align and move around the functions so that they actually match $y=x^2$. Essentially, a bias lets you translate the function


text:
How do you make a regression model from a binary labeled dataset?
Suppose I have a dataset with hand images. Hand completely opened is labeled as 0 and hand completely closed (fist) are labeled as 1. I also have a bunch of unlabeled images of hands which, if properly labeled would have values between 0 and 1. Because they are not completely opened and not completely closed.

Extra info I have is the ordering between all the pairs of unlabeled images. For example, given image A and B, I can tell you which image should be predicted with higher value, but I cannot tell you what exactly is the value. The unlabeled dataset is collected by recording a video of a closing hand from completely opened to completely closed.
What are some machine learning techniques that I can use to give the label or predict the values of hands not completely closed and not completely opened? I expect it to be based on ordering or ranking system. If it doesn't even require the ordering label (A > B?) then that would be a very smart algorithm.
I want the values between 0 and 1. If there's a code for it that would be a plus. Thank you.


text:
How much overfitting is acceptable?
I have a deep learning configuration in which I obtain good results on the validation set but even better results in the training set. From my understanding this means that there is overfitting to some extent. What does this mean in practice? Does it mean that my model is not good and that I should not use it? If I decrease the gap between the validation and training accuracy (decreasing the overfitting) but at the same time decrease the validation accuracy, which of the two models is better?
Below are some images to illustrate the two situations outlined previously:

Higher validation is almost always better. If you think about it, the best network in regards to keeping the difference between training and validation is one that's terrible, because it will perform equally bad on both. If you're validation accuracy is increasing, you can be confident that if the network is over-fitting to the training data, it isn't to the point where it impacts performance. There will always be some level of over-fitting, but if validation accuracy continues to increase, that's ok.

This is not overfitting, but the (always expected, in some degree) generalization gap.

Validation results will almost never be as good as training results; that's just natural. As long as they are not too different, you should be fine. What "too different" means depends on the particular data set and model you're using.
If you plot the curves for varying parameter values, when the training error keeps going down but the validation error starts going up again, that's when you know there is overfitting. In your second graph, after 14 epochs, we might see the start of overfitting. If you continue this until 20 epochs or so, it should be even more clear. I would guess that 12 is probably a good value for the number of epochs for that problem.
In the first graph, we don't see that happening yet. The model might not be well-suited (the gap between training and validation results is a bit larger)  but that can also be because of too little data, or other factors. Perhaps that's just the best you can do; there might be noise in the data or something.


text:
Does replacing 3x3 filters with 3x1 and 1x3 filters improve the performance?
Recently I have come up with a VGG16 model for my binary classification task. I have relatively simple signal images

Therefore (maybe?) other deeper models like resnet18 and Inceptionv3 were not as good. As known, VGG uses 3x3 filters for convolving the images to make feature maps. I have tried several hyper-parameters to get a desired performance. However, there are still some things I need to do. I was thinking of replacing the 3x3 conv filters with 3x1 followed by 1x3 filters to reduce the compute. I think it will definitely do so considering the multiplications (9 operations for 3x3and 6 for 3x1 followed by 1x3).
Then I came to think: If I replace all the 3x3 filters with separable filters, will I get any performance improvement?
What are the benefits of replacing 3x3 filters with separable ones?
Thanks

Do you have to use an image of the signal plot like your example here? Do you have access to the raw signal data instead?

yes I can only use images like shown above. I wish I had a raw signal data.

If the filter is separable, that is, the NxM kernel can be mathematically equal to the convolution of a Nx1 filter and a 1xM filter, there are a very important increase in performance.
Using separable convolution, the network can made an optimal usage of the shared memory and of the parallelism in memory access. See this excellent article for details. These improvements are bigger for bigger kernels.
Also the training is improved, starting by the simple fact that a NxM filter has a number of parameters proportional to N*M but the related separable one has N+M.

the linked document doesn't provide any info that separable filters increase the performance, although it does say that it increases efficiency.

@bit_scientist: see chapters "Separable Filters Can Increase Efficiency" and "Performance" (this one with a comparative graphic againts one of the other usual methods, convolution texture)

First of all, Keep in mind that maths operations aren’t the only thing that contribute to performance. Memory bandwidth can also be a factor.
And most importantly, we want to capture as much area as we can in lowest possible number of operations. So in 3x3 kernel case, we can capture 9 cells in one shot, but with 3x1 followed by 1x3, we have to compute 6 times to capture 9 cells. Which clearly states that 3x3 kernel is far more efficient than these two sequential kernels.
So, answer to your question will be no, it will not improve performance, instead it will increase computation overhead for your system.

" but with 3x1 followed by 1x3, we have to compute 6 times": no. "3x3 kernel is far more efficient than these two ": no. I think you must review what is a separable convolution.


text:
How to quantify the amount of information lost by the decoder NN in an AE?
Is there a way to quantify the amount of information lost in the lossy part of an autoencoder where the original input is compressed to a representation with less degrees of freedom?
I was thinking maybe to use somehow the mutual information either in the image or frequency domain.
$$
\mathrm{I}(X ; Y)=\sum_{y \in \mathcal{Y}} \sum_{x \in \mathcal{X}} p_{(X, Y)}(x, y) \log \left(\frac{p_{(X, Y)}(x, y)}{p_{X}(x) p_{Y}(y)}\right)
$$
where $p_{(X,Y)}$ is the joint probability density function which is deduced somehow empirically from a set of $N$ input and output to the network.
Maybe it's not even an interesting question since the loss function evaluates exactly that?


text:
Why is sine activation function not used frequently since we know from fourier transforms that sine functions can combine to fit any function?
Pretty much the title.
I'm no expert but from what I know, if you add up enough sine functions with proper amplitudes and frequencies you can get any function you want as a result. With that knowledge, wouldn't it make sense to have neuron's activation function be a sine function?

There was once a similar question, which I answered by saying that the sine is periodic, and that may not be desirable. Doesn't the Fourier transform assume that you have like complex coefficients or something? I don't recall anymore.


text:
What is a wavefront algorithm?
I am designing and researching algorithms which I call of a wavefront nature. It is image analsyis agorithms when every pixel may change many times during the processing. I have heard this name before, but it seems it is not used widely. Are there other "wavefront algorithms"?


text:
Relation between a value function of an MDP and a value function of the corresponding latent MDP
In paper "DeepMDP: Learning Continuous Latent Space Models for Representation Learning", Gelada et al. state in the beginning of section 2.4

The degree to which a value function of $\bar{\mathcal M}$, $\bar{V}^{\bar\pi}$ approximates the value function $V^\bar\pi$ of ${\mathcal M}$ will depend on the Lipschitz norm of $\bar V^\bar\pi$ .

where $\mathcal M$ is the Markov Decision Process(MDP) defined in the original state space $\mathcal S$ and $\bar{\mathcal M}$ is the MDP defined in the corresponding latent space $\bar{\mathcal S}$. $\bar\pi$ is a policy defined on the latent space, which can be applied to $\mathcal M$ by first mapping $s\in \mathcal S$ to $\bar{\mathcal S}$.
My question is how they draw the connection between "The degree to which a value function of $\bar{\mathcal M}$, $\bar{V}^{\bar\pi}$ approximates the value function $V^\bar\pi$ of ${\mathcal M}$" and " the Lipschitz norm of $\bar V^\bar\pi$"?


text:
How does Hartigan & Wong algorithm compare to Lloyd's and Macqueen's algorithm in K-means clustering?
As far I know, this is how the latter two algorithms work...
Lloyd's algorithm

Choose the number of clusters.
Choose a distance metric (typically squared euclidean).
Randomly assign each observation to a cluster and compute the cluster centroids.
Iterate below until convergence (i.e. until cluster centroids stop changing):

Assign each observation point to the cluster whose centroid is closest.
Update cluster centroids only after a complete pass through all observations.

Macqueen's Algorithm

Choose the number of clusters.
Choose a distance metric (typically squared euclidean).
Randomly assign each observation to a cluster and compute the cluster centroids.
Perform a complete pass of below (i.e. go through all observations):

Assign an observation to a cluster whose centroid is closest.
Immediately update the centroids for the two affected clusters (i.e. for the cluster that lost an observation and for the cluster that gained it).

Update centroids after a complete pass.

How does the Hartigan & Wong algorithm compare to these two above? I read this paper in an effort to understand but it's still not clear to me. The first three steps is the same as Lloyd's and Macqueen's algorithm (as described above), but then what does the algorithm do? Does it update the centroids as often as Macqueen's algorithm does, or as often as Lloyd's algorithm does? At what point does it take into consideration the within-cluster sum of squares and how does it fit into the algorithm?
I'm generally confused when it comes to this algorithm and would very much appreciate a step-wise explanation as to what's going on.


text:
Why weighting by lambda that sums to 1 ensures convergence in eligibility trace?
In Sutton and Barto's Book in chapter 12, they state that if weights sum to 1, then an equation's updates have "guaranteed convergence properties". Actually why it ensures convergence?
There is a full citation from the mentioned fragment in Richard S. Sutton and Andrew G. Barto. Second Edition:

Now we note that a valid update can be done not just toward any n-step return, but toward any average of n-step returns for different ns. For example, an update can be done toward a target that is half of a two-step return and half of a four-step return: $\frac{1}{2}G_{t:t+2} + \frac{1}{2}G_{t:t+4}$. Any set of n-step returns can be averaged in this way, even an infinite set, as long as the weights on the component returns are positive and sum to 1. The composite return possesses an error reduction property similar to that of individual n-step returns (7.3) and thus can be used to construct updates with guaranteed convergence properties.

authors: محمدعباس الخيواني


text:
How are Energy Based models really connected to Statistical Mechanics?
From statistical mechanics, the Boltzmann distribution over a system's energy states arises from the assumption that many replicas of the system are exchanging energy with each other.  The distribution of these replicas in each energy level is the maximum entropy distribution subject to the constraint that their total energy is fixed, and that any one assignment of energy levels to each replica, a "microstate", satisfying this constraint, is equally probable.
From machine learning, the so-called Energy-based model defines a Hamiltonian (energy function) to its various configurations, and uses Boltzmann's distribution to convert an "energy" to a probability over these configurations.  Thus, an EBM can model a probability distribution over some data domain.
Is there some viewpoint by which one can interpret the EBM as a "system" exchanging energy with many other replicas of that system?  What semantic interpretation of EBMs connects them to the Boltzmann distribution's assumptions?


text:
How to let the agent choose how to populate a state space matrix in RL (using python)
I have an agent (drone) that has to allocate subchannels for different types of User Equipment.
I have represented the subchannel allocation with a 2-dimentional binary matrix, that is initialized to all zeros as there is no requests at the beginning of the episode.
When the agent chooses an action, it has to choose which subchannels to allocate to which UEs, hence populating the matrix with 1s.
I have no idea how to do it.

Can you please be a little more intuitive on what you would like to do? What is a subchannel, UE, and in general what a human being would face in this problem. Describe a use case scenario. This would help to define our env, space and actions.

To extend ddaedalus' comment, it would help to understand what the goals of the allocation are, and what the difference is between good and bad allocations

I think this question is hinting at the problem of choosing an exploration strategy.
The simplest strategy is to use the so called epsilon-greedy strategy (or $\epsilon$-greedy). This means that you select an action at random $x$ percent of the times that an agent has to select an action. The other times, the agent takes the action that its current policy dictates. Usually, $x$ declines throughout the learning process.
There are many different exploration strategies to choose from, with the aforementioned one being really the most simple one. For getting an impression of how diverse these different approaches to exploration strategies are, please be referred to Google Scholar (searching for "reinforcement learning exploration strategies").

It might be right, but I think you may have jumped in too early with an answer here. It is not clear to me that the op is asking about exploration strategy, they appear to be stuck earlier in the design process


text:
What is human-level performance for semantic segmentation?
I see so many papers claim to have an algorithm that beats 'human-level performance' for semantic segmentation tasks, but I can't find any papers reporting on what the human-level performance actually is. An analysis of the similarity between segmentations drawn by multiple different human experts would be good. Could someone point me towards a paper that reports on something like that?


text:
Why don't ensembling, bagging and boosting help to improve accuracy of Naive bayes classifier?

You might think to apply some classifier combination techniques like ensembling, bagging and boosting but these methods would not help. Actually, “ensembling, boosting, bagging” won’t help since their purpose is to reduce variance. Naive Bayes has no variance to minimize.

The above paragraph is mentioned in this article.

How can they say the purpose of these methods is to reduce variance?
Naive Bayes has no variance, is it true?

Thanks in advance


text:
Should I prefer cropped images or realistic images for object detection?
I am new to the field of AI but due to the high level of abstraction that comes with services such as Google VisionAI I got motivated to write an application that detects symbols in photos based on tensorflow.js and a custom model trained in Google Vision AI.
My App is about identifying symbols in photos, very similar to traffic signs or logo detection. Now I wonder if

I should train the model based on real, distorted and complex photos that contain those symbols and lots of background noise
if it was enough to train the model based on cropped, clean symbols
A hybrid of both

I started with option a and it works fine, however it was a lot of work to create the training dataset. Does the model need the distorted background to work?

In general, you should choose the photo as close to your test case as possible. Take a look at Machine Learning Yearning of Andrew Ng, in part 6, he described a scenario that the performance in dev set is really good but very poor when implementing in real life (test set). My advice is to choose your data carefully, make the variance enough to handle all situations and the distribution is balanced to prevent overfitting to one class only.

Google recommandation seems to answer this:

The training data should be as close as possible to the data on which
predictions are to be made.
For example, if your use case involves blurry and low-resolution
images (such as from a security camera), your training data should be
composed of blurry, low-resolution images. In general, you should also
consider providing multiple angles, resolutions, and backgrounds for
your training images.

https://cloud.google.com/vision/automl/object-detection/docs/prepare
Would you agree, also in the case of symbols?

If you know how your test images are, there is no point in training your neural network on images with different settings (e.g. different dimensions or resolutions). Your ultimate goal is to perform well on the test. Of course, if the test set  (or distribution) was to change, then you should also take this into account. The part "you should also consider providing multiple angles, resolutions, and backgrounds for your training images." is about data augmentation, which can help your neural network to learn more "robust" representations.


text:
Training a CNN for semantic segmentation of large 4600x4600px images
I am trying to implement a CNN (U-Net) for semantic segmentation of similar large grayscale ~4600x4600px medical images. The area I want to segment is the empty space (gap) between a round object in the middle of the picture and an outer object, which is ring-shaped, and contains the round object. This gap is "thin" and is only a small proportion of the whole image.
In my problem having a small a gap is good, since then the two objects have a good connection to each other.
My questions:

Is it possible to feed such large images on a CNN?
Downscaling the images seems a bad idea since the gap is thin and most of the relevant information will be lost. From what I've seen CNN are trained on much smaller images.

Since the problem is symmetric in some sense is it a good idea to split the image in 4 (or more) smaller images?

Are CNNs able to detect such small regions in such a huge image? From what I've seen in the literature, mostly larger objects are segmented such as organs etc.

I would appreciate some ideas and help. It is my first post on the site so hopefully I didn't make any mistakes.
Cheers


text:
Are Autoencoders for noise-reduction only suited to deal with salt-and-pepper kind of noise?
I'm currently looking at NN to deal with noisy data. I like the Autoencoder approach https://medium.com/@aliaksei.mikhailiuk/unsupervised-learning-for-data-interpolation-e259cf5dc957 because it seems to be adaptive and does not require to be trained on specific training data.
However, as it is described in this article it seems to rely on having none-noise samples in the input data that are true to the ground truth, so I wonder if an autoencoder also could work in the case of white or blue noise instead of salt-and-pepper noise?

The autoencoder proposed in the link is similar to a Denoising autoencoder (DAE), in sense that both starting from a noisy image try to reconstruct the original image. The difference is that the noisy pixel are ignored during the backpropagation. The DAE takes in input an image, where before the forward phase noise is applied on the image. On output the DAE have to reconstruct the image without noise.
So, given an image $I$ and $\epsilon \sim \mathcal{N}(0,\,\sigma^{2})$ the noise is applied to the image obtaining the image $\hat{I}$. Now, $\hat{I}$ is the DAE input and the autoencoder is trained to minimize the function $L(I,g(f(\hat{I})))$. Where $f$ is the encoder and $g$ is the decoder.

The image on the left is an image of the MNIST dataset where $\epsilon \sim \mathcal{N}(0,\,0.3)$ and on the right there is the reconstruction of a deep DAE. You can change the type of noise in the image, the results should be the same. So the answer to your question is no, autoencoder are suited also for other type of noise.
If interested you can read the paper "What Regularized Auto-Encoders Learn from the Data-Generating Distribution" Guillaume Alain, Yoshua Bengio where is shown that the DAE is learning an approximation of the gradient of the data distribution.

Hello. Welcome to AI SE. Thanks for contributing to our site (and maybe, if you have some time, take a look at https://ai.stackexchange.com/help/on-topic to know more about our scope) :) Anyway, it seems that you do not directly address the question "Are Autoencoders for noise-reduction only suited to deal with salt-and-pepper kind of noise?".


text:
speech comment detection by deep speech mozilla for data set
I want to create a system so that when a human being says a word or command through a microphone, such as "shut down", the system can execute that command "shut down".
I used the Deep Speech algorithm on the Persian language database, which takes audio data through a microphone and returns the text. The problem I have now is what I have to do from this point on. I need to explain that my system has to work offline and also the number of commands I have is limited.

The problem you state is a well known problem, and it is called "keyword spotting" os KWS. If you add a wake up word before it (like "hey google/siri"), you can also use "voice command" system to alleviate the problem.
There are two kind of KWS systems: those which develop to detect a hard coded set of keywords, and those who are flexible to amend the keyword set. As you state that you trained an acoustic model using deep speech, it looks that you choose the latter approach.
If you trained a deep speech model, you done the easy half of the solution. The other half which is more tricky is to develop a search algorithm based on your acoustic model. I think wfst based algorithms and prefix beam search algorithms are what you should be looking for.
Regards


text:
Machine Learning Techniques for Objects Location/Orientation in Images
what Machine Learning tool can understand in which location and orientation a picture was taken from?
That is from pictures of similar objects, say for example pictures of car interiors.
So given a side vent picture it will show me all the pictures with a side vent with similar views (if a picture shows a vent from the cockpit it will not show me pictures of vents taken from outside the car with open door).
If the problem is too complicated for just one tool, could you address me to which particular field of Artificial Intelligence should I research in?
this goes close to what I am looking for:
https://ai.googleblog.com/2020/03/real-time-3d-object-detection-on-mobile.html
But I thought to ask if there are more specific appropriate examples.


text:
What is the status of the capsule networks?
What is the status of the capsule networks?
I got an impression that capsule networks turned out not to be so useful in applications more complicated than the MNIST (at least according to this reddit discussion).
Is this really the case? Or can they be a promising research direction (and if so, is there any specific application for which they seem the most promising)?

ML is full with things that supposed to work better (in theory).

Sigmoid function seems better than ReLu.

L1 seems way better than L2.

Spikes neural network seem to be better than standard neural network.

A shallow neural network with a lot of neurons has more parameters than a deep one with the same amount of neurons. So, in theory, has to be more powerful.

Is common to forget the big impact of the training capacity. And easy to get charmed by its theoretical capacity.
Training process are still in diapers. We lack a good understating of it.
Today, Capsules neural network are a very powerful architecture, hard to train.


text:
Why is automated theorem proving so hard?
The problem of automated theorem proving (ATP) seems to be very similar to playing board games (e.g. chess, go, etc.): it can also be naturally stated as a problem of a decision tree traversal. However, there is a dramatic difference in progress on those 2 tasks: board games are successfully being solved by reinforcement learning techniques nowadays (see AlphaGo and AlphaZero), but ATP is still nowhere near to automatically proving even freshman-level theorems. What does make ATP so hard compared to board games playing?

What makes you think that automated theorem proving is very similar to board games? Games have fixed set of rules and clear rewards, I can't see how that is the point for theorem proving?

I am not familiar with the state-of-the-art of ATP, but, to support your assumptions/conclusions, this post would benefit from having a link to a research paper or book that mentions or describes the state-of-the-art in this domain. Btw, welcome to AI SE :)

There are two ways to look at the problem, one in terms of logic and the other in terms of psychology.
To get any -start- on automation of mathematics, you need to formalize the part you want. It has only been since the early part of the 20th c that most day to day math had been formalized with logic and set theory. And even though Gödel's incompleteness theorems say (very loosely) that there is no algorithm to decide theorem-hood for mathematical statements (that include a theory of arithmetic), that still leaves a lot of math that -can- be decided. But that has taken the Reverse Mathematics program (still ongoing) to say specifically what subsets of math are decidable or to what degree (what logical assumptions are necessary) they are undecidable.
So theorems in arithmetic of just '+' (that is, dropping '*') can be decided, Euclidean geometry can be decided, single variable differential calculus can be decided but not single variable integral calculus. These examples show that what we know to be decidable is pretty elementary. And most of the things we care about are very un-elementary (almost by definition).
As to psychology, the theorems and proofs that you learn in mathematics classes are nowhere near like their formalizations. Most mathematicians aren't pushing symbols around in their heads like a computer does. A mathematician is more like an artist, visualizing dreams and connecting metaphors just on their barely conscious images borne out of repetition. That is, machines and mathematicians just work on different representations (despite what non-mathematicians might imagine).

To address your specific question, yes, mathematical theorems and the systems to prove them are very similar in a technical sense. Games (often, not always) can be modeled as trees, And likewise proofs can often be modeled as trees. Without writing you a library of books about games and proofs, let's just say that the mathematical proofs that are like games that are won by Alpha Zero are not for particularly interesting theorems. Winning a game of go is more like proving that a very very large boolean formula. Most mathematical theorems require a lot of ingenuity in introducing steps in their proof trees. It may be mechanical after the fact to check that a proof is correct, but discovering the proof almost needs magic to come up with a step in the game. Sure, some things in math are automatable (as mentioned before, derivatives), but some mathematical systems (such as integration) are provably impossible to find proofs of all true statements.
Another difference between theorem proving and games is that proofs have to be air tight on all paths, whereas with games one side just has to eke out a single win over the other side.

A separate issue entirely that may contribute to the difficulty is that we just may not yet have the tooling available, ie editors, notation, proof assistants that make it easy to do what should be easy. Or it could just be that mathematicians don't have the fluency with theorem proving systems.
Or it could be that if there were automated theorem provers good enough, mathematicians just wouldn't care too much for them because they'd take away the fun of finding the proofs themselves.

The first part of this answer is interesting, but the second part would really benefit from some facts: i.e. what is the state-of-the-art in automated theorem proving (maybe some cite some research paper that surveys the SOTA tools and results in this field)?

@nbro Are you looking for facts about the psychology part or about decision theories for subsystems of mathematics?

Facts about the state-of-the-art in automated theorem proving. For example, you could say something like "with current systems, we were able to prove this conjecture $X$" (or something like that), and then you cite a research paper (or something reliable) that supports that claim. It may also be a good idea, if you find something, to link us to a research paper/book that describes more in detail what are the approaches and why they are not good.

@nbro Sure... I can only hint at such things (like I've already done). But yes I'll attempt to clarify. (sadly the OP probably won't know to follow up here on AI but it is still a really good AI question. Hopefully others will chime in.

I'm curious if we get to the point where an AI can prove everything a human can prove including some uninteresting theorems beyond a humans reach but the AI is unable prove anything new/uninteresting. It's one thing to train something to be good enough at what we already know, I think its an even more difficult problem to design a system to synthesize BEYOND what we know/could do (though I don't doubt it will be possible one day)


text:
Is training a CNN object detector on an image containing multiple targets that are not all annotated will teach it to miss targets?
I want to train a convolutional neural network for object detection (say YOLO) to detect faces. Consider this image:

In this training image, I have many people, but only 2 of them are annotated. Is having this kind of images (where target classes are not all annotated) will train the network to ignore positives?
If yes, are there any techniques to solve the issue apart from annotating the data (I don't have enough resources to do that).

The neural network will learn what we teach it, for example with that image only, after finish training, your model will hard to recognize humans with dark skin, glasses, big eyes, etc, the features that two annotated targets don't have.
If your data is big enough, and contain all the feature of humans face, the result should be good.
If not, I recommend a semi unsupervised learning method which called Noisy Student. Quick explanation, you take a part of data, add noise (augmentation, drop-out, stochastic depth), and train. Then use that model to label the rest of the dataset and train a new bigger teacher (such as YOLOv3 > YOLOv1) and repeat it. They told that this method was even better than you labeled all the dataset.
You will need to choose the data to train the teacher really carefully, then pray for the result.


text:
What is the advantage of using cross entropy loss & softmax?
I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some errors, but I had a few more ideas. For one, I am using the sigmoid activation function and MSE as an error function, but the internet suggests that I should rather use softmax for the output layer, and cross entropy loss as an error function. Now I get that softmax is a nice activation function for this task, because you can treat the output as a propability vector. But, while being a nice thing to have, that's more of a convinience thing, isn't it? Easier to visualize?
But when I looked at what the derivative of softmax & CEL combined is (my plan was to compute that in one step and then treat the activation function of the last layer as linear, as not to apply the softmax derivative again), I found:
$\frac{δE}{δi}$ = $t$ − $o$
(With $i$ being the input of the last layer, $t$ the one hot target vector and $o$ the prediction vector).
That is the same as the MSE derivative. So what benefits does softmax + CEL actually have when propagating, if the gradients produced by them are exactly the same?

Where did you find that derivative? Also, even if that's the correct derivative (I didn't think about it and I don't want to think about it now), note that, if $o$ is the prediction vector, then $t - o$ is different if $o$ is a probability vector or an unnormalised one.

Short answer: larger gradients
That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to simplify the derivative of the softmax function. In the end, you do end up with a different gradients. It would be like if you ignored the sigmoid derivative when using MSE loss and the outputs are different. Using softmax and cross entropy loss has different uses and benefits compared to using sigmoid and MSE. It will help prevent gradient vanishing because the derivative of the sigmoid function only has a large value in a very small space of it. It is similar to using a different cross entropy loss where the combined derivative of the loss and sigmoid is $t - o$.
Information on derivatives of cross entropy with sigmoid function and with softmax function. I would also suggest some more research on cross entropy loss functions beyond my links.

so cross entropy makes loss convergence faster, compared to mse?

@datdinhqouc When the last layer is the sigmoid function then cross entropy should make loss convergence faster and prevent vanishing gradients.

If you look at the definition of the cross-entropy (e.g. here), you will see that it is defined for probability distributions (in fact, it comes from information theory). You can also show that the maximization of the (binomial/Bernoulli) log-likelihood is equivalent to the minimization of the cross-entropy, i.e. when you minimize the cross-entropy you actually maximize the log-likelihood of the parameters given your labelled data. Hence the use of the softmax is theoretically founded.
Regarding the supposed derivative of the cross-entropy loss function preceded by the softmax, even if that derivative is correct (I didn't think about it and I don't want to think about it now), note that then $t - o$ is different depending on whether $o$ is a probability vector or an unnormalized vector (which can take arbitrarily large numbers). If $o$ is a probability vector and $t$ a one-hot encoded vector (i.e. also a probability vector), then all numbers of $t - o$ will be in the range $[-1, 1]$. However, if $o_i$ can be arbitrarily large, e.g. $o_i = 10$, then $t_i - o_i \in [-10, -9]$. So, the propagated error would be different if $o$ was not a probability vector.


text:
How to train the NN of simple agents given a reward system?
I'm not an expert in AI or NN, I gathered most of the information I have from the internet, and I'm looking for advice and guidance.
I'm trying to design a NN that is going to be used by all the agents of my simulation (each agent will have its own matrix of weights). This is what I plan to have:

The NN will have 1 input layer and 1 output layer (no hidden layers).
The number of inputs will always be greater than the number of outputs.
The outputs represent the probability of an action being taken by the agent (the output node with the highest value will identify the action that will be taken). Which means there are as many output nodes are there are actions.

When an agent takes an action it receives a reward: a number that represents how well the agent performed. This happens "online" that is, the agent is trained on the fly.
What I would like to know if how to best train the NN: that is, how to update the weights of my matrix to maximise the rewards long term.
From the research I made it seems this is close to the concept of Reinforcement Learning, but even if it was, it's not clear to me how to apply it to such a simple NN shape.

You might be able to glean what you want from Chapter 13 or Sutton & Barto's Reinforcement Learning: An Introduction, which deals with policy gradient algorithms, and includes pseudocode for a variety of agents based on linear approximation using softmax regression. From your description, you appear to be using - or should consider - softmax regression for the policy function.
Probably a good bet using your setup will be REINFORCE with baseline, which does not add requirements to track anything more than your current weights, sum of rewards seen after each state & action pair, and mean rewards (or other baseline). It does require an episodic environment though, such as a game that ends.
Essentially REINFORCE and its variants require you to figure out the return (sum of rewards received after each action to the end of the episode) and perform a gradient step update multiplied by that return (minus a baseline which can help with stability).
With REINFORCE and a discrete distribution, the gradient due to action selection is the same as supervised learning for multi-classification evaluated as if whichever action the agent took was the correct one. When you multiply this gradient by the return as well, it will make steps in the direction of better action choices larger. Regardless of the fact that you may train towards selecting many different actions, the better actions will win the race in the end due to this weighting, and have the highest probability of being selected after a large number of training episodes.
A multi-agent problem may bring you some problems, such as instability due to dependency of performance on what the other agents are doing. It is not possible to say how much of an issue this will be. However, I would advise not jumping straight into your main problem, and try some reinforcement learning methods with a single agent on toy problems first. If you build up to solving the problem you are currently facing, then you will be much more confident in how you have implemented the learning. There are plenty of stepping-stone projects and pre-built environments to solve if you search for them. A good starting point is OpenAI's Gym, which presents a standardised set of toy - and not so toy - problems that you can test any learning algorithm against.


text:
How to use residual learning applied to fully connected networks?
Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional?
I've read the ResNet paper and it says that the applications should extend to "non-vision" problems, so I decided to give it a try for a tabular data project I'm working on.
Try 1: My first try was to only skip connections when the input to a block matched the output size of the block (the block has depth - 1 number of layers with in_dim nodes plus a layer with out_dim nodes :
class ResBlock(nn.Module):
    def __init__(self, depth, in_dim, out_dim, act='relu', act_first=True):
        super().__init__()
        self.residual = nn.Identity()
        self.block = block(depth, in_dim, out_dim, act)
        self.ada_pool = nn.AdaptiveAvgPool1d(out_dim)
        self.activate = get_act(act)
        self.apply_shortcut = (in_dim == out_dim)

    def forward(self, x):
        if self.apply_shortcut:
            residual = self.residual(x)

            x = self.block(x)
            return self.activate(x + residual)
        return self.activate(self.block(x))

The accompanying loss curve:

Try 2: I thought to myself "Great, it's doing something!", so then I decided to reset and go for 30 epochs from scratch. I don't have the image saved, but this training only made it 5 epochs and then the training and validation loss curves exploded by several orders of magnitude:

Try 3: Next, I decided to try to implement the paper's idea of reducing the input size to match the output when they don't match: y = F(x, {Wi}) + Mx. I chose average pooling in place of matrix M to accomplish this, and my loss curve became.

The only difference in my code is that I added average pooling so I could use shortcut connections when input and output sizes are different:
class ResBlock(nn.Module):
    def __init__(self, depth, in_dim, out_dim, act='relu', act_first=True):
        super().__init__()
        self.residual = nn.Identity()
        self.block = block(depth, in_dim, out_dim, act)
        # squeeze/pad input  to output size
        self.ada_pool = nn.AdaptiveAvgPool1d(out_dim)
        self.activate = get_act(act)
        self.apply_pool = (in_dim != out_dim)

    def forward(self, x):
        # if in and out dims are different apply the padding/squeezing:
        if self.apply_pool:
            residual = self.ada_pool(self.residual(x).unsqueeze(0)).squeeze(0)
        else: residual = self.residual(x)

        x = self.block(x)
        return self.activate(x + residual)
)

Is there a conceptual error in my application of residual learning? A bug in my code? Or is resididual learning just not applicable to this kind of data/network?


text:
Root finding in Deep Equilibrium Models
In the Deep Equilibrium Model the neural network can be seen as "infinitely deep". Training learns a nonlinear function as usual. But there is no forward propagation of input data through layers. Instead, a root finding problem is solved when data comes in.
My question is, what is actually the function for which roots are searched, I'm struggling to see what would be unknown when data is available and parameters have been found in training?


text:
Is Monte Carlo tree search guaranteed to converge to the optimal solution in two player zero-sum stochastic games?
I'm aware that convergence proofs for Monte Carlo tree search exist in the case of deterministic zero sum games and Markov decision processes.
I have come across research which applies MCTS to zero-sum stochastic games, however I was unable to find proof that such an approach is guaranteed to converge to the optimal solution.
If anyone is able to provide references or an explanation explaining why or why not MCTS is guaranteed to converge to the optimal solution in this setting I would appreciate it a lot.


text:
How Restricted Boltzman Machine (RBM) generates hand-written digit?
I am reading RBMs from this paper. In Fig1 they show an example of generating hand-written digit using RBMs. This is the figure they are showing:

In the learning step first we sample $h$ from $h \sim P(h|v)$ and then reconstructe $v$ from $v \sim P(v|h)$. Now in the generating step they are sampling $v$ from $v \sim P(v|h)$. My question is, in generating step if we do not sample $h$ from $h \sim P(h|v)$ how can we get $P(v|h)$?


text:
Is it possible to know the distance objects are from camera based on only knowing one object's height?
I am doing a project where I have to know distance a particular object is from camera. In the photo I only know one of the object's height, but I don't know how far away that object is and I don't know how tall are other objects. Is it possible to write a code or do some geometry to know other objects distances from camera using only the height of one object? For example I have an image where 5 meters away there is a box which is 1 meter high, I wanna know the distance to human who is 12 meters away, or to know a distance to a dog who is 7 meters away. Maybe you guys know any datasets or models which deal with the same problem as I am facing. Any help will be appreciated.

No, it is impossible. Assume you see a circle, than in the image has 50 pixels of diameter (and a focal angle of 45º). The circle can have a real size of 1 meter and be at 1 meter distance, or 2 meters and be at 2 meters distance, ... . Impossible to know the real values.

Monocular depth estimation basically does this and it implicitly brings in knowledge about object size. But you need to have prior information.