text:
How much can/should the non-player character know about the game's world?
I'm about to write a non-player character (NPC). I wonder how much the AI should know about the game's world. So, my question isn't about the amount of training data the AI has to collect. I'm interested in how much the AI is allowed to know about what's going on in the game's world.
For example, can (shall) it have knowledge about the build queue of the player?
To provide more details: while a human plays a game against another human, not all information of what the opponent is doing is available (e.g. the queue of the units your opponent is building). This could give you an advantage (so that you can prepare for a rush, when he's building many cheap units). Theoretically, an NPC could access and make use of that knowledge and, in addition, spare resources for scouting/spying/exploring.
But is this the way of constructing an NPC AI? Or should this data also be restricted? I have never done anything like this before.
I don't know where to ask else wise or what more information I could provide. So, if something in my question is unclear or unfit, please let me know what exactly.

I am not an expert on the topic, but my intuition says that you could have different versions of the AI with different levels of difficulty, which the human player can set (or could be aware of). However, I think you should wait for the answer of someone that has some experience in the field.

Art of NPC creation
I'm assuming this is a standard game, not a game theory application. These AIs tend to be far simpler in theory than actual artificially intelligent agents which are used to solve real world problems. The challenge of building games is that there are few right answers. A game NPC opponent could know practically nothing of the player and be guided by random world exploration activating certain characteristics (attack, strategy) whenever a human is in range or on the other end could know basically everything and have some limiting factors built into the code which reduces their challenge level to the player. This is the art aspect of gaming design. NPCs with a high level of complexity could learn from their interactions and build a "pattern" with some internal reward scoring system for success (similar to reinforced learning). This would represent a very challenging game NPC, especially if the strategy changes upon failure. Think about who will be playing your games, the audience that you would tend to draw, and hone the NPC challengers to them. Hardcore gamers deserve a challenge, and hobby gamers deserve a chance to win. Where that balance between rewarding game experience and challenging game experience lies depends on the 'feel' which you are cultivating as the game designer. You mentioned resources and exploration as elements of the game. For exploration types of games, you may rely on environmental interactions to offset player interactions. A distracted NPC (gathering resources) is less likely to heavily attack the human player unless this player enters the perimeters of the NPC base or scouting area. This may work to keep the gameplay balanced. Keep in mind that some limitations for your NPCs must be absolute, or the game will experience unexpected behavior (bugs). A completely unrestricted NPC is a liability. If the NPC goes too far from the player or gets stuck, it completely ruins the feel of your game.


text:
Is batch learning with gradient descent equivalent to "rehearsal" in incremental learning?
I am learning about incremental learning and read that rehearsal learning is retraining with old data. In essence, isn't this the exact same thing as batch learning (with stochastic gradient descent)? You train a model by passing in batches of data and redo this with a set number of epochs.
If I'm understanding rehearsal learning correctly, you do the exact same thing but with "new" data. Thus, the only difference is inconsistencies in the epoch number across data batches.

In rehearsal, you do not necessarily train with all old training data, but you can just use some of it [1], which you add to your current (or new) training data.
In batch learning, at every epoch, you typically train with all training data, every step with a different batch (or subset) of the training data; so, if you have $N$ training examples and your batch size is $M$, then you will have $\lfloor N/M \rfloor $ gradient descent steps.
There is also pseudo-rehearsal, which is an alternative to rehearsal that is useful when you may not have access to the previously used training data [1] (for example, because it is expensive to store).

TODO: I really need to explain what "pseudo-rehearsal" is, which I don't do it now.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How do I write production systems?
I understand that I can draw a state-space graph for any problem. However, here is the problem: I can't really figure out how to make production systems.
I am solving the FWGC (Farmer, Wolf, Goat, Cabbage) River Crossing Puzzle using a state-space search. So, my tasks are that:

Represent the state-space graph (which I know how to do)

Write production systems.

My questions: How do I write production systems?
The thing that confused me, was the production system example in Rich's book (about the water jug problem), where he has imagined all the states possible and wrote the next state for them.
Here in the FWGC problem, I see some problems while writing the production system.
For instance, for a given state, there are multiple possible next states, i.e. a farmer can take Goat, Cabbage, Wolf, or go alone to the other side (assuming that all states are safe, just for the sake of simplicity).
So, how would I represent the same state going to multiple next states in production systems?
What I have tried-:
Then, I googled a pdf
https://www.cs.unm.edu/~luger/ai-final2/CH4_Depth-.%20Breadth-,%20and%20Best-first%20Search.pdf

Is that what I call the production system for this case?
But, here are my reasons why it should not be called a production system:

There might be other possible states as well.

It is showing only 1 solution.

So, how do I actually learn to create production rules (I know how to make state-space representation better as I have read that J.Nillison's book which was GOLD in this matter)? And, what would be the production rules in this case?

I suggest that you simplify this post as much as possible to contain only the information that is needed to answer your question. Also, please, ask your main question immediately at the beginning, so that we understand which one it really is. Is your question 1. "how to make a production system?", 2. "What is a production system?", or maybe something else? It's not very clear. I think this post can be simplified and clarified.

done. i hope it is clear now.

I am not an expert on the topic, but, if I understand and remember correctly, a production system is composed of the so-called "productions", i.e. "if-then" statements, and a database (or working memory) with facts. Now, in your case, you know that the wolf and goat cannot be together, without the human, otherwise the wolf may eat the goat. So, I think that, in your case, you need to create these if-then statements (or productions) and the facts in the database. Section 2.1 of this paper https://www.doc.ic.ac.uk/~rak/papers/LPS.pdf explains what a production system is, so it may help you.

See also http://refpersys.org/ and the blog of the late Jacques Pitrat on http://bootstrappingartificialintelligence.fr/WordPress3/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Use of virtual worlds (e.g. Second Life) for training Artificial General Intelligence agents?
There is emerging effort for Third Wave Artificial Intelligence (Artificial General Intelligence) (http://hlc.doc.ic.ac.uk/3AI_HLC_2019.html and https://www.darpa.mil/work-with-us/ai-next-campaign) and it covers the open-ended life-long machine learning as well. Currently machine learning agents are being run on quite immutable and simple games like Atari and Go. But what about the efforts to build and run machine learning adaptable agents (or even teams of them) in the virtual worlds (like Second Life) which are complex, expanding and in which the interaction with human representatives happens? Are there efforts to do that?
I have found some articles from 2005-2009, but Google gives no recent literature on queries like Reinforcement Learning Second Life etc.
So - maybe there are some efforts to do this, but I can not just Google it.
My question is - are there references for machine learning agents for virtual worlds and if not - what are the obstacles for trying to build them? There are little risks or costs for building them for virtual worlds?
https://meta-guide.com/embodiment/secondlife-npc-artificial-intelligence is some bibliography and it is lacking recent research, for example.

To clarify, you're asking about agents that can "live" inside a virtual world and not necessarily about any research on artificial general intelligence, right?

I have spent several days looking exactly at the same question. First, let me say that I am very far from this field. It seems that now embodied AI research shifted from game-like virtual worlds to small indoor simulated environments. "A survey of embodied AI: from simulators to research tasks" is an example with plenty of references. So it appears that now this kind of research has more applied nature, eyeing deployment to household assistant robots.

The only example of the kind of research I was looking for (and apparently @TomR too) and was able to find was Microsoft Project Malmo, but it does not seem very active.

Similarly to @TomR, I am kind of puzzled by the lack of this type of research. For me as a non-expert, it seems that such embodied AI research using open-ended virtual worlds, populated with human players, could be one of the few currently realistic ways to get actually intelligent AI in near future. I would appreciate response from somebody more knowledgeable than me.

Yes, there are actually quite a few examples of this:
https://www.deepmind.com/publications/open-ended-learning-leads-to-generally-capable-agents
https://ieeexplore.ieee.org/abstract/document/9089532/
One alternative use for these kinds of methodologies is when real-world experience comes at a premium(ex. robotic system).
In these cases it is very useful to bootstrap your way towards a effective agent by creating a simulated environment and training until convergence(whatever that means for your situation).


text:
What are the mathematical prerequisites needed to understand research papers on neural networks?
I know we have developed some mathematical tools to understand deep neural networks, gradient descent for optimization, and basic calculus. Recently, I encountered arxiv paper that describes higher mathematics for neural networks, such as functional analysis. For example, I remember universal approximation theorem was proved with the Hann-Banach theorem, but I lost the link of that article, so I need to find similar papers or articles to develop my understanding of neural networks mathematically (like with functional analysis, in short, I need to learn more advanced math for research), can you suggest some books or arxiv papers or articles or any other source that describes mathematics for deep neural networks?

To me, it's not really clear what you're asking. What kind of "mathematical tools" are you really looking for? Are you trying to understand why neural networks can approximate continuous functions? It's not clear what your level of understanding of neural networks is and what you want to understand that you don't understand.

@nbro that is a good point, for example, I want to learn mathematics on the research level. I see many AI labs output brilliant papers and I think they must be using some advanced mathematics, it won't be just random calculation, the should think ahead, so I want to learn mathematics on the research level. is it clear, sorry if It is not clear now, please say and i will go more in details.

Can you please provide an example of paper that you want to understand? There are so many mathematical topics that it's difficult to know them all and so it's difficult to answer your question. However, if you share with us the specific paper or papers that you want to understand, maybe we can provide more details about the type of mathematics that you need to understand those papers. Moreover, I suggest that you edit your post to include these details and also to include a description of your current knowledge of NNs. What do you know? What's your level?

@nbro that is exactly my problem, I don't have a mentor to help, so I want to get into the research, but before I get into it, I want to learn enough mathematics for that, so I was wondering what advanced mathematical topics I should know to really be able to make a research by myself. i know mathematical analysis(for multiple and single variables) and I was wondering what should I have to learn next ( I also know stats and linear algebra) i think researchers should know a lot of math, but I don't know what math, because I am confused.

We already have a similar question then: https://ai.stackexchange.com/q/7352/2444, although it is not specific to neural networks. I don't know the quality of the answers, but I think you can trust this answer, given that it was written by someone with a PhD in AI. Anyway, if you think that this question is a duplicate of the just mentioned one, let me know and I will mark your question as a duplicate.

I've changed the title of your post. Let me know if the question in the title is your actual question or not.

@nbro when I saw steward calculus I immediately shut down the page, it will never be enough for neural networks, guys do functional analysis and even that is not enough

Please, provide an example of a paper that you want to understand, otherwise, it's really difficult to answer your question, because there are research papers that involve neural networks but that use only them in a superficial way (in the sense that they are used to solve a task and there is no theory e.g. about their approximation capabilities involved). I think you're looking for the mathematical prerequisites that help you understand the universal approximation theorems, given that you mention that, but it's not clear.

@nbro I don't have a specific paper, if I had I will google what math is involved in that, but I don't want to learn math fragment by fragment. suppose the question is: I want to be in top 10 researchers in the AI field, which mathematical fields I have to master(or at leas know)?

That's difficult to answer because you don't necessarily need big math skills to be an important researcher. It depends on what you research. Also, there are many things in AI, not just neural networks. The short answer to your question is that: you don't need to know all the math to be a good researcher. For example, if you do research on variational inference, you definitely need to know a good dose of calculus, statistics, and probability theory. So, it's difficult to say exactly what you need to learn because there are so many topics and you will almost surely only focus on 1-2 of them.

So, please, ask a more specific question, and you will get more useful answers. We cannot answer your question if you don't tell us what you are interested in. If you say you're interested in becoming an AI researcher, that doesn't mean much, because one AI researcher could be doing research on reinforcement learning, another in expert systems, and another in variational inference. Nobody does research on "AI". Typically, people do research on specific topics or subfields of AI.

Knowing you want to focus on the theory, I think that a good choice is Deep Learning Book from Ian Goodfellow et al., which is publicly available. It has three main parts. On the first one the author presents the math/ statistic tools that will be needed to understand the following parts. On the second part, the author explains the current state of the art in Deep Learning and on the last part more advanced topics are introduced. The author also uses references so as to facilitate extra resources to dive more into the theory.
On the other hand I strongly recommend you Google Scholar, there you have plenty of information given by articles/ papers of the current state of the art techniques in the field. There you can also find papers related on what that you mentioned about the Hahn-Banach Theorem.


text:
What are some programming-oriented resources for reinforcement learning?
I have been reading: Reinforcement Learning: An Introduction by Sutton and Barto. I admit it's a good read for learning RL whereas it's more theoretical with detailed algorithms.
Now, I want something more programming oriented resource(s) maybe a course, book, etc. I have been exploring Kaggle, Open-source RL projects.
I need this to learn and grasp a deeper understanding of RL from the perspective of a developer i.e optimized way of writing code, explanation about using the latest RL libraries, cloud services, etc.

Please, have a look at this, this and this question. If any of those answers answer your question, let me know.

Thank you @nbro, I got what I needed. I see you are leading the AI SE community impressively. Keep up the good work.

If you found what you were looking for and it's in the answers below, please, click on the "check" mark to accept the answer. If it's somewhere else (e.g. in the answer to another question), please, let us know.

Arthur Juliani has some interesting Medium articles on reinforcement learning with TensorFlow backed up with code on GitHub.

Part 0 — Q-Learning Agents
Part 1 — Two-Armed Bandit
Part 1.5 — Contextual Bandits
Part 2 — Policy-Based Agents
Part 3 — Model-Based RL
Part 4 — Deep Q-Networks and Beyond
Part 5 — Visualizing an Agent’s Thoughts and Actions
Part 6 — Partial Observability and Deep Recurrent Q-Networks
Part 7 — Action-Selection Strategies for Exploration
Part 8 — Asynchronous Actor-Critic Agents (A3C)

As nbro pointed out Denny Britz has a good repository: https://github.com/dennybritz/reinforcement-learning
As you have seen with Sutton & Barto's book the code is mostly in Lisp.  Shangtong Zhang has replicated the code in Python: https://github.com/ShangtongZhang/reinforcement-learning-an-introduction
Sudharsan and Ravichandiran wrote a book "Hands-On Reinforcement Leraning with Python" which uses OpenAI Gym and TensorFlow.  You can find more information on the book along with their code repository at Hands-On Reinforcement Learning with Python

You may also want to include this repo https://github.com/dennybritz/reinforcement-learning in your answer. It contains notebooks with programming exercises where you need to implement RL algorithms.

I suggest you to have a look at this repo. It contains state-of-the art algorithms, papers, frameworks, courses and some implementations. You can also check "Deep Reinforcement Learning Hands On" book examples written by Max Lapan here. This repo contains many programming and reinforcement learning examples with PyTorch framework.

Thanks for addressing my concern, but how does this directly answers the question? The OP was looking for resources with programming exercises, if I understood correctly. Maybe you can provide a more specific answer.

Double edited :)


text:
Why do we need to go back to policy evaluation after policy improvement if the policy is not stable?

Above is the algorithm for Policy Iteration from Sutton's RL book. So, step 2 actually looks like value iteration, and then, at step 3 (policy improvement), if the policy isn't stable it goes back to step 2.
I don't really understand this: it seems like, if you do step 2 to within a small $\Delta$, then your estimate of the value function should be pretty close to optimal for each state.
So, why would you need to visit it again after policy improvement?
It seems like policy improvement only improves the policy function, but that doesn't affect the value function, so I'm not sure why you'd need to go back to step 2 if the policy isn't stable.

There is a difference between accurate value function estimates, and optimal value functions. An optimal value function is more specifically the value function of an optimal policy.
Value functions are always specific to some policy, which is why you will often see the subscript $\pi$ in e.g. $v_{\pi}(s)$ when there is a defined policy.
The policy evaluation step (step 2) in policy iteration converges to an accurate value function estimate for whatever the current policy is. In general this will not be an optimal value function, except on the last time that step 2 is used, and there is no change to the policy in the next stage policy improvement (step 3).
The policy improvement stage (step 3) can only usefully be run once for any value function estimate. The policy is updated to be greedy with respect to the value function from step 2 - that will always give the same results from the same value function estimate. If the value function is accurate then this new policy is guaranteed to be as good or better than the previous policy. Once step 3 is done, further improvements are only possible if the new policy is accurately evaluated.
Comparison with value iteration
The difference with value iteration is that it never accurately evaluates any interim policies. In value iteration, the implied policy changes every time the maximising action changes due to new value estimates. In the later stages, when the optimal policy has been found and is stable, then the value function will converge to the optimal value function. In value iteration, most of the interim value functions are not accurate, but when it becomes accurate it will also be the optimal value function.


text:
What is the efficiency of trained neural networks?
Training neural networks takes a while. My question is, how efficient is a neural network that is completely trained (assuming it's not a model that is constantly learning)?
I understand that this is a vague and simply difficult question to answer, so let me be more specific: Imagine we have a trained Deep Neural Net, and even to be more specific it's a GPT-3 model.
Now, we put the whole thing on a Raspberry Pi. No internet access. The whole process takes place locally.

Will it run at all? Will it have enough RAM?

Now let's say we give it some text to analyze. Then we ask it a question. Will it take milliseconds to answer? Or is it going to be in the seconds? Minutes?

What I'm trying to understand, once a model is trained is it fairly performant because it's just essentially a bunch of very simple function calls on top of each other, or is it very heavy to execute? (perhaps due to the sheer number of these simple function calls)
Please correct any misunderstanding about how the whole process works if you spot any. Thank you.

There's this article detailing why some models aren't suitable for practical applications: https://towardsdatascience.com/too-big-to-deploy-how-gpt-2-is-breaking-production-63ab29f0897c

It seems to me that you're asking how efficient using a trained model is. Performance refers to how good the answers are in terms of correctness. So, I suggest that you edit your post to clarify this.

The GPT-3 model specifically is a monster, it requires powerful machine with over 350GB of RAM just to run inferences (and probably mulit-core processors too). There are plenty of others which are smaller, and likely there will be some that could run on a Rapberry Pi. So to prevent the answer being "it depends", could you clarify your use case? Sadly, there is no general answer here, and also if you want to run GPT-3 on any consumer hardware you are out of luck for a few years yet

@NeilSlater Your comment answers my question. We wrote a custom albeit simple NLP processor and were wondering if we made the wrong decision instead of using GPT-3. Your answer clarifies that we didn't, because we would not be able to run such a model with our resources.

@Anton: No problem. There are smaller versions of GPT-3 mind . . . I quoted stats for the largest one, which also has the best accuracy. I looked it up after the comment and saw there were options to use smaller variants that use the same approach but are less accurate. I don't know if the smallest one would fit on a Raspberry Pi, or would be useful for your project.

authors: علي مصطفى برشة


text:
Can we identify only the objects in specific parts of an image with computer vision?
I am studying computer vision for the past 3 months. I have come across the object identification problem, where given an image, CV would identify various parts in the image.
If I give an image, and a rectangle coordinates, can CV identify the parts' names within that rectangle? For example, can I train a model to identify the parts in the below image (mountain, river, in this case)? The model should not identify other parts like flowers, sky, etc., as they come outside the rectangle).
I tried searching but could not find similar problems. Can anyone give me a direction to solve this problem?

Relevant keywords: semantic segmentation, instance segmentation

Giving this small rectangle to a model will probably not provide him enough information to identify what is inside this rectangle. An approach that could work would be to do image segmentation, having the identity of everything in the image, and then extract the label of the pixels in the rectangle.

Yeah. Or crop the region and compute only for the region of interest. This can speed up.


text:
WGAN-GP Loss formalization
I have to write the formalization of the loss function of my network, built following the WGAN-GP model. The discriminator takes 3 consecutive images as input (such as 3 consecutive frames of a video) and must evaluate if the intermediate image is a possible image between the first and the third.

I thought something like this, but is it correct to identify x1, x2 and x3 coming from Pr even if they are 3 consecutive images? Only the first is chosen randomly, the others are simply the next two.
EDIT:

EDIT 2:
I replaced Pr with p_r(x1, x3) and p_r(x1, x2, x3) to reinforce the fact that x2 and x3 are taken after x1, so they depend on the choice of x1. Is it more correct this way?

Can you confirm that the generator for $\tilde{x}$ is being conditioned on $x_1, x_3$? Also that $\hat{x}$ is a random mix of real $x_2$ and a generated $\tilde{x}$?

@NeilSlater yes, I confirm. I added a change to the original post under EDIT regarding the second thing

Please, put your main specific question in the title.


text:
What is the impact of scaling the KL divergence and reconstruction loss in the VAE objective function?
Variational autoencoders have two components in their loss function. The first component is the reconstruction loss, which for image data, is the pixel-wise difference between the input image and output image. The second component is the Kullback–Leibler divergence which is introduced in order to make image encodings in the latent space more 'smooth'. Here is the loss function:
\begin{align}
\text { loss }
&=
\|x-\hat{x}\|^{2}+\operatorname{KL}\left[N\left(\mu_{x}, \sigma_{x}\right), \mathrm{N}(0,1)\right] \\
&=
\|x-\mathrm{d}(z)\|^{2}+\operatorname{KL}\left[N\left(\mu_{x^{\prime}} \sigma_{x}\right), \mathrm{N}(0,1)\right]
\end{align}
I am running some experiments on a dataset of famous artworks using Variational Autoencoders. My question concerns scaling the two components of the loss function in order to manipulate the training procedure to achieve better results.
I present two scenarios. The first scenario does not scale the loss components.

Here you can see the two components of the loss function. Observe that the order of magnitude of the Kullback–Leibler divergence is significantly smaller than that of the reconstruction loss. Also observe that 'my famous' paintings have become unrecognisable. The image shows the reconstructions of the input data.

In the second scenario I have scaled the KL term with 0.1. Now we can see that the reconstructions are looking much better.

Question

Is it mathematically sound to train the network by scaling the components of the loss function? Or am I effectively excluding the KL term in the optimisation?

How to understand this in terms of gradient descent?

Is it fair to say that we are telling the model "we care more about the image reconstructions than 'smoothing' the latent space"?

I am confident that my network design (convolutional layers, latent vector size) have the capacity to learn parameters to create proper reconstructions as a Convolutional Autoencoder with the same parameters is able to reconstruct perfectly.
Here is a similar question.
Image Reference:
https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73

Ans 1.
The motive of Variational Inference(on which VAE is based), is to decrease $KL(q(z|x)||p(z))$, where p(z) is our chosen distribution of the hidden variable z. After doing some math, we can write this expression as-
$
KL(q||x) = log(p(x)) - \Sigma_z q(z)log(\frac{p(x,z)}{q(z)})
$
For a given x, the first term of RHS is constant. So we maximise the second term so that the KL divergence goes to zero.
We can write the second term as
$E_{q(z)}[log(p(x|z)] - KL(q(z|x)||p(z))$
(try writing p(x,z) as $\frac{p(x|z)}{p(z)}$ and then expand. Here, p(z) is the distribution of our choice, i.e. a Gaussian distribution). We argue that the process from z to x is deterministic and z is gaussian. So, the first term could be replaced by $exp(log(-||x-\hat{x}||^2))$(we replaced z by $\hat{x}$ because it's deterministic- this is now the exact proof). So we have-
$
maximize(-||x-\hat(x)||-KL(q(z|x)||p(z)))
$
and we get our loss function.
We also know that variational autoencoders almost never find the optimal solution, so I am not sure how would playing around with the weights affect it(Nor do I know if it makes sense mathematically).
Ans 2.
We can say that the KL divergence has a regularising effect.
This page has some nice experiments which will help you understand what happens to the latent space when you decrease the KL divergence part.
Ans 3.
Yes, you can say that. You are fixing the dimensions, but are lenient on the distribution. In fact, you are approaching autoencoders by doing this.
Separate-
I want to point you towards this article. It explains why we choose to minimise $KL(q(z|x)||p(z))$ instead of $KL(p(z)||q(z|x))$ (the latter is intractable) and what would happen if we choose less independent variables for our estimator q(z).
Also, have you tried increasing the dimensions of the latent space? It can also have a 'de-regularizing' effect. It seems that the model is underfitting the data- the reconstruction loss is high with the normal loss, compared to when you decrease the regularizing term.
Hope it helps.

I have to read this answer carefully to make sure that all info is correct. However, I would like to point out that there's some discussion (in the literature, so you can find some papers that talk about it) on how to scale the KL divergence term in the loss functions of Bayesian neural networks (based on variational inference, i.e. mean-field variational Bayesian neural networks), which have a loss function similar to the VAE, i.e. they also have the KL divergence term. So, you may want to have a look at those papers and maybe edit your answer to include some info that you find there.


text:
In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?
While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6)
$$
J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\psi}\left(\mathbf{s}_{t}\right)-\mathbb{E}_{\mathbf{a}_{t} \sim \pi_{\phi}}\left[Q_{\theta}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)-\log \pi_{\phi}\left(\mathbf{a}_{t} \mid \mathbf{s}_{t}\right)\right]\right)^{2}\right]
\tag{5}\label{5}
$$
$$
\hat{\nabla}_{\psi} J_{V}(\psi)=\nabla_{\psi} V_{\psi}\left(\mathbf{s}_{t}\right)\left(V_{\psi}\left(\mathbf{s}_{t}\right)-Q_{\theta}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\log \pi_{\phi}\left(\mathbf{a}_{t} \mid \mathbf{s}_{t}\right)\right)
\tag{6}\label{6}
$$
The following quote:

where the actions are sampled according to the current policy, instead of the replay buffer

In the context of deriving the formulation of the (estimated) gradient for the value function square residual error (Equation 5 in the paper)
I'm having a hard time understanding why they use the action sampled from the current policy instead of the replay buffer. My intuition tells me that this is because SAC is an off policy Reinforcement Learning algorithm, and Q-learning uses $\max Q$ in one-step Q-value function update (to keep it off-policy), but why would sampling one action from the current policy still make it off-policy?
I first asked a friend of mine (researcher in RL) and the answer I got was

"If the action is sampled with the current policy given any state the update is on-policy."

I've checked SpinningUpRL by OpenAI's explanation of SAC but they only make it more clear which action is sampled from the current policy, and which one is from the replay buffer, but does not specify why.
Does this have anything to do with the stochastic policy? Or the entropy term in the update equation?
So I'm still quite confused. Link/references to explanation are also appreciated!

I was also not able to find an answer on the internet. I think the reason for using the current policy starts with the entropy term in the Q and policy updates. Reading very closely through this section of the Spinning Up explanation helped me. Maybe the reason is that the entropy is supposed to be minimized and it would not work to use entropy from the current policy but actions from a different policy. And it does not seem like using entropy from a past policy would make sense.

If I get the time, I might try to see if SAC can still learn with using a past policy for the entropy, next actions, or both.

Interestingly, SAC learned the Pendulum environment the same with and without a target actor giving the entropy term and the next actions, though it might have learned faster without the target actor. It even learned with the target actor not updating at all because I forgot to add the code in. Maybe there is no target actor just because it isn’t needed.

Hello, thank you for the comment (and also even trying it! :D ) S2673! Just to be sure we are on the same page, by "target actor" you mean the stored action in the replay buffer (from another policy, most often previous one), the a in $(s_t, a_t, r_t, s_{t+1}) \in D$ right?

$a_t$ was from whatever policy was operating when the experience batch was collected. I used the target actor to get the next-state actions($a_{t+1}$) because that is how DDPG and TD3 work. SAC usually gets $a_{t+1}$ from the current policy.

Just like in TD3 and DDPG, the actor’s parameters were copied to the target actor at the beginning of training and every time the actor updates, the target actor was pushed slightly in the same direction with polyak averaging.  The difference between what I did and normal SAC, is the existence of a target actor and using it to get $a_{t+1}$ and the the log probabilities of the next actions (the log probabilities affect the entropy part of the update equation) during the updates of the Q networks.

Ah ok, thank you for the clarification! Hm still have slight hard time wrapping my head around it but I'll keep looking too.

It is a really good question and I hope you get an answer.

I added a bounty to see if someone will answer.

authors: Владислав Арсентьев


text:
How is Google Translate able to convert texts of different lengths?
According to my experience with Tensorflow and many other frameworks, neural networks have to have a fixed shape for any output, but how does Google translate convert texts of different lengths?

Usually, in natural language processing (NLP), they are using Sequence to Sequence Learning (Seq2Seq) with Neural Networks, such as Recurrent Neural Networks or more recently the Transformer (you can find two very good papers here, and here).
During training, to ensure the same size of the input and output they can just search for the longest sentence they have in the dataset or select a number that is high enough and pad all the other sentences with 0. In addition, they add a stop token where the sentence ends, so that the model is aware of this. When decoding (inference), the decoder is going to predict one word at a time until it predicts the stop token, which signals that the translation is done.
If you're interested to see an actual implementation, I would recommend looking at this tutorial which does a good job at explaining the code and how it works.

Transformers have surpassed RNNs (even LSTMs) in some NLP tasks (including translation), and they are now used by Google Translate (check this), so you probably should mention them in this answer.

authors: Максим Василенко


text:
Why should the weight updates be proportional to input?
I'm reading the book Grokking Deep Learning. Regarding weight updates during training, it has the following code and explanation:
direction_and_amount = (pred - goal_pred) * input
weight = weight - direction_and_amount

It explains the motivation behind multiplying the prediction difference with input using three cases: scaling, negative reversal and stopping.

What are scaling, negative reversal, and stopping? These three attributes have the combined effect of translating the pure error into the absolute amount you want to change weight.  They do so by addressing three major edge cases where the pure error isn’t sufficient to make a good modification to weight.

These three cases are:

Negative input,
zero input and
the value of input (scaling).

Negative and zero cases are very obvious. However, I didn't understand scaling. Regarding scaling, there's the following explanation:

Scaling is the third effect on the pure error caused by multiplying it by input. Logically, if the input is big, your weight update should also be big. This is more of a side effect, because it often goes out of control. Later, you’ll use alpha to address when that happens.

But I didn't understand it. Considering the linear regression problem, why weight update should be big if the input is big?

Just to be clear, what is the loss function the book is using in that section? What neural network is the book trying to describe the training of? This seems to be just an interpretation of the derivative of the loss function (i.e. back-propagation, aka the application of the chain rule to compute derivatives of compositive functions).

Thanks for the response. It's MSE loss. Well, it's just an introduction in the book, single input single output linear function. I also read many loss and transfer/activation functions. In every combination, somehow error is multiplied by the input at the end. I know this multiplication comes from chain rule, but I want to know if there can be an intuition behind it other than calculus definition.


text:
Why do we need to provide false labels to the discriminator on purpose to train GANs?
This is the tutorial that I used to learn about GANs. In this tutorial, it taught us to intentionally provide false labels to "fool" the discriminator, but does it make the discriminator actually inaccurate? I don't quite understand his explanation, can anyone help me?

We want to improve the G so we feed into D fake images and tell they are true in order to calculate a gradient towards real images. Take a look at the second part of this answer. I created an illustration there.

in this tutorial, it taught us to intentionally provide false labels to "fool" the discriminator, does it make discriminator actually inaccurate?

When training GANs, the training steps for the generator and discriminator are separate:

There is a training stage for the discriminator, where it is presented with a mix of generated and real data, all correctly labelled. It is important at this stage to not update the generator (otherwise it will get worse by helping to make the fake data look more fake).

There is a training stage for the generator, where the discriminator is presented with only generated data, all labelled incorrectly as if it were real. It is important at this stage to not update the discriminator.

Typically you will alternate between the two stages frequently, making small updates to discriminator and generator separately. Some GANs use metrics to decide how much of each to do, because you don't want either the generator or discriminator to win outright and stop progress - at least at the start.
So yes there is a stage where you deliberately "fool" the discriminator, because being able to do so is the goal of the generator. However, one key detail of this stage is that the discriminator weights are not updated from that faked data. Instead the gradients from that stage are used only to update and improve the generator.
It may help if you don't think of the false labels as being "fool the discriminator", but instead they are "measure how well the generator is fooling the discriminator".


text:
Why do we minimise the loss between the target Q values and 'local' Q values?
I have a question regarding the loss function of target networks and current (online) networks. I understand the action value function. What I am unsure about is why we seek to minimise the loss between the qVal for the next state in our target network and the current state in the local network. The Nature paper by Mnih et al. is well explained, however, I am not getting from it the purpose of the above. Here is my training portion from a script I am running:
for state, action, reward, next_state, done in minibatch:
    target_qVal = self.model.predict(state)

    # print(target_qVal)

    if done:
        target_qVal[0][action] = reward #done
    else:
        # predicted q value for next state from target model
        pred = self.target_model.predict(next_state)[0]
        target_qVal[0][action] = reward + self.gamma * np.amax(pred)

    # indentation position?
    self.model.fit(np.array(state),
                   np.array(target_qVal),
                   batch_size=batch_size,
                   verbose=0,
                   shuffle=False,
                   epochs=1)

I understand that the expected return is the immediate reward plus the cumulative sum of discounted rewards looking into the future $s'$ (correct me if I'm wrong in my understanding) when following a given policy.
My fundamental misunderstanding is the loss equation:
$$L = [r + \gamma \max Q(s',a'; \theta') - Q(s,a; \theta)],$$
where $\theta'$ and $\theta$ are the weights of the target and online neural networks, respectively.
Why do we aim to minimize the Q value of the next state in the target model and the Q value of the current state in the online model?
A bonus question would be, in order to collect $Q(s,a)$ values for dimensionality reduction (as in Mnih et al t-sne plot), would I simply collect the target_qVal[0] values during training and feed them into a list after each step to accumulate the Q values over time?

could you please edit your question so the loss function is written using LaTeX as at the moment it is unreadable.

I have edited your post to fix some formatting issues, etc. Please, make sure that it is still consistent with your original post.

thank you! apologies for not correctly entewring the equation initially. this is correct and does convey my question around why it is the Q of s' and a' minus q val of current s and a.

The loss function is designed in a way to approximate the bellman optimality for $Q^*(s,a)$. Given an optimal policy $\pi^*$, $Q^*(s,a)$ satisfies the equation $$Q^*(s,a) = r(s) + \gamma max_{a'}\sum_{s'}P(s'|s,a)Q^*(s',a')$$
At convergence, the highest $Q$ value that I can get taking action $a$ in state $s$ is equal to the reward I get for taking action $a$ and the max $Q$ values at the next state.
You can see in the loss function that the DQN tries to attain a $Q^*(s,a)$ value that closely approximates the equation above.
On a side note, in model based RL setting, the bellman optimality for $Q^*$ is expressed as $$Q^*(s,a) = r(s) + \gamma \sum_{s'}P(s'|s,a)V^*(s)$$ $V^*(s')$ is used in the model based case because $V^*(s)$ by definition represents the highest possible value attainable at state $s'$ following $\pi^*$. In model free setting, $V^*$ is replaced by $Q^*$ because computing $V^*$ is not useful in achieving model free control without a transition model $P(s'|s,a)$

OP is asking about use of target network for setting TD target (as opposed to the "current" Q function approximation)


text:
Given a dataset and a neural network, is there some heuristic or theorem to determine whether this neural network has enough capacity?
What is the consensus regarding NN "capacity" or expressive power? I remember reading somewhere that expressive power grows exponentially with depth, but I cannot seem to find that exact paper.
If I have some dataset and some neural network, is there some heuristic, theorem, or result that may help me to decide whether this particular neural network I've chosen has enough expressive power (or capacity) to learn that data?
Something similar to VC-dimension and VC-inequality, but regarding more complex models such as NNs.
I suspect that there is no simple answer, but, generally, what would be the answer to this question?
Overfitting on some subset might be a start, but that doesn't really tell me how the model behaves when there's more data, it only tells me that it's not fundamentally broken and can learn something.
I know it's a complex matter, but I'll be grateful for any help, be it some practical stuff, as well as some references, papers, etc. Of course, I googled some papers, but if you have something particularly interesting, please share.

What is your main specific question? Is it 1. What is the consensus regarding NN "capacity" or expressive power? (consensus?) or 2. what if I have some dataset and some model. Is there some heuristic, theorem or result that may help me to decide whether this particualr model I've chosen has enough expressive power to learn that data? Moreover, note that the VC dimension has also be calculated for neural networks. Moreover, a very similar question has already been asked in the past: https://ai.stackexchange.com/q/18220/2444. Please, tell us how your question is different.


text:
Which ML approach could determine that a number greater than 5 is not prime, knowing that a number is not prime if it ends with an even digit or 5?
I have started studying ML just a short while ago, so that my questions will be very elementary. That being so, if they are not welcome, just tell me and I'll stop asking them.
I gave myself a homework project which is to make an ML algorithm be able to learn that, if the last digit of a number $n$ is $0, 2, 4, 5, 6$ or $8$, then it cannot be a prime, provided $n > 5$. Note that, if a number $n$ ends with $0, 2, 4, 6, 8$, then it is even, so it is divisible by $2$, hence not prime. Similarly, numbers ending in $5$ are divisible by $5$, so they cannot be prime.
Which ML approach should I choose to solve this problem? I know that I don't need ML to solve this problem, but I am just trying to understand which ML approach I could use to solve this problem.
So far, I have only learned about two ML approaches, namely linear regression (LR) and $k$-nearest neighborhoods, but they both seem inappropriate in this case since LR seems to be a good choice in finding numerical relations between input and output data and KNN seems to be good at finding clusters, and "primality" has neither of these characteristics.

Your homework project is more suited to use neural networks. You might find it easier to approach that problem when you have some knowledge about them. But even then, for this particular task, you might not get the results you currently have in mind. See this related question: https://ai.stackexchange.com/questions/3389/could-a-neural-network-detect-primes

Ok, have you already tried to build the labeled dataset? Of course, you can build this easily. Now, the first problem is deciding whether this should be a classification or regression problem. Given that the output of the ML model should either be "prime" or "not prime", then this is a categorical/nominal output/dependent variable, hence this is a classification problem, specifically, a binary classification problem. You can try to use any ML method that is able to perform classification. Another problem is how to represent the inputs.

@nbro I have made a labeled dataset with the numbers 1-10000. Thanks for the answer. I'll look for classification algorithms tomorrow (day off today). If you know a good source of interesting ML exercises to practice and learn, i'd appreciate them

Anyway, in principle, you can just pass the last digit of the input numbers to the neural network (of course, this assumes that you exclude all numbers between $0$ and $5$ when passing them to the network). The neural network will just need to determine whether the numbers are even or $5$, so essentially we have $10$ digits, and $6$ of them will need to be memorised and mapped to one class, and the other $4$ will need to be mapped to another class.

@nbro, yes, it does seem like NN are the way to go, as also mugoh pointed out.

@VniVersvs Oh, not necessarily. I just said "network" but I actually meant any ML model that performs classification and accepts different types of inputs.


text:
How is MuZero's second binary plane for chess defined?
From the MuZero paper (Appendix E, page 13):

In chess, 8 planes are used to encode the action. The first one-hot plane encodes which position the piece was moved from. The next two planes encode which position the piece was moved to: a one-hot plane to encode the target position, if on the board, and a second binary plane to indicate whether the target was valid (on the board) or not. This is necessary because for simplicity our policy action space enumerates a superset of all possible actions, not all of which are legal, and we use the same action space for policy prediction and to encode the dynamics function input. The remaining five binary planes are used to indicate the type of promotion, if any (queen, knight, bishop, rook, none).

Is the second binary plane all zeros or all ones?  Or, something else?
How is it known if the move is off the board?  For my game, I know if it is a legal move on the board, but do not know if the move is off the board.

Is the second binary plane all zeros or all ones? Or, something else? How is it known if the move is off the board? For my game, I know if it is a legal move on the board, but do not know if the move is off the board.

The second binary plane is one-hot by definition, there is a single one and everything else is zero. If this definition is not met, it's no longer "one-hot".
The paper doesn't state how exactly to implement "off the board" implementation. Research paper wouldn't go into coding level. However, detecting "off the board" is not a challenging task.

https://webcache.googleusercontent.com/search?q=cache:djj-G4T_PwgJ:https://craftychess.com/hyatt/boardrep.html+&cd=2&hl=en&ct=clnk&gl=au

The next step in board representation evolution is to enclose the board inside a larger array, so that illegal squares are "off" the edge and are easily detectable.

A possibility is add borders to your board. Crafty did that. Extend the board to 10x10. Not 9x9 because you need to deal with knight jumping.
Exactly how you should be doing it is implementation defined. We don't know what Google did because AlphaZero is not open source. I'm here just giving you an example.

Thanks! Greatly appreciate you taking time to answer.


text:
Are there any downsides of using a fixed seed for a neural network's weight initialization?
For example, if we set the random seed to be 0, will we run into any problems? For example, maybe for seed 0, we can only reach a certain training error, but other seeds will converge to a much lower error
I'm specifically concerned about supervised learning on point cloud data, but curious about whether it matters in general whenever you use a neural network.

Do you know what the seed is used for? (I did not downvote, by the way).

When you use a particular seed, it actually ceases to become a random initialization and is instead fixed. I believe the only reason to actually do this would be for reliable reproduction in research and not as a method of training production models.

But why doesn't it work? If the "random" initialization is important, then we would expect different convergence results for different initializations - for supervised learning this is rarely the case.

different random initializations will give different convergence results (slightly different). If you read research papers, they will often cite a $\pm$ after the results which are the different values on different random initializations. They use the seed so that they can reproduce the EXACT same results again. Different random initializations without a seed will give slightly different results. If you use a fixed seed and flip a coin 100 times, you will get the exact same pattern of H and T every time with that seed. While a different random seed would give a random pattern

Why would we ever care about having slightly different convergence results?

You just gave me a good reasoning for why we can set and forget seed=0 in production as well. +- .001% is not going to make a difference (plus, you have to maximize over the seed to get the best result, not just randomly pick one)


text:
What is the difference between sensitivity analysis and parameter tuning?
I tried different values of genetic algorithm operators:

many crossover rates from 20% to 80%
many crossover rates from 1% to 20%
varying the population size

The study of different parameter values is called quantitative parameter tuning or sensitivity analysis. What is the difference between the two terms?

I would generally assume that parameter tuning is the process of finding the combination of hyperparameters (e.g., population size, crossover and mutation operators and rates, etc.) that yield the best performance on your problem. When you're thinking about the way that performance varies with parameter choice, this is the "what". What is the best choice of parameters.
Sensitivity analysis is the "how" or "how much". If I change my crossover rate from 0.9 to 1.0, how significant is the change in performance of my algorithm. Is my performance more or less stable across a wide range of choices (good) or is it highly dependent on finding this one little peak in the parameter space and every other choice is much worse (bad).

so by sensitive analysis we can have a good parameter tuning ?


text:
What is the difference between exploitation and exploration in the context of optimization?
In the paper Moth-flame optimization algorithm: A novel nature-inspired heuristic paradigm (2015, published in Knowledge-Based Systems)

The test functions are divided to three groups: unimodal, multi-modal, and composite. The unimodal functions ($F1 - F7$) are suitable for benchmarking the exploitation of algorithms since they have one global optimum and no local optima. In contrary, multi-modal functions ($F8 - F13$) have a massive number of local optima and are helpful to examine exploration and local optima avoidance of algorithms

I imagine that exploration means it goes searching for something in unknown regions from a starting point. But exploitation would search more around the starting (or current point).
It is more or less that? What else differentiates both concepts?

In the specific context of reinforcement learning (which can also be thought of as a collection of optimization techniques), exploitation means "to choose the best current action", while exploration means "to choose a random action". However, in your context, these terms seem to be used slightly differently. However, to provide a reasonable answer to your question, we should also know what a test function is in this context. Either you explain it, or we need to read the paper.

Test functions are fitness functions, such as the sum of (Xi)², going from 1 to n,. This if f1 for instance. From F1 to F7, all are sum of Xi after doing something else with it. All except F6, that it is the max value of the |xi| from 1 to n.


text:
What adapts an algorithm to continuous or to discrete action spaces?
Some RL algorithms can only be used for environments with continuous action spaces (e.g TD3, SAC), while others only for discrete action spaces (DQN), and some for both
REINFORCE and other policy gradient variants have the choice of using a categorical policy for discrete actions and a Gaussian policy for continuous action spaces, which explains how they can support both. Is that interpretation completely correct?
For algorithms that learn a Q function or a Q function and a policy, what places this restriction for their use on either discrete or continuous action spaces environments?
In the same regard, if an algorithm suited for discrete spaces is to be tuned to handle continuous action spaces, or vice versa. What does such a configuration involve?

So, essentially, you're asking two questions: 1. in general, what makes an RL algorithm applicable to continuous and/or discrete action spaces? 2. how can we adapt an RL algorithm that is applicable to continuous action spaces to discrete ones (and vice-versa)?

Yes @nrbo. Would it be more recommended to ask them separately in this case?

These are really very related questions, so I am not so sure. Maybe, in this case, I would leave it like this, because these 2 questions can be merged into 1 where you're asking which RL algorithms can deal with continuous (or discrete) action spaces, either if they support it out of the box or not.

Sure, they can be merged into one but I also feel asking for which RL algorithms deal with continuous or discrete spaces will leave out the question about what "features" make an algorithm application to a certain action spaces, and how they would be transferrable to an algorithm that's applicable to a different action space.


text:
What are the use-cases of self-replicating neural networks?
I started researching the subject of self-replication in neural networks, and unexpectedly I saw that there is not much research on this subject. I should mention I am new in the field of NNs.
This idea seems to be very appealing, but now I am having problems coming up with an actual use case. The paper Neural Network Quine from 2018 seems to be one of the main ones addressing this topic.
So, what are the use-cases of self-replicating neural networks? Why isn't this subject more thoroughly researched?

You asked a very similar question here. In any case, in the abstract of that paper that you cite, the authors write "We suggest that a self-replication mechanism for artificial intelligence is useful because it introduces the possibility of continual improvement through natural selection.". Moreover, what you are quoting from Reddit, i.e. "it's really just take machine learning model and have it predict its own weights", it's exactly what's written in the abstract of the same paper you're citing. Have you read that paper?

I asked again trying to be more specific, as suggested in the other question. Of course I read it. I just wanted to start a discussion on the subject and hear what other (more qualified) people think about it, but it seems I dont know how to ask a question around here to make people respond

It's also possible that nobody here is familiar with or interested in similar work. That paper actually contains a section "Motivation" where the authors try to motivate their work. I suspect that people still don't see many use cases of these systems, and that's why there may not be much research work (I actually don't know, because I have not researched anything on the topic). It's possible that this type of neural network, which refers to itself, in some way, may turn out to be useful. Later, maybe I will read something about the topic, and maybe I will attempt to provide an answer.

Thanks for the answer! Would be helpful if you find the time

This paper Self-Replication in Neural Networks by Thomas Gabor et al. could also be useful to answer your question (I didn't read it yet).

I have read it. It is about tuning NNs to accept other NNs as inputs with the purpose of having themselves as input and thus self-replication. To make this possibile they use several reductions. It doesnt really say much about the end goal, or purpose. It is more for research purposes

I'm pasting here the link to the Reddit post that you mentioned: https://old.reddit.com/r/MachineLearning/comments/js4bkz/d_why_arent_self_replicating_neural_networks_more/.

@Ezeeroc Have you reached any conclusion which you could share?

@Rexcirus only more abstract ones and nothing very specific


text:
What is the appropriate way of passing a list of integers that represents the environment to a neural network's dense layer?
I'm training an RL agent using the DQN algorithm to do a specific task. The environment is represented by a list of $10$ integer numbers from $0$ to $20$. An example would be $[5, 15, 8, 8, 0, \dots]$.
Is it okay to pass the list as floats to the dense layer, or would that impede the learning process? What is the right way to go about passing integer numbers to the neural network?

It looks like 10 numbers, it's not that much as input for dense layers. I would recommend normalizing them either to float in the range from 0 to 1 or [-1;1] depending on data specifics.

This seems to help my model's performance!


text:
If random rotations are included in the data augmentation process, how are the new bounding boxes calculated?
When studying bounding box-based detectors, it's not clear to me if data augmentation includes adding random rotations.
If random rotations are added, how is the new bounding box calculated?

I don't have much practical experience with them, but, I guess, if you rotate the images, you just need to rotate the boxes (i.e. labels) too. Check this (which may not be a reliable implementation: I don't know!).

The problem is the predicted boxes are not rotated - so would you just create a new box from the rotated images bounding box corners? This works but might make the predicted box a little larger,.


text:
How can I discourage the RL agent from drawing in a zero-sum game?
My agent receives $1, 0, -1$ rewards for winning, drawing, and losing the game, respectively. What would be the consequences of setting reward to $-1$ for draws? Would that encourage the agent to win more or will it have no effect at all? Is it appropriate to do so?

It can depend on many factors, for instance, your environment and data specifics. So, perhaps, there are no generic answers. Only some specifics observed during my experiments - it can slow down learning and create suboptimal agents which tend to make too risky decisions. Why don't you try it with your case and compare with my observations? That would be an answer for your question.

Yes, maybe you should tell us what zero-sum game you are trying to solve with RL, so that people could also try out your setting.

Thanks for answers. This time I'm trying to solve the mancala game for different board sizes. My project is still on early stage (bugs/subtle bugs), so it's hard for me to really isolate the affect in this case. I will share the results if/when I make progress, but I can see now how it makes sense that this would experiment-specific.


text:
What are the benefits of using ELU over other activation functions in CNNs?
I have come up with some examples of CNNs (segmentation CNNs) that use ELU (exponential linear unit) as an activation function.
What are the benefits of this activation function over others, such as RELU or leaky RELU?

Sigmoid returns a number between 0 and 1. If input is large, the gradient of the function give no useful information, since outside the 0-1 range the function is flat. in compare to sigmoid, the function like ELU and ReLU are not flat outside the range. take a look at the form of these functions


text:
How to translate sudoku XV boards in CNF format?
I'm trying to implement the logic for a Sudoku XV puzzle, that it's essentially a standard sudoku with the addition of X and V markers between some pairs of squares. X markers in adjacent pairs requires that the sum of the two values is 10. Similarly, the V marks requires that the sum of the values is equal to 5.
(Assume that $$ S_{xyz} $$ stands for [digit][row][column])
I've written the following CNF formulae that handle the logic of a standard Sudoku puzzle:
There is at least one number in each entry:
$$ \bigwedge_{x=1}9\bigwedge_{y=1}9\bigwedge_{z=1}9S_{xyz} $$
Each number appears at most once in each row:
$$ \bigwedge_{y=1}9\bigwedge_{z=1}9\bigwedge_{x=1}{8\bigwedge_{i=x+1}9}{(\lnot S}_{xyz\ }\vee\lnot S_{iyz\ }) $$
Each number appears at most once in each column:
$$ \bigwedge_{x=1}9\bigwedge_{z=1}9\bigwedge_{y=1}{8\bigwedge_{i=x+1}9}{(\lnot S}_{xyz\ }\vee\lnot S_{xiz\ }) $$
Each number appears at most once in each 3x3 sub-grid:
$$
\bigwedge_{z=1}9\bigwedge_{i=0}2\bigwedge_{j=0}{2\bigwedge_{x=1}2\bigwedge_{y+1}3\bigwedge_{k=x+1}3\bigwedge_{l=1,\ \ y \neq l}3}{(\lnot S}_{(3i+x)(3j+y)z\ }\vee\lnot S_{(3i+k)(3j+l)z\ })
$$
Unfortunately, I'm stuck, and I don't really know how I can express the logic for X and V markers, and most importantly how to invalidate squares that contain neither an X nor a V marker that have digits summing to 5 or 10.

I'm a new contributor so I can only add a comment in an answer post. This is a very general comment. The solving logic of a sudoku puzzle resides least in the visible digits (entries) than in the invisible candidates. The candidates state space can be represented by a 9x9x9x9 boolean matrix, the coordinates of which are (candidate, row, column, block). In that space the X and V logics can easily be expressed.


text:
Has the logistic map ever been used as an activation function?
I find the logistic map absolutely fascinating. Both in itself (because I love fractal) and because it is observed in nature (see: https://www.youtube.com/watch?v=ovJcsL7vyrk).
I'm wondering if anyone tried it as an activation function in some way or another with any kind of success.
I like it because it has some kind of "I'm not sure what to do" above ~3.0 and the less confidence the more chaotic the response is. It gives the possibility to explore some other solution to escape a local optimum (not sure I use this word correctly).  And below 3 it's still a nice and smooth activation function like, eg, a tanh.
Eg : the reward i got isn't the reward i expect, and the higher the difference the more i'll explore other solution. But it's still gradual, from 1 choice, to 2 choice, 4, 8, 16, ... until it become chaotic. (giving the possibility to experiment some pseudo-random choice). And below this threshold it still act as a usable "good old" activation function.
Another good side is that it's gpu-friendly and don't need many iteration for this application since a little bit of uncertainty (even below the threshold) isn't undesirable. see : https://upload.wikimedia.org/wikipedia/commons/6/63/Logistic_Map_Animation.gif
Edit : so, ok, i tested it on my extremely naive racetrack. (feedforward, no feedback, no error, no fitness, only genetic selection for the car that didn't crash). It does work, for sure. I don't see any advantage in practive but with such a naive NN, there isn't much i can tell.
My implementation :
def logi(r):
    x = .6  # the initial population doesn't matter so i took .6
    for _ in range(random.randrange(10,40)):
        x = r * x * (1 - x)
    return x

The activation take 8% of my laptop cpu (while is was invisible on my radar with leaky leru)

Can you explain more in detail how you would use the "logistic map" as an activation function? Maybe it would be a good idea if you briefly describe what the "logistic map" is, how it is defined, and why you think it could be used as an activation function.

i'm not quite sure, i'm still struggling with my code and don't have the leisure to experiment like this, yet. But the general idea is to allow the NN to have some level of chaotic behavior. Eg : the reward i got isn't the reward i expect, and the higher the difference the more i'll explore other solution. But it's still gradual, from 1 choice, to 2 choice, 4, 8, 16, ... until it become chaotic. (giving the possibility to experiment some pseudo-random choice). And below this threshold it still act as a usable "good old" activation function.

The formula is still simple https://ipython-books.github.io/121-plotting-the-bifurcation-diagram-of-a-chaotic-dynamical-system/ but a bit slow since it is an iterated function.

Another good side is that it's gpu-friendly and don't need many iteration for this application since a little bit of uncertainty (even below the threshold) isn't undesirable. see : https://en.wikipedia.org/wiki/Logistic_map#/media/File:Logistic_Map_Animation.gif

I suggest that you edit your post to include more details that explain your idea.

i edited my post. as for what the logistic map is, the veritasium video i linked explain it better than i can. Biological Neuron also have some behavior that look like a logistic map

Wait, it's not clear here what you mean by this "(feedforward, no feedback, no error, no fitness, only genetic selection for the car that didn't crash)". So, you have a MLP with this activation function. What does this MLP get as input and produce as output? What exactly are you changing with "genetic selection"?

I have a selection process (car that didn't crash + multiple mutated clone (child) of the fastest car) and randomly change (multiplied by a decreasing learning rate) the weight and bias of a randomly chosen layer based on some mutation chance. I'll have to change the code to chose a random neuron instead of a random layer.

The input are some kind of lidar (raycasting from -90 to 90°) to detect the racetrack + speed of the car + lateral velocity (+/- G lateral force, for drifting)

this is similar, afaik, to what seems to be implemented in the game "gridworld" https://store.steampowered.com/app/396890/Gridworld/

I suggest that you apply your idea (of using that function as an activation) to a simple neural network in a simple task. Start with a dataset like Iris Flowers Dataset. You're trying to combine multiple things, each of which may not work, consequently, it will be hard to make something work.

or someone with the necessary expertise to evaluate/test/quantify an activation function could do it ?

my main concern is that it may simply amount to having a "noisy input" and nothing else.

So, you said that the input to your neural network is "some kind of lidar (raycasting from -90 to 90°) to detect the racetrack + speed of the car + lateral velocity (+/- G lateral force, for drifting)", but what is the output of this neural network supposed to represent? I still don't understand your setup and what problem you're trying to solve. You should edit your post to start by saying: 1. what problem you're trying to solve, 2. which techniques are you using and how exactly you're using to solve that problem, 3. where this logistic map comes into play exactly in this process.

I know that this is not the main part of the question, but I still don't understand where you're using this logistic map. Your implementation suggests that this logistic map gets $r$ as input, but $r$ is just a hyper-parameter that defines the specific logistic map curve, and not the real input to a neuron of a NN (or is $r$ supposed to be that?). Where exactly are you using this logistic map? Are you using it as an activation function of a neural network or something else? Are all your activation functions in the NN equal to the logistic map? What $r$ value are you using for each of them?

Also, what are these numbers " from 1 choice, to 2 choice, 4, 8, 16,"? Are the numbers you use for $r$ or what? If yes, then I don't understand at all how you're using it as an activation function of the neurons of your neural network (NN).

that's how the logistic map work. After N iteration, whatever the initial condition is, it converge to a single point if r (the input) is <  ~3, then it oscillate between 2 point if ~3 < r < ~3.44. then 4 points if ~3.44 < r < ~3.54, then 8, 16, ... and above ~3.56 it become chaotic; then there are some "island" for r where it oscillate again, then chaotic again, ... It works as an activation function since you can map an input (r) to an output (x). it is, however, extremely sensitive to initial condition above a certain threshold (that's why it's chaotic)

the video i linked in the post explain it all. it's an extremely simple formula.

my own problem/playground to test this does not matter. The point of the post is to know if anyone else used/tried/tested/evaluated the logistic map as a valid activation function (instead of using relu, tanh, etc ...) and what they think of it. (i know it works on my playground, but does it always ? is it better in some case or another ? i'm not good enough to test and tell)


text:
Is it possible to pre-train a CNN in a self-supervised way so that it can later be used to solve an instance segmentation task?
I would like to use self-supervised learning (SSL) to learn features from images (the dataset consists of similar images with small differences), then use the resulting trained model to bootstrap an instance segmentation task.
I am thinking about using Faster R-CNN, Mask R-CNN, or ResNet for the instance segmentation task, which is pre-trained in an SSL way by solving a pretext task, with the aim that this will lead to higher accuracy and also teach the CNNs with fewer examples during the downstream task.
Is it possible to use SSL to pre-train e.g. a faster R-CNN on a pretext task (for example, rotation), then use this pre-trained model for instance segmentation with the aim to get better accuracy?

Is it possible to use SSL to pre-train e.g. a faster R-CNN on a pretext task (for example, rotation), then use this pre-trained model for instance segmentation with the aim to get better accuracy?

Yes, it's possible and this has already been done. I don't know the details (because I have not yet read those papers), but I will provide you with some links to some potentially useful papers (based on their titles and abstracts) and associated code.

Learning Instance Segmentation by Interaction (code, 2018)
Self-supervised Transfer Learning for Instance Segmentation through Physical Interaction (code, 2019)
White Matter Tract Segmentation with Self-supervised Learning (code, 2020)
Self-supervised Depth Estimation to Regularise Semantic Segmentation in Knee Arthroscopy (code, 2020)
Simulation of Brain Resection for Cavity Segmentation Using Self-supervised and Semi-supervised Learning (code, 2020)

You can probably find more relevant papers here, where I also found some of the just cited papers.
The pre-text tasks designed in these papers could be useful in your case, but it may also turn out that you need to develop other pre-text tasks or combine multiple of them.
Maybe you can start from some pre-trained faster R-CNN or some appropriate model for instance segmentation (that you can find on the web, for example, here), which has been pre-trained on some imagery data similar to yours (either with SSL or by other means), then try to fine-tune this model with your labeled dataset for instance segmentation, and see if you get better results than just training a faster R-CNN from scratch. Eventually, if this pre-trained model does not lead to higher performance, you could pre-train it yourself with some SSL technique that you can come up with or one that is described in the literature. Of course, you should probably use a pre-trained model that has been pre-trained with data that is relevant for your downstream task (i.e. the instance segmentation task). You didn't describe the details of your unlabelled and labeled data, so I cannot be more specific (and I wouldn't currently be able to, in any case, because I didn't fully read those papers, and my experience with SSL techniques is mostly theoretical).
For more info about SSL, take a look at this and this answers.


text:
How to use sigmoid as transfer function when input is not (0,1) range in ANN?
I am building my first ANN from scratch. I know that I need a transfer function and I want to use the sigmoid function as my teacher recommended that. That function can be between 0 and 1, but my input values for the network are between -5 and 20. Someone told me that I need to scale the function so that it is in the range of -5 and 20 instead of 0 and 1. Is this true? Why?

I believe it's not true because the scaling is done by the weights, which are learned by the ANN during the optimization. Hence, it's the ANN job, not ours.

The sigmoid function can take any value but it is recommended that inputs are scaled to be lower. The most common sigmoid function can only output values between 0 and 1. If you mean that the values supposed to be outputted by then network are between -5 and 20 then you could add 5 and divide the values by 25 and reverse that after training. Whoever said that the input values needed to be between -5 and 20 was wrong or not understanding your question.

I presume when you say input you may be referring to the target values (the things you are trying to predict). If not, then some parts of your question might not make sense, like your proposal to apply a scaling.
In any case I would consider what the target distribution is before using a sigmoid and applying a scaling. The thing about a sigmoid is that the model has to be very confident in order to predict a value close to the boundaries. So in your case, the model would have to be really confident about something for it to output a -5, and it would have to be really confident of the opposite to output a +20. I would be surprised if this is what you are really looking for in this case.
Let's say your target distribution is uniform. Then your best bet is to use a linear activation, and maybe use max/min to clip it. Then it would behave a lot like a ReLU.
If I haven't quite nailed it with this answer, can you give more info on what you are predicting?

There are several functions that can be denoted as sigmoid functions, such as the logistic function and the hyperbolic tangent, given that they have an $S$-shaped curve. You can find more info about them in the related Wikipedia article.
However, when people use the term sigmoid function, they typically refer to the logistic function, which is a function of the form $$\sigma: \mathbb{R} \rightarrow [0, 1],$$ So, it accepts any real number and produces a number between $0$ and $1$, so you can pass any real number (as input) to the sigmoid.
The standard logistic function $\sigma$ is mathematically defined as follows
$$\sigma(x)=\frac{1}{1+e^{-x}}\in [0, 1]$$
Here is the graph of the logistic function. The $y$-axis contains the output values, i.e. $\sigma(x)$ for any $x \in \mathbb{R}$, while the $x$-axis contains the input values, i.e. $x \in \mathbb{R}$.

That function can be between 0 and 1 but my input values for the network are between -5 and 20. Someone told me that I need to scale the function so that it is in the range of -5 and 20 instead of 0 and 1. Is this true? Why?

No, in principle, you do not have to scale your inputs, for the reasons I have just explained. However, this does not mean that, in practice, it may not be a good idea to standardize or normalize your inputs before feeding them into the neural network. See the vanishing gradient problem (VGP) in this article (by J. Brownlee), or this one (by M. Nielsen), or in this video (by A. Ng). Moreover, you should probably actually use another activation function, such as the ReLU, which does not suffer from the VGP (which, nevertheless, should not occur, if your neural network does not contain more than a couple of hidden layers).


text:
Is non-negative matrix factorization for machine learning obsolete?
I am taking a course about using matrix factorization for machine learning.
The first thing that came into my mind is by using the matrix factorization we are always limited to linear relationships between the data, which is very limiting to predict complex patterns.
In comparison with neural networks, where we can use a non-linear activation function. It seems to me that all the tasks that matrix factorization can achieve will score better using a simple multilayer neural network.
So, can I conclude that NMF and matrix factorization for machine learning, in general, are not that practical, or there are cases where it's better to use NMF?

I know for sure that matrix factorisation is revolutionary in other fields and calculations.
My first intuition is that by using matrix factorization in ML application is similar to finding patterns in the data. And therefore why bother using for such tasks as long as we know that neural networks can perform way better.
Thanks for your comment.

@RamiZK The related Wikipedia page contains a section Applications that provides an answer to your own question. After reading it and consulting the cited resources there, you could try to provide an answer below to your own question (if no one else attempts to do it) ;) By the way, welcome to AI SE! Take at a look at https://ai.stackexchange.com/help/on-topic to know more about our scope, if you have some time.

I will give you a few scenarios where matrix factorisation stills works pretty well.

Topic Modelling : Given a matrix of Document as Rows and Terms/Words as column you can use Non Negative Matrix factorisation to identify Topics. Number of Topics is defined by user or can be treated as hyperparameter.

Image Ref : https://towardsdatascience.com/nmf-a-visual-explainer-and-python-implementation-7ecdd73491f8

Collaborative filtering which is a class of recommendation engines used matrix factorisation extensively

Matrix factorisation can also be used as a feature reduction for a lot of ML algorithm. Infact most commonly used feature reduction techinue PCA is kind of matrix factorisation

Yes, we may have advanced algorithms which may perform better but dont forget.
No Free Lunch Theorem in Machine Learning : No ML algorithm will outperform every other algorithms for every use case
So yes matrix factorisation is still relevant and forms backbone of a lot of advanced ML algotithms


text:
CSP heuristic to simultaneously reduce conflicts and find near optimal assignment
I am trying to design a good heuristic to solve a constraint satisfaction problem (CSP). I think that a possible heuristic to use is
$$h_1(\text{state}) = \text{number of conflicts in state}$$
However, of the possible solutions to the CSP, some have a lower cost (are better). So I can't just use $h_1$ as my heuristic.
Since the state space is pretty huge, I want to use the local search with a heuristic $h$, that guides my variable assignments towards a low-cost solution while reducing the conflicts. The way I am thinking about going about this is: of the variable assignments which do not cause conflicts (are valid), apply $h$ to them, pick the variable assignment which has the lowest/best value $h$ value. So $h$ would not handle conflicts, I would make sure any assignments considered by $h$ are guaranteed to be valid.
Ideally, though, I want $h$ to both drive down the conflicts to 0 and simultaneously guide the assignments to the lowest cost solution. Is this generally possible?


text:
What is the best approach for sentiment analysis when the text is very brief?
I'm working on a project to do sentiment analysis but my data is not long and properly formatted text. It's more likely to be very short sentences, e.g. tweets (in full tweet lingo), quick reviews of maybe 2-5 short sentences, etc.
If my text is of that nature, what approach would you recommend? E.g. CNNs (spaCy has a ready-made text classifier), LSTM (e.g. something like Keras), etc.
What are the pros/cons of your suggested approach (i.e. why is it better suited for classifying short paragraphs/sentences)?
I'm starting out in the area so any links/papers/etc. will be most welcome!


text:
What does Dice Loss should receive in case of binary segmentation
I implemented Dice loss class in pytorch:
import torch
import torch.nn as nn

class DiceLoss(nn.Module):
    def __init__(self, weight=None, size_average=True):
        super(DiceLoss, self).__init__()

    def forward(self, inputs, targets, smooth=1):
        smooth = 1.

        input_flat = inputs.contiguous().view(-1)
        target_flat = targets.contiguous().view(-1)

        intersection = (input_flat * target_flat).sum()
        A_sum = torch.sum(input_flat * input_flat)
        B_sum = torch.sum(target_flat * target_flat)

        dsc = (2. * intersection + smooth) / (A_sum + B_sum + smooth)
        return 1 - dsc

Now I tested it in 2 scenarios:

where inputs is the prediction from the network without applying activation (in my case sigmoid), only convolution with a kernel of size 1.
where inputs are the result of the network including activation of the sigmoid.

Now I get comparable results between the 2 ways, but I was wondering what is the "Right" way out of the 2.


text:
Are there any new weight initialization techniques for DNN published after 2015?
Considering weights initialization in my personal projects, I always used some standard techniques such as:

Glorot (also known as Xavier) initialization (2010).
Mertens initialization (2010).
He initialization (2015).

As it is a very active research field, are there some innovations in recent years that have increased the performance of DNNs?
I am thinking specifically of architectures such as DNNs and CNNs with activation functions, such as ReLU,  ELU, PReLU, Leaky ReLU, SELU, Swish, and Mish.

I found this paper Improving Weight Initialization of ReLU and Output Layers (2019). There's only 1 citation for now (so I don't know how much it's reliable), but, according to their abstract, they obtained good results. You can find the source code here. Maybe you could have a look at it, and later provide an answer to your own question (if nobody does it).


text:
Object detection approaches without anchors and NMS
The Context
From all of the problems I have worked with in computer vision, the most challenging one is the object detection. This is not because the problem itself is complex to understand or bad formulated. But because we need to inject some strong priors about how we understand the world. Those priors are the anchors (which are priors about object shapes, aspect ratio...).
This prior information, although very simple to understand, it is very hard to inject on the training logic. Hence making the computation of the ground truth very messy and prone to errors. It is even  harder when different object detection backbones propose different ground truth computation methods.
The Question
From mid-2019 till now there is a growing trend on research about one-stage object detectors that do not rely on anchors: hence dropping the costly NMS postprocessing and in some cases even the IoU computation. I would like to do a proof of concept with some of them so here is my question:
What are some good object detectors that do not use anchors? Or said in other words, what are the go-to object detectors for this new research trend?

i don't believe avoiding anchor points would be a good approach, as humans always focus on a point before identifying what it is

Another approach is, as you say, identify object as points (see paper with the same name "Objects as Points"), I feel the anchors compute is something the network should learn by itself not y us injecting them

The network you are looking for is DETR by Facebook.


text:
Can I use a CNN for template matching, so that there is robustness, as the background of the target image is not that good?
I have to extract part of a source image, then I have to check if it is similar or almost similar to any of the 10 target images, so that I can do further processing on that one specific target image, which is similar to the source image. It's like template matching, but they have to loop over 10 different images to find whether a matching template is found in any of those images or not.
I wanted to use a CNN-based solution, as a classical distance-based solution is giving poor results.
Can I use a CNN for template matching, so that there is robustness, as the background of the target image is not that good, and it causes a problem? If some resource can be pointed that would be great too.

Do you have enough training instances for each of the 10 (target) classes for which you have the prototypical target images? Because in that case, the task would boil down to a simple classification task (of a class label). Otherwise, for template matching, have you tried using the SIFT transform method, possibly in combination with pre-processing to reduce the noise in images?

Yes I have reasonable value to train right now but problem is that in future classes will be keep on expanding and new classes will be slightly different from existing classes, that's why I was thinking to use template based CNN solution. I have tried SIFT and SURF based algorithm but using them didn't gave better result. I have thought about zero shot learning/incremental learning as for future classes I will have some details available like how new class of image should be. But not sure which approach to take.

A CNN, if you don't use it for classification, only produces a highly-nonlinear encoding of some input image. From there on, you would probably have to resort to other template matching techniques again. Alternatively. have you tried LVQ already?

Thanks for suggestion, will try both CNN and parallelly LVQ to check which one works. But since my old try with classical algorithm failed, or it was very slow to find matching.As destination where the template is to be matched is very noisy and filled with extra unnecessary details which confused classical methods for matching.

You can always try improving robustness of almost any ML method by applying data augmentation, i.e. by adding additional artificial training data to your existing training dataset, where the extra images are copies of existing training images to which noise (maybe even of different magnitude, so to say) has been added. That might help as well.


text:
What is the first short film completely made by AI?
I have created (not me exactly) a short film entirely made by AI. There many short films (like Sunspring) 'written' by AI but were acted out by humans. In my short film, the story is by the AI, the music is by the AI, the title art is by the AI, the visuals and acting is by the AI (yes the AI can act) and the dialogue is by the AI. So, everything is AI. What I wanted to know is if this is the first one like this. I can't seem to find others online.

Hello. I suppose that you don't want to share your film for now. Right?

I didn't know about Sunspring, but I've just watched it, and it makes no sense: the actors and the director made their best to give any meaning to this non-sense. Rather than a science fiction movie, it looks more like either a comedy or a drama in a mental hospital (even a crazy person would say more sensical words and sentences than those generated by the AI) or maybe a dream but without making sense at all (i.e. dreams make more sense than that). Here is the screenplay for those of you that want a big laughter due to non-sense.

@nbro I could share it if i wanted. But I want to know if it's the first of it's kind. It's not for the publicity (there's that too)... but I just wanna know