text:
Can the normalization factor for the belief state update be zero?
In order to update the belief state in a POMDP, the following formula is used:
$$b'(s')=\frac{O(a, s', z) \sum_{s\in S} b(s)T(s, a, s')}{\mathbb{P}(z \mid b, a)}$$
where

$s$ is a specific state in the set of states $S$
$b'(s')$ is the updated belief state of being in the next state $s'$
$T(s, a, s') = \mathbb{P}(s' \mid s, a)$ is the propability (function) of having been in $s$ and ending up in $s'$ by taking action $a$;
$O(a, s', z) = \mathbb{P}(z \mid s', a)$ the probability (function) of observing $z$, performing action $a$ and ending up in $s'$
$\mathbb{P}(z \mid b, a)$ is defined as follows $\sum_{s \in S}b(s)\sum_{s' \in S} T(s, a, s')O(a, s', z)$

Looking at $\mathbb{P}(z \mid b, a)$ it is possible that the result is $0$. This would be the case if the agent is in a state where no further actions are possible. But, in that case, there is a problem with updating $b'(s')$, since this causes a zero division. Is this a common problem and is the only possibility to avoid that a programming solution like an if-statement? Or is $\mathbb{P}(z \mid b, a)$ always non-zero?

I think that the normalisation factor is assumed to be non-zero. So, in practice, I guess, you must eventually check that $P(z \mid b, a)$ is non-zero (even though, I guess, it will likely never be zero because of round-off errors in computers).
The formula to calculate $b'(s')$ comes from its definition, which is based on Bayes' theorem, where the denominator is assumed to be non-zero (in general).
The definition of $b'(s')$ is $P(s' \mid z, a, b)$, that is, the new belief $b'$ of being in state $s'$ is defined as the probability of landing in the next state $s'$, given that we have observed $z$, have taken action $a$ from the previous state $s$ and we had the previous belief $b$. We will expand this definition, but first let us recall a few probability definitions.
Recall that $P(A, B) = P(A \mid B) P(B) = P(B \mid A) P(A)$, where $A$ and $B$ can actually be multiple events (that is, $A$ could actually be the intersection of multiple events). In other words, suppose we want to calculate $P(A, B, C)$, we can actually consider e.g. $B$ and $C$ as one event. Let $(B \cap C) = (A, B) = D$ (note that the notation $(A, B)$ means the "intersection" of events $A$ and $B$, in the case $A$ and $B$ are events). Then
\begin{align}
P(A, B, C)
& = P(A, (B, C)) \\
&= P(A, D) \\
&= P(A|D)P(D) \\
&= P(A|B, C)P(B, C) \\
&= P(A|B, C)P(B|C)P(C) \\
\end{align}
In general, this idea generalises to more variables/events.
Note also that $\frac{P(A, B)}{P(B)} = P(A \mid B)$.
At this point, we are prepared to expand $P(s' \mid z, a, b)$ and understand its expansion.
We can expand $P(s' \mid z, a, b)$ as follows
\begin{align}
P(s' \mid z, a, b)
&= \frac{P(s', z, a, b)}{P(z, a, b)}\\[0.7em]
&= \frac{P(s', z, a, b)}{P(z \mid b, a)P(a|b)P(b)}\\[0.7em]
&= \frac{P(z \mid s', a, b) P(s' \mid a, b) P(a | b) P(b)}{P(z \mid b, a)P(a|b)P(b)} \\[0.7em]
&= \frac{P(z \mid s', a, b) P(s' \mid a, b)}{P(z \mid b, a)}
\end{align}
It then turns out that (I will maybe explain this more in detail later)
\begin{align}
P(s' \mid z, a, b)
&= \frac{P(z \mid s', a, b) P(s' \mid a, b)}{P(z \mid b, a)} \\[0.7em]
&=\frac{O(a, s', z) \sum_{s\in S} b(s)T(s, a, s')}{\sum_{s \in S}b(s)\sum_{s' \in S} T(s, a, s')O(a, s', z)} \\[0.7em]
&=b'(s')
\end{align}
So, by assumption, $P(z \mid b, a) = \sum_{s \in S}b(s)\sum_{s' \in S} T(s, a, s')O(a, s', z)$ must be different from zero for the equality above to hold.
You can see this from a very simply example of the Bayes' theorem. Let $P(A \mid B) = \frac{P(B \mid A) P(A)}{P(B)}$. Now, intuitively, $P(A \mid B)$ (which is what we want to calculate using Bayes' theorem) means the probability of $A$ occurring given that $B$ has occurred, which means that $B$ couldn't have had a probability of $0$ of happening if we wanted to calculate $P(A \mid B)$, so $P(B)$ couldn't have been zero if we wanted to calculate $P(A \mid B)$ using Bayes' theorem. We can also apply this reasoning to the definition of $b'(s')$ above.
For completeness, note also that, in the normalisation factor $$\sum_{s \in S}b(s)\sum_{s' \in S} T(s, a, s')O(a, s', z),$$ $b(s)$, $T(s, a, s')$ and $O(a, s', z)$ are probability distributions, which means that not all terms of $b(s)$, $T(s, a, s')$ and $O(a, s', z)$ can be zero, for all $s$, $s'$ and $a$ (given they must sum up to $1$).
Note also that $\sum_{s' \in S} T(s, a, s')O(a, s', z)$ is a convex combination of all $O(a, s', z)$ (for all $s'$), where the coefficients are $T(s, a, s')$. The normalisation factor is also a convex combination where the coefficients are $b(s)$ (for all $s$).


text:
How can I generate a document from a single word using GPT or BERT?
I have a dataset of 100000 documents each labelled with a topic to it. I want to create a model such that, given a topic, the model can generate a document from it.
I came across language models GPT, GPT-2 and BERT. I learned that they can be used for generation purposes. But I did not find anywhere whether they can generate sentences given only a word.
I am inclined to use GPT for my task, but I am not sure how to proceed with it. I wanted to know whether it is possible or not? It would be helpful if anyone can help me give a start in the right direction.

Why rely on models like GPT and do it manually from the array?

Because I want to leverage the linguistics knowledge of the pre-trained models and just fine tune it to my specific task.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Is there any system that generates website designs?
I was thinking about what I can do for my thesis in upcoming academic semester, and I came across an idea. The idea is like: "If there is any kind of system that generates website designs itself." If no, then I can go for it, and I will be lucky if anyone has the same idea. We can collaborate. In case, if there's any project or system (Open source or not) that can do this or has been initiated in this context, I want to contribute solely. If anyone has any clue or knowledge over this kind of system, please do inform me. As I haven't done anything like this before, I want to learn. Any kind of suggestion or assistance on this idea will be so helpful for me.

The closest research that I am aware of is in artificial intelligence designed user interfaces(1,2).
The scope of which vary tremendously. Some teams are trying to generate UIs based on some user-defined parameters, others are trying to generate based off of images(as in the second link). I think part of the reason research is focused here is two-fold: One, we aren't very good at generating UIs automatically yet, and 2, it is an exponentially harder problem to develop a backend that has integrated calls and features required for a fully-featured website or application.
With that being said, one could adapt this research into things like one-page sites or other simple implementations that might bear fruit.

i thought of this system like, it'll only design frontend and not backend. Will it be fruitful to spend my important time for it? What do you think? I'm seeking motivation actually and really want to implement a system like this.

I don't like recommending research paths, but if you have a significant interest in the problem I would say yes.

That's okay actually. I'd love to learn from you.

Cool, if you deem my answer acceptable, go ahead and check it. If you want, give me a way to correspond and I will do so

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Cold start collaborative filtering with NLP
I’m looking to match two pieces of text - e.g. IMDb movie descriptions and each person’s description of the type of movies they like. I have an existing set of ~5000 matches between the two.
I particularly want to overcome the cold-start problem: what movies to recommend to a new user? When a new movie comes out, to which users should it be recommended?
I see two options:

Run each description of a person through an LSTM; do the same for each movie description; concatenate the results for some subset of possible combinations of people and movies, and attach to a dense net to then predict whether it’s a match or not
Attempt to augment collaborative filtering with the output from running the movie description and person description through a text learner.

Are these tractable approaches?

From what I understood you will not have any cold start problem because you basically process the user preferences description against movies descriptions to get recommendations. So you don't use other users feedback at any time of the process which is not collaborative filtering.

Here is instead the approach I would suggest in your case to get movies recommendations for each user:

Compute the similarity between the user description and each movie description. This can be done using the Universal Sentence Encoder. It's an 2018 paper by Google which represents any sentence as a vector of 215 values (i.e embeddings).
The semantic similarity between 2 sentences is computed using the dot product of their embeddings. Fortunately, the implementation was integrated to Tensorflow Hub and easily be used (see my answer here for details).
Choose the highest similarity values and recommend the corresponding movies to the user. Notice that you can still use this approach along with a collaborative filtering one.


text:
What causes a model to require a low learning rate?
I've pondered this for a while without developing an intuition for the math behind the cause of this.
So what causes a model to need a low learning rate?

I wondered about it too and I am curious why RNNs have a smaller learning rate than CNNs. From what I know, model complexity (deepness) and/or huge datasets require a finer hyperparameter for the lr.

Gradient Descent is a method to find the optimum parameter of the hypothesis or minimize the cost function.

where alpha is learning rate
If the learning rate is high then it can overshoot the minimum and can fail to minimize the cost function.

hence result in a higher loss.

Since Gradient descent can only find local minimum so, the lower learning rate may result in bad performance. To do so, it is better to start with the random value of the hyperparameter can increase model training time but there are advanced methods such as adaptive gradient descent can manage the training time.
There are lots of optimizer for the same task but no optimizer is perfect. It depends on some factors

size of training data: as the size of the training data increases training time for model increases. If you want to go with less training model time you can choose a higher learning rate but may result in bad performance.
Optimizer(gradient descent) will be slow down whenever the gradient is small then it is better to go with a higher learning rate.

PS. It is always better to go with different rounds of gradient descent

This is good start, as it shows the difference between low and high learning rates in general. You also need to explain why the good learning rate varies depending on the task - and the OP was specifically asking why some problems require a lower learning rate than others

That's a good point. I have edited it. Since there is not a specific problem is mention I am going with general one.

I still think that this does not answer the question. The OP is not asking about the optimiser or data, it is asking about the model. How does the model (its architecture, number of parameters, etc.) affect the learning rate? I think this is the actual question, which you do not answer. Everything else is quite irrelevant to the question and will only confuse readers that can't distinguish between these concepts.

Thanks for the feedback. Irrespective of the model architecture as the number of the parameter, size of data and range of the data (solution use normalized data) is high result in the higher training time so according to it, we should change the learning rate. This applies for the model such as linear regression, logistic regression, SVM etc since they use GD for optimization. Any response is always welcome :)

Any proof that assesses your claim "irrespective of the model architecture"? This answer still does not answer the OP question. You're answering to the question "how does the learning rate change in general, depending on the machine learning setting" (and your answer is not exhaustive, of course, because it does not mention "how the learning rate changes depending on the model", i.e. the actual question).

What is machine learning setting? Gradient descent is algorithm which takes some parameter, Algorithm does not change with change in model it remains same. Of course there is change in number of parameter, size, range of value etc.

I mentioned that gradient descent find out local minimum by considering non convex curve and there are also other better optimizer are present.


text:
What is the difference between A2C and running an agent in an environment vector?
I've implemented A2C. I'm now wondering why would we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector?
I personally think this will be more efficient since now all actions can be calculated together by only going through the network once. I've done some tests, and this seems to work fine in my test. One reason I can think of to use multiple actors is implementing the algorithm across many machines, in which case we can have one agent on a machine. What else reason should we prefer multiple actors?
As an example of environment vector based on OpenAI's gym
class GymEnvVec:
    def __init__(self, name, n_envs, seed):
        self.envs = [gym.make(name) for i in range(n_envs)]
        [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]

    def reset(self):
        return [env.reset() for env in self.envs]

    def step(self, actions):
        return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))

Can you clarify what you mean by an "environment vector"?

Hi @PhilipRaeisghasem, please refer to this for an example.

@Maybe Where did you take the code related to GymEnvVec?

I believe if you run a single agent in multiple parallel environments many times you will get similar actions in similar states, the reason behind multiple agents is that you will have different agents with different parameters and you can also have different explicit exploration policies so your exploration will be better and you will learn more from environment (see more state space). With single agent you can't really achieve that, you would have a single exploration policy, single parameter set for the agent and most of the time you would be seeing similar states (at least after a while). You would be speeding up your learning process but that's just because you're running multiple environments in parallel (compared to the regular actor-critic or Q-learning). I think quality of learning would be better with multiple different actors.

In addition, the multiple agents approach is used for similar reasons to DQN's experience replay - it helps to reduce correlation within training batches. This can be important for stability of a neural network within the A2C framework.

Hi, I still feel confused. In general, A2C is implemented with an on-policy policy-gradient algorithm, If we use different parameters for different agents, then the learning is not on-policy at all. Why would we do that? BTW, @NeilSlater, I cannot see how multiple agents reduce correlation in the way multiple environments do not. Could you please shed some light on it?

@SherwinChen: There is no difference between "multiple agents" and "multiple environments" by default, unless you are deliberately connecting the agents in the former (e.g. forcing them to the same state on each time step) or deliberately changing the environments in the latter (e.g. changing the state transition rules). Neither of those things are suggested here. The point is to reduce correlation. In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources.

@NeilSlater, thanks for responding, but I'm sorry that I cannot really understand what you were trying to convey at the end: "In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources." I've updated my question to include a simple environment vector, I cannot see why the data will be less i.i.d. than that collected by multiple agents. I'm looking for your response

@SherwinChen: A single agent's vectors will be correlated. E.g. you can predict the next observation to train from with some degree of accuracy better than random, even knowing nothing about the agent. Whilst if you take the next observation from a different agent, your ability to predict what the state is in advance is reduced. Self-correlated data is not i.i.d. and you want to avoid that, especially when training incrementally using online learners such as neural networks

Hi @NeilSlater, and thanks again. I'm still quite confused about why, in the case of using environment vector, we can predict the next observation to train from in the way that we cannot do the same to multiple agents. In A2C, even we use multiple agents, they are just nothing but copies of the learning agent since every time we update the learning agent, we synchronize the updated parameters with all agents. Environments are different in both cases since we use different seeds. What on earth makes the difference? Or am I misunderstanding something?

@SherwinChen: Clearly I am not able to explain this in your terms in the comments. Consider asking another question on the site

@NeilSlater Awesome explanation Neil!

When you run multiple actors you actually make copies of your current agent and run these copies in non-vector environments. The environments are usually also the same.
In the case of Atari, yes of course, using a vectorized environment is much more efficient. But if you want to train an agent to play DotA, or drive a car, then there is no vectorized environment.
Running different actors instead of copies of the same agent is not really possible with on-policy algorithms. You have to collect data using one and the same policy, otherwise it is not on-policy.
A slight exception to this is the case of asynchronous training. In this case you start with the same copies of your agent, but update these copies asynchronously, and only once in a while sync the policies. However, you still have to make sure the actors' policies don't deviate too much from one another. But note that the synchronous approach maybe performs better than the asynchronous. And again, in case you can vectorize your environment I don't really see the need to do this.


text:
How to predict human future location?
I have billions of anonymized location coordinates of people movement collected from app. I want to improve user experience by using location data.

For example identify if user is at home or at office so that what they view in app changes.
Where will they be tomorrow at particular hour -  so that I can suggest to secure their homes from IOT device if they are out.

Regarding the first point, I tried to use the following rule: at night they are at home, and at day they are at work or school. Regarding the 2nd point, I have no idea how to proceed. Is there any way I could use AI to predict future location and home location?

point 1 really doesn't need machine learning, you can just ask the user to set their home address (they probably already have that in their address book anyway). From code you can match if the current location is close to the home address.

@Kokodoko the app don't have this function.

Point 1 you can still solve just by analyzing the data. You assume they are home at night and work during the day. So you can always compare their current location with those locations.

Your problem is often called (in the literature) human mobility prediction. There has been some research in this area. Have a look at it on the web.
In general, you might want to use any statistical or machine learning model that uses the historical data to predict the future. For example, you could try to use an hidden Markov model (for both point 1 and 2). However, before that, you might also need to do some feature engineering, if you only have locations (and not e.g. the time of the day when those locations were recorded).

I will try to find some materials.


text:
Similarities and differences between UCT algorithms in (i), (ii), (iii) and (iv)?
I am trying to understand the similarities and differences between: (i) the UCT algorithm in Kocsis and Szepesvári (2006); (ii) the UCT algorithm in Section 3.3 of Browne et al (2012); (iii) the MCTS algorithm in Silver et al. (2016); (iv) the MCTS algorithm in Silver et al. (2017).
I would be really grateful for some help identifying the similarities and differences in these papers, I am doing some research and really struggling right now.
(i) http://ggp.stanford.edu/readings/uct.pdf
(ii) http://mcts.ai/pubs/mcts-survey-master.pdf (Section 3.3)
(iii) https://storage.googleapis.com/deepmind-media/alphago/AlphaGoNaturePaper.pdf
(iv) https://deepmind.com/documents/119/agz_unformatted_nature.pdf

The algorithm in the 2012 survey article (your second link) is the most common / standard implementation. Whenever someone mentions using MCTS or UCT, without explicitly stating any other info, it's safe to assume that that pseudocode is what they're using.
The paper by Kocsis and Szepesvári from 2006 (your first link) is (one of) the original publication(s) on UCT. It is very similar to the "standard" implementation as described in the survey paper. If I recall correctly, the only important difference is that the algorithm as described in 2006 keeps track of at which point in time during an episode a reward is observed, and accounts for that timing in the backpropagation phase (i.e. if a reward is observed at time $t$ in an episode, credit for that reward is not assigned to states/action after time $t$). It also used a discounting factor $\gamma$ to discount the importance of reward depending on temporal distance, which is uncommon otherwise in MCTS literature.
Both of those differences are due to the original, 2006 paper having more of a "Markov decision process" or "Reinforcement Learning" flavour, whereas otherwise MCTS has especially become popular in AI for (board) games where we often by default assume that there is only a single nonzero reward (win or lose) at the end of every episode anyway, which makes those differences less meaningful.

Both of the AlphaGo papers (AlphaGo and AlphaGo Zero) use a "foundation" of MCTS that is mostly similar to the one from the 2012 survey article. The core components are all the same. Both of those systems add on a lot of (very important, and some quite complex) bells and whistles though.
Going off the top of my head (should be mostly accurate, but best details are in the original source of course!) AlphaGo (your third link) added Neural Networks trained in various ways to output policies (mapping from states to probability distributions over actions) and value functions (mapping from states to "value estimates", or estimates of win percentage). Trained policy networks were used in a variant of the "selection" phase (no longer UCB1 strategy as in UCT) to bias selection. A different (simpler function, not a deep net) policy was used to run play-outs (no long uniform at random action selection as in UCT). A combination of observed reward at end of play-out + value function estimate at start of play-out was used for backpropagation (no longer only backpropagating reward observed in play-out as in UCT).
AlphaGo Zero is, if we're purely looking at the MCTS part, quite similar to AlphaGo. The Neural Networks were a bit different (a single one, with multiple outputs for policy + value), and play-outs were no longer used at all (just immediate backpropagation of value function estimates after MCTS' selection + expansion phases). Apart from that, the primary differences going from AlphaGo to AlphaGo Zero were in the learning process used to train the Neural Networks really, not so much in the MCTS side of things.


text:
What are some possible projects that can be finished in a 4-week time-frame?
I am taking AI this semester and we have a semester project that will last 4 weeks. We can choose just about anything.
So, what are some possible semester projects that can be finished in a 4-week time-frame?
Some background information: I am a graduate student in CS, but this is my first AI course. My research area is in the space of data mining and analytics. I am open to doing anything that seems interesting and creative.

There are plenty of impressive answers already - all I'll add is that I'd love to collaborate on projects, if you (or anyone reading this) would like that. Feel free to get in touch with me!

@cogito_ai You could ask a question on the site. For example, "Are there sites where people can form teams to solve AI problems? One such site would be Kaggle, but there may be others. Do you know of any?" I think it's consistent with our scope, but just make sure not to ask for opinions. However, note that a similar question https://ai.stackexchange.com/q/3515/2444 has already been asked. Maybe you could ask for alternatives or maybe you are interested in a specific topic, project or AI subfield. It depends on what you really want.

Welcome to AI.SE @Kate_Catelena!
I teach AI courses at the undergraduate level, and so have seen a lot of semester projects over the years. Here are some templates that often lead to exciting outcomes:

Pick a new board or card game, and write a program to play it. Your course has probably covered Adversarial Search, and may also have covered Monte Carlo Tree Search, or self-play reinforcement learning approaches. These projects are often fun to mark and creative because they are easy enough to be well done, and yet there are always new, exciting, domains to apply these algorithms to. Some examples of past projects that I thought were neat were an AI to play the boardgame Tac (mostly A* Search), and an AI to play the card game Love Letter (mostly Counter-factual minimax regret, the algorithm used to solve poker).
Pick a question that you would like to know the answer to, that could be addressed with machine learning. Then implement your own ML algorithm (decision tree learners are fairly easy), gather your own data, and show a result. Examples of interesting projects I've seen in the past are using ML to find out which of a number of factors most strongly influenced a students' subjective quality of sleep; and which items are most commonly purchased along with camping supplies (using association rule mining).
Anything involving reinforcement learning. RL projects are always neat to see if accompanied by a visualization showing the learner's behavior at different stages. A strong past project involved a student simply replicating Sutton & Barto's Acrobot experiments with their own implementation of the SARSA-Lambda algorithm.  Other things that might be neat include making a trainable "pet" that the user can influence, or solving games using self-play.
Theoretical results might seem intimidating but are often more accessible than one might think, especially if your discrete math skills are strong. I have had many student projects where the student went away to look at theory papers in ML or Multiagent Systems, found a suggestion in the future work sections that wasn't a big result, but that was fairly easy to prove, and proved it. Sometimes these are even publishable.
Replications. Go find an interesting AI paper (use scholar.google.com), and then see if you can do exactly what the authors suggest and if you get the same result or not. Then, if you have time, see if you can improve on the results. These are often most interesting when you find a paper written in a different field that uses AI. Often the authors of such papers know less about AI than you do, and so it can be fairly easy to improve on their results. I have had several students do projects like this to great effect.

Those categories are a bit vague, but remember: AI touches almost anything. Pick your favorite hobby, and see whether you can relate it to AI using one of the approaches above. Nothing makes a project stand out like one that applies AI to solve some real issues in an exciting domain. Good luck!

Here are some possible options

Music Generation using GA/MA
Open AI's gym projects
2048 on RL and search algorithms
Fixing bugs in the source code of some AI software project


text:
Why is description logic decidable but first order logic is not decidable?
Description logic is a fragment of first order logic, but description logic is decidable and first order logic not decidable. Why is that? what is the role of variables in first order logic to make it undecidable?

This has to do with the fact that you can define arithmetic inside the axiomatic system or not. In description logic you cannot speak about arithmetic sentences and in first order logic you do.
if you look at the proof of incompleteness you will understand this in depth. This demonstration depends on an arithmetic coding of statements, and this representation is fundamental to obtain the conclusion.

This seems like a good answer, but it would be a lot more useful if it cited references. Could you edit your answer to include relevant links?

Sorry its not clear to me yet... can you give examples?


text:
What are the technological challenges that AI faces today?
I am writing a field report on AI. I was wondering what the technological challenges are that AI is facing today. I have written the following so far.

AI needs common sense like a human common
AI needs curiosity
AI needs to understand cause and effect
AI needs to be creative

Are there any other hardware-tech-related obstacles?

Welcome to AI.se! Your question is very broad. I recommend reading the AI Wikipedia page and exploring the outline it has on the side.

There are several challenges, but they depend on the ultimate goal of AI researchers (and, in general, people that are developing AI). Is the ultimate goal the creation of human-like artificial intelligence (strong or general AI)? Or is it just the creation of narrow AI that can be applied to certain fields/problems?

Computational Creativity is not an unassailable challenge (depending on who you talk to;) Philosophers have claimed algorithms can't be creative, but Marcel Duchamp, one of the most significant artists in modernity, famously stated that:

"All artists are not chess players, but all chess players are artists"

This would seem to have been validated by commentators referring to move 37 in game 2 of  Lee Sedol's match with AlphaGo as "beautiful"  (In the game of Go, aesthetics are considered in regard to strategy, not just outcomes.)  The takeaway is it was a choice humans would never have considered, because the structure of our brains is different, and thus our approach to creativity different, than automata.
Current algorithmic creativity is a function of monte carlo methods, which utilize randomness, Monte Carlo Tree Search as a major method. MCTS has great utility in intractable models such as non-trivial combinatorial games, which produce complexity akin to nature, so it's not surprising it's slowing extending to real world applications.
The main issue with computational creativity is it is still not as efficient as human creativity, requiring a great deal of processing power for non-trivial problems.  (This was why it took so long for a computer to beat the best human a Chess--human insight seems rooted in semantics/understanding.)
Procedural content generation is an area of research that is steadily progressing, and includes games, music and visual art.

Algorithmic Bias is the most pressing issue facing the field of AI and Machine Learning methods specifically, which are statistical.  (If the dataset is incorrect, imcomplete or biases, the output will be biased.)

@DuttaA It's based on the idea that humans cannot go nearly as deep down a game tree as automata.  (How many move ahead can a Chess or Go player realistically think?  Maybe 15 plies, but that may not even be necessary.  By contrast, current automata have to go much deeper, and conduct more self-play games than a human could play in a hundred lifetimes, to match human strength.)


text:
How does Hindsight Experience Replay cope with multiple goals?
What if there are multiple goals? For example, let's consider the bit-flipping environment as described in the paper HER with one small change: now, the goal is not some specific configuration, but let's say for the last $m$ bits (e.g. $m=2$), I do not really care if there is $1$ or $0$.
In the paper, there is section 3.2 multi-goal RL, where they mention an example with two-dimensional coordinates ($x$ and $y$), but they are interested only in the $x$ coordinate, so they only use the $x$ coordinate as a goal.
Applying this strategy to my example would result in cutting the last $m$ bits from the goal and only use the other bits. Is this logic correct?
Another approach I could think of would be to train with all possible goal configurations, as there are not many in my case. But this seems impractical as the number of goal configurations grows.


text:
How can we encourage data analysts and decision-makers to adopt AI?
Nowadays, there is too much data for humans to work on alone, and it is very normal for data analysts to use AI techniques to treat and process these data so it can lead to a faster and more accurate result. But many data analysts and decision-makers still don't trust AI methods or techniques and are reluctant to use them. How can we encourage them to accept or prefer these AI solutions?
For example, if AU gives advice to solve a problem, then decision-makers must trust the results and data analysts must trust the mechanism, so that decision-makers can be confident in the AI work and also data analysts can concentrate their activities on added value.
How can we encourage data analysts and decision-makers to adopt AI?

Not all of the mistrust aimed at AI systems is unjustified, particularly when it comes to neural networks and other such systems that rely on large training data sets. There are a number of high profile cases, facial recognition being one that has often (understandably) received a lot of flak, where improperly configured training data has resulted in skewed and questionable results.
If you want to foster more trust in the systems, it will require better tools for analyzing how they are approaching problems and reaching decisions, as well as determining if there are holes in the training data. It will require a community working in the field that gives a lot more thought and care to how they approach their training data and what unintended biases they may be introducing by forgetting something than has often been displayed presently.
Of course, some people are just distrustful of new technology, but I think it's more interesting to address the more legitimate concerns.

I think it would be interesting to cite a case where NNs failed catastrophically or were highly "biased".

I really like the fact you mention to use better tools for analyzing on how data analysts and decision makers are approaching problems and reaching decisions. But can you think of may be one tool that can could help in the process? May be in any industry sector? and what do you exactly mean by holes in the training data? is it like a bad(faulty) datasets?


text:
Skip-Gram Model Training
Suppose we want to predict context words $w_{i-h}, \dots, w_{i+h}$ given a target word $w_i$ for a window size $h$ around the target word $w_i$. We can represent this as: $$p(w_{i-h}, \dots, w_{i+h}|w_i) = \prod_{-h \leq k \leq h, \ k \neq 0} p(w_{i+k}|w_i)$$ where we model the probabilities of a word $u$ given another word $v$ as $$p(u|v) = \frac{\exp(\left<\phi_u, \theta_v \right>)}{\sum_{u' \in W} \exp(\left<\phi_{u'}, \theta_v \right>)}$$ where $\phi_u, \theta_v$ are some vector representations for words $u$ and $v$ respectively and $\left<\phi_u, \theta_v \right>$ is the dot product between these vector representations (which represents some sort of similarity between the words) and $W$ is a matrix of all the words.
In Skip-Gram Negative Sampling, we want to learn the embeddings $\phi_u, \theta_v$ that maximize the following: $$\sum_{u \in W} \sum_{v \in W} n_{uv} \log \sigma(\left<\phi_u, \theta_v \right>) +k \mathbb{E}_{\bar{v}} \log \sigma(-\left<\phi_u, \theta_{\bar{v}} \right>)$$

Question. How exactly does this work? For example, suppose $k=5$, the target word $w_i$ is $\text{apple}$ and we want to find
  $p(\text{pie}| \text{apple})$. Let $n_{uv} = 10$ (number of times pie
  co-occurs with apple). Then we sample $5$ random words $\bar{v}$ that
  did not occur with $\text{apple}$ and whichever term in the sum is
  bigger is the one we predict? For example, if the first term in the
  sum is larger than the second term then we would predict that
  $p(\text{pie}| \text{apple}) \approx 1$? Otherwise we predict that
  $p(\text{pie}| \text{apple}) \approx 0$? Is this the correct
  intuition?

Source. Here at around the 10:05 mark.

Almost, but no. When you maximize that objective function, you do so by adjusting the parameters $\phi$ and $\theta$. After you're done with training, you can use your word embeddings for other NLP tasks. You don't, however, do any prediction directly from the skip-gram model.
To maximize the first term, co-occuring words must have large inner products. That is, they must be "similar". To maximize the second term**, the randomly sampled words must have a small inner product with $\phi_u$. That is, they must be "dissimilar" to $\phi_u$. Moving these word embeddings around in the vector space to make some words similar and others not is the only thing that happens during skip-gram training.
$$$$
** $\sigma(-x)=1-\sigma(x)$, so maximizing $\sigma(-x)$ is minimizing $\sigma(x)$

But isn't the idea of using Skip-Gram to bypass the computational complexity of the log-likelihood maximization? So instead of predicting a word given another word, we are predicting whether certain words appear together or not (implicitly)? See 9:43 of the video.

No. But that idea is similar to one behind the GloVe method of creating word embeddings. Also, I'm not really a fan of the video you keep linking. Higher quality videos from Stanford's 224d class are here.

Why aren't you a fan of the video?


text:
Is it possible to analyse a very large 3D input volume at once with a 3D CNN?
I would like to use a 3D convolutional network on a 2000x2000x2000 volume for segmentation. I know I can break the volume into chunks that can fit in VRAM, but I was wondering if there was a way to analyze the entire 3D volume at once.

Hi, welcome to AI.se! You'll probably have better luck with this question over at DataScience or StackOverflow.


text:
Are there any advantages of using rules-based approaches versus models for detecting spam?
Suppose that we have unlabeled data. That is, all we have are a collection of emails and want to determine whether any of them is spam or not. Let's say we have $1,000$ rules to determine whether a particular email is spam or not. For example, one rule could be that a sender's email address should not contain the text  no_reply . If an email does contain this text, then it would be classified as spam.
What are the advantages/disadvantages of a rules-based approach for detecting spam vs. a non-rules-based approach/unsupervised methods for detecting spam?
Would there even be a point in constructing a non-rules based model given that we already have a rules-based model? Could we use the rules-based model to create some labeled training data and then apply supervised techniques?

Yes, there would be a point. Assuming your rule set is accurate, then you can use data classified with it to train a model. This model can be expected to be more robust and properly categorise emails that your rule-set will not handle.
Why? Machine learning algorithms generally work on features, and identify relationships between those features that lead to a classification decision. A human rule author basically does the same, but they might not notice subtle relationships; a good ML algorithm, however, might pick those up.
So you could have a hybrid model, where you first use your rule-based classifier, and anything that does not get classified is then run through the ML classifier.

This. But you should also take into account that your rules and therefore your training data have incorrect classifications. For example, no_reply as a localpart is relatively common for automatically generated emails that are not spam at all, but requested by the recipient. My experience (I've built a deep learning spam classifier) is that with a very basic feature set reasonable results can be achieved (after training on manually classified samples).

However, for production quality, you need several iterations of looking at the results (especially misclassifications) and adjusting feature extraction.


text:
Why is KNNBasic better than KNNWithMeans with the default parameters, but KNNWithMeans performs better with folds?
I'm learning a bit about the use of the Surprise library and I have a set of data with users and ratings. I'm training a network with this library, using KNNBasic and KNNWithMeans, this last algorithm is the same as KNN but averages the ratings before calculating the distances between the points.
If I don't use any measure of similarity, i.e. using the two algorithms with the default parameters, KNNBasic predicts the results better than using KNNWithMeans. But if I train the nets using subsets, 10 folds, where the algorithm iterates over 9 of them for training and the other one for validating, KNNWithMeans gives better results.
Do you know why this can happen? Why KNNBasic is better in the first case, and increasing the number of folds is better KNNWithMeans?


text:
Reinforcement learning for inventory management with dynamic changes to available products
Consider a shop owner who has to deal with having to buy for one week from a different supplier with several different brands. Another week a brand is removed or added from the market. Yet another week, the manager decides to skip three assortments of fruit soda and exchange them with a different selection of three assortments of fruit sods, and so on and so forth.
Is dealing with such issues possible with existing public implementations of reinforcement learning?
The only research that I saw dealing with dynamically changing stuff like this is the DeepMind FTW bot playing Quake capture the flag. It deals with changing layouts to the map being played, but the implementation is not public and it doesn't resemble the inventory management situation I outlined.


text:
Why is the max a non-expansive operator?
In certain reinforcement learning (RL) proofs, the operators involved are assumed to be non-expansive. For example, on page 6 of the paper Generalized Markov Decision Processes: Dynamic-programming and Reinforcement-learning Algorithms (1997), Csaba Szepesvari and Michael L. Littman state

When $0 \leq \gamma < 1$ and $\otimes$ and $\oplus$ are non-expansions, the generalized Bellman equations have a unique optimal solution, and therefore, the optimal value function is well defined.

On page 7 of the same paper, the authors say that max is non-expansive. Moreover, on page 33, the authors assume $\otimes$ and $\oplus$ are non-expansions.
What is a non-expansive operator? Why is the $\max$ (and the $\min$), which is, for example, used in Q-learning, a non-expansive operator?

In laymen's terms, a non-expansive operator is a function that brings points closer together or at least no further apart.
An example of a non-expansive operator is the function $f(x) = x/2$. The two numbers $0$ and $5$ are a distance of $5$ apart. The two output numbers $f(0) = 0$ and $f(5) = 2.5$ are 2.5 apart (which is smaller than $5$ apart). It is easy to see that $f$ brings everything closer together except when the two input numbers are the same: in which case, the distance between the outputs of the function at those numbers is at least no further apart than distance between the two input numbers.
$\max$  is a two-input function (or n-input, but the intuition should be clear from the 2-input case).  We can think of max as a function that maps pairs of numbers $(x, y)$ to single numbers (picking whichever of $x$ and $y$ is larger).
Suppose that we chose to measure the distance between pairs using Euclidean distance, and the distance between single numbers using the Euclidean distance as well. Here's an example:
The distance between (0,0) and (3,3) is $\sqrt{3^2 + 3^2} = \sqrt{18}$. The distance between $\max(0,0) = 0$ and $\max(3, 3) = 3$ is $\sqrt{{(\max(0,0) - \max(3, 3))}^2} = \sqrt{9} = 3$.
Let's consider the general case. The Euclidean distance between the 2D points $(a, b)$ and $(c, d)$ is $\sqrt{(a-c)^2 + (b-d)^2}$. There are four cases to consider:

Suppose that $a\geq b$ and $c \geq d$. In this case, the distance between max(a,b) and max(c,d) is just |a-c|, which is clearly at most $\sqrt{(a-c)^2 + (b-d)^2}$.
Suppose that $a\leq b$ and $c \leq d$. In this case, the distance is |b-d|, which is also at most the original distance.
Suppose that $a\geq b$ but $c \leq d$. Then the distance is |a-d|. Suppose that $a > d$. Since $d > c$, then $|a-d| <= \sqrt{(a-c)^2 + (b-d)^2}$ since |a-d|<=$\sqrt{(a-c)^2}$, and a symmetric argument holds for the case $d > a$.
$a\leq b$ but $c \geq d$, we can construct an argument identical to the one for case 3 above.

Since max is always bringing things closer together, or at least, no further apart, it is a non-expansive operator.

I haven't gone through your whole answer in detail, but you consider only the L2 norm, the Euclidean distance metric. Is the $\max$ operation non-expansive for other notions of distance?

@PhilipRaeisghasem That's a good question. I tried to keep the answer closer to laymen's terms. In fact, I think the analysis will hold for any metric (it requires the triangle inequality). The more formal definition of a non-expansive operator does state that it holds only for functions between metric spaces, so I think that is correct.

Suppose that $X$ and $Y$ are metric spaces. A metric space is a set equipped with a metric, which is a function that defines the intuitive notion of distance between the elements of the set. For example, the set of real numbers, $\mathbb{R}$, equipped with the metric induced by the absolute value function (which is a norm). More precisely, the metric $d$ can be defined as $$d(x,y)=|x - y|, \forall x, y \in \mathbb{R} \tag{1}\label{1}.$$
Let $f$ be a function from the metric space $X$ to the metric space $Y$, that is, $f: X \rightarrow Y$. Then, $f$ is a non-expansive map (also known as metric map) if and only if
$$d_{Y}(f(x),f(y)) \leq d_{X}(x,y) \tag{2} \label{2}$$
where the subscript $_X$ in $d_X$ means that the metric $d_X$ is the metric associated with the metric space $X$. Therefore, any function $f$ between two metric spaces that satisfies \ref{2} is a non-expansive operator.
To show that the max operator is non-expansive, consider the set of real numbers, $\mathbb{R}$, equipped with the absolute value metric defined in \ref{1}. Then, in this case, $f=\operatorname{max}$, $d(x, y) = |x - y|$ and $X = Y = \mathbb{R}$, so \ref{2} becomes
$$|\operatorname{max}(x) - \operatorname{max}(y)| \leq | x - y | \tag{3} \label{3}$$
Given that $\operatorname{max}(x) = x, \forall x$, then \ref{3} trivially holds, that is
\begin{align}
|\operatorname{max}(x) - \operatorname{max}(y)| &\leq | x - y | \iff \\
|x - y| &\leq | x - y | \iff \\
|x - y| &= | x - y | \tag{4} \label{4}
\end{align}
For example, suppose that $x=6$ and $y=9$, then \ref{4} becomes
\begin{align}
|\operatorname{max}(6) - \operatorname{max}(9)| &\leq | 6 - 9 | \iff \\
|6 - 9| &\leq | -3 | \iff \\
|-3| &= 3
\end{align}
There are other examples of non-expansive maps. For example, $f(x) = k x$, for $0 \leq k \leq 1$, where $f : \mathbb{R} \rightarrow \mathbb{R}$.
See also https://en.wikipedia.org/wiki/Contraction_mapping and https://en.wikipedia.org/wiki/Contraction_(operator_theory).


text:
Can machine learning be used to improve the average case complexity of an algorithm?
I am developing an algorithm that, in certain moment, must explore an exponential number of objects derived from a graph:
for o in my_graph.getDerivedObjects():
  if hasPropertyX(o):
    process(o)
    break;

If one of the derived objects has property $X$, then the algorithm process it and then stops. The theory ensures that at least one of these derived objects has property $X$. Now, I strongly suspect that there is a strong correlation between some topological aspects of the graph, and which derived objects actually have property $X$. I want to predict some of the derived objects that have property $X$ using Machine Learning. So, the idea is:

Predict a derived object $o$ that supposedly has property $X$ - or maybe predict $n$ of them for some number $n$.
If any of them is useful, I use them. If not, I run the exponential algorithm.

Of course, this isn't an optimization in the worst-case complexity of the algorithm. Also, I suppose I should also develop some statistical tests in order to show that the prediction algorithm actually works.
Is this type of optimizations common? Could you please provide some examples? The literature on the subject would also be greatly appreciated.

To the best of my knowledge, there haven't yet been many academic publications in this area, which could be broadly said to fall within Search-Based Software Engineering. Here are the ones I know of.

Jerry Swan and Nathan Burles. Templar - A Framework for Template-Method Hyper-Heuristics. In: Genetic Programming - 18th European Conference, EuroGP 2015, Copenhagen, Denmark, April 8-10, 2015, Proceedings. 2015, pp. 205–216. DOI: 10.1007/978-3-319-16501-1_17.

This paper describes 'Hyper-quicksort', a quicksort variant that uses Machine Learning (ML) to generate a pivot function:

A. E. I. Brownlee, N. Burles, and J. Swan. Search-Based Energy Optimization of Some Ubiquitous Algorithms. In: IEEE Transactions on Emerging Topics in Computational Intelligence 1.3 (2017), pp. 188–201.

This paper uses ML to generate energy-efficient variants of some widely-used algorithms

David R. White, Leonid Joffe, Edward Bowles, and Jerry Swan. Deep Parameter Tuning of Concurrent Divide and Conquer Algorithms in Akka. ISBN: 978-3-319-55792-2.

This paper uses ML to optimise the FFT, matrix multiplication and quicksort for concurrency

Nathan Burles, Edward Bowles, Alexander E. I. Brownlee, Zoltan A. Kocsis, Jerry Swan, and Nadarajen Veerapen. Object-Oriented Genetic Improvement for Improved Energy Consumption in Google Guava. DOI: 10.1007/978-3-319-22183-0_20.

This one optimises Google Guava for energy consumption

Zoltan A. Kocsis, Geoff Neumann, Jerry Swan, Michael G. Epitropakis, Alexander E. I. Brownlee, Saemundur O. Haraldsson, and Edward Bowles. Repairing and Optimizing Hadoop hashCode Implementations. In: Search-Based Software Engineering: 6th International Symposium, SSBSE 2014, DOI: 10.1007/978-3-319-09940-8_22.

This one fixes broken hashCodes by using ML to generate new ones

There was also one (from Microsoft Research, I think) entitled something like "The case for self-adjusting data structures". I'll add an edit if I can find it.


text:
How can we recognise musical notes in low-resolution or blurry images?
I was looking for an approach to recognise musical notes from photos.
I found this repository https://github.com/mpralat/notesRecognizer. However, it doesn't seem good enough. If you look into the bad folder, you can see that just tiny variations of lightning can already cause problems. One should be able to read musical notes with lower quality images.
I found other projects. However, they all use high resolution images.

https://github.com/suyalcinkaya/music-note-recognition (example of image)
https://github.com/nikolalsvk/note-play (example)

Now, this is unsatisfying. If you want to snap a photo of some tunes, and want them to be recognized.
So, what could one do to achieve a good solution?
I was thinking about treating the musical notes just like written letters. The computer can easily learn written characters with the Arabic symbols.
I wonder, though, how easy would it be for a non-Arabic? For example, in Chinese or Japanese, several characters combine into one.
The same applies to musical notes, they can be connected and form something slightly different through that. For example,

or:

in contrast to just simple notes like:

What would be a good approach to recognize musical notes even for slightly low-resolution images or bit blurry deformed images?
I'm not saying to read out a symphony out of a thumbnail. But less than optimal captures.

Its possible if you can cut the images to chunks of musical notes. as they are evenly spaced and within 5 lines boundary. problem to tackle here is detecting rotation. would be first step to doing so. and pass through identifier to guess what note is it will be a better way of going forward.

any subjective ideas or comments are more than welcome

Not a complete answer, but some ideas:
Your goal is subdivided into many tasks. It's not exactly the same as OCR because you also need to find the vertical alignment for each note.

One should be able to read musical notes with lower quality images.

If you want your model to perform on low quality image, you'll need such database.
But instead of labeling and taking pictures of printed sheets, you could just generate the images and virtually apply all sorts of distortion on them.

refer to MNIST on kagle
to train model to recognize given set of notes in your case.

if you start training model will understand pattern depending on bits in your image.
issue comes with blurry images when last 2 sets of pics with minute change. will be difficult task for computer to distinguish.
there is one sure shot method which works almost every time but is resource intensive and needs plethora of data to train. is DNN.
this is my take on your question.


text:
Comparison and understanding of different version of DDQN?
There are several version of DDQN floating around. Sutton gives one that is a simple symmetric random update of the two Q functions. I think other papers (Silver paper for example) use a kind of delayed and split update rule.
Is there anything systematic describing the properties of the bias corrections and their respective advantages?

Double Q-learning was proposed by van Hasselt not Sutton neither Silver, there is an original Double Q-learning paper by van Hasselt that presents the algorithm for tabular case and there is also adaptation of that algorithm for function approximation case


text:
Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?
Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained on different samples. In double DQN, I understand that target Q and current Q are relatively independent because their parameters are quite different.
But in TD3, Q1 and Q2 are trained on exactly the same target. If their parameters are initialized the same, there will be no difference in their output and the algorithm will be equal to DQN. The only source of independence/difference of Q2 to Q1 I can tell is the randomness in the initialization of their parameters. But with training on the same target, I thought this independence will become smaller and smaller as they converge to the same target values. So I don't quite understand why TD3 works in combating overestimation in Q-learning.

I emailed the author of the paper and he replied that randomness in the parameter initialization is the only difference between Q1 and Q2. This difference is enough in practice. Moreover, TD3 method is more concerned with overestimation induced by function approximation error rather than stochasticity in the environment.

I highly recommend, if you haven't done so already, to take a look at the author's implementation of TD3, https://github.com/sfujim/TD3. It's extremely clean and simple, so it's very easy to read. I know two of the authors personally and they've told me they can code up the algorithm and train it on HalfCheetah or something during a presentation, so like really quickly.


text:
How do I improve accuracy and know when to stop training?
I am training a modified VGG-16 to classify crowd density (empty, low, moderate, high). 2 dropout layers were added at the end on the network each one after one of the last 2 FC layers.
network settings:

training data contain 4381 images categorized under 4 categories (empty, low, moderate, high), 20% of the training data is set for validation. test data has 2589 images.
training is done for 50 epochs.(training validation accuracy drops after
50 epochs)
lr=0.001, decay=0.0005, momentum=0.9
loss= categorical_crossentropy
augmentation for (training, validation and testing data): rescale=1./255, brightness_range=(0.2,0.9), horizontal_flip

With the above-stated settings, I get the following results:

training evaluation loss: 0.59, accuracy: 0.77
testing accuracy 77.5 (correct predictions 2007 out of 2589)

Regarding this, I have two concerns:

Is there anything else I could do to improve accuracy for both training and testing?
How can I know if this is the best accuracy I can get?

Is there anything else I could do to improve accuracy for both training and testing?

Yes, of course, there are a lot of methods if you want to try to improve your accuracy, some that I can mention:

Try to use a more complex model: ResNet, DenseNet, etc.
Try to use other optimizers: Adam, Adadelta, etc.
Tune your hyperparameters (e.g. change your learning rate, momentum, rescale factor, convolution size, number of feature maps, epochs, neurons, FC layers)
Try to analyze your data, with ~75% and 4 categories, is that possible there is one category that difficult to classified?

In essence, you have to do a lot of experiments with your model until you think "it is enough" (if you have a deadline). If you don't have a hard deadline, you can keep improving updating your ML model.

How can I know if this is the best accuracy I can get?

No, you can't until you compare it with other models/hyperparameters. If you do more experiments (some ways like the one I mentioned above) or compare with other people's experiments that using the same data, you'll find which one is the best. For an academic paper, for example, you need to compare at least 3 to 4 models that similar or experiment with hundreds of different hyperparameters combination.

thank you. currently i will be trying Resnet, also analyzing the missclassified data is a good point that i missed.  regarding the optimizers, i thought they were related to the type of the network task, like for classification task SGD is often used.

One option is not mentioned by malioboro is getting more data. Getting bigger dataset is almost always improve training results. If it's too hard to obtain more labeled data you can use data augmentation on existing data - small random transformations while keeping the same label.
For images most common methods of augmentations are (applying padding if needed):

small random zooming in/out
small random shifts of image
adding random noise to image
small random change in brightness, contrast, color balance and similar parameters

There are more complex methods, but they are dependent on specific of dataset/goal of training
Stopping:

you should stop training if you see training error is not decreasing - your have reached (possibly local) minima.
you should stop if testing or validation error start to increase - your method is overfitting (data augmentation can help in that case)


text:
How can I make meaningful English sentences from given set of words?
I have a set of topics and each topic consists of a set of words. I want to make meaningful English sentences from these words. Each topic consist of 5 to 10 words and these words are relevant to each other, like {code, language, test, write and function} and {class, public, method, string, int} are two sets. I want to generate a sentence from these set of words using API.

Could you clarify what you are aiming at for "meaningful"? With latest ML techniques on their own you can aim for grammatically correct and some rough semantic correctness (e.g. people nouns taking active roles, objects often their typical properties). However, the results are often nonsense - entertaining perhaps, but devoid of any real meaning and more like abstract poetry. Would that sort of thing meet your goals? Perhaps some more explanation of what you are looking for and some examples would help if you added them to the question

What would be the source of"meaning"? The words of each topic do not suggest any specific meaning. If you want to construct formally correct sentences, you'd need to build a model of the syntactic and semantic roles of the words, and create sentences in which the words take on the given roles. The choice of programming language is irrelevant, and it pretty unclear what you expect from an API here...

a simpler method (but likely not as powerful as @NeilSlater 's solution) might simply construct random sentences based off a list of syntax trees, but admittedly i don't know enough linguistics to evaluate this claim; also iirc there is a lot of difficulty of even evaluating the part of speech of words for many sentences

Sounds a lot like something I wrote in high-school in the late 1960s, a Fortran program with valid English sentence syntaxes and a sets of words classified by grammatical type.  It produced "meaningful" sentences like: "their man to their good warmth expands the crowds on its any hands", "my faces smell a lot of me", "every baby of you grows very orange", "his girl kisses into the blue women", "the girls burp her dress again again more", "the face of few sadistic desires chews her boys down a softly quick crowd", "the girls smell the desire also", and "my baby laughs sadly".

Two Approaches:

Naive Bayes
LSTM

Train Naive Bayes on a whole dataset learning the probability of the next word given a word.
You can even go with any LSTM approaches, but I'd bet on Naive Bayes.
Eg:
text: hello how are you hello how are you hello No how
to get the suggestion of next word depending on current word - hello
p(how | hello) = 3/4
p(No | hello) = 1/4
take argmax of probabilities.
Also remember to smooth, and train on huge dataset. Training is just finding the probabilities before hand.
Hope it helps ;)


text:
Should we multiply the target of actor by the importance sampling ratio when prioritized replay is applied to DDPG?
According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is defined
$$
w_i=\left({1\over N}{1\over P(i)}\right)^\beta
$$
in which $1/N$ is the probability of drawing a sample uniformly from the buffer, and $P(i)$ is the probability of drawing a sample from PER.
I'm wondering if we have to do the same to the target of the actor when we apply PER to DDPG. That is, multiplying $-Q(s_i, \mu(s_i))$ by $w_i$, where $\mu$ is the outcome of the actor.
In my opinion, it is necessary. And I've done some experiments in the gym environment BipedalWalker-v2. The results, however, is quite confusing: I constantly get better performance when I do not apply importance sampling to the actor. Why would this be the case?


text:
How to apply or extend the $Q(\lambda)$ algorithm to semi-MDPs?
I want to model an SMDP such that time is discretized and the transition time between the two states follows an exponential distribution and there would be no reward between the transition.
Can I know what are the differences between $Q(\lambda)$ and Q-learning for this problem (SMDP)? I actually want to extend the pseudo-code presented here to an SMDP problem with discretization of time horizon.

If you really just want an SMDP-version of the algorithm, which only needs to be capable of operating on the "high-level" time scale of macro-actions, you can relatively safely take the original pseudocode of whatever MDP-based algorithm you like, replace every occurrence of "action" with "macro-action", and you're pretty much done.
The only caveat I can think of in the case of $Q(\lambda)$ is that the "optimal" value for $\lambda$ is probably somewhat related to the amount of time that expires... so intuitively I'd expect it to be best if the value for $\lambda$ decreases as the amount of time expired during execution of the last macro-action increases. A constant $\lambda$ probably still works fine as well though.

If you actually want your algorithm to also be aware of lower-time-scale MDP underlying an SMDP, and not only treat macro-actions as "large actions" and be done with it... I'd recommend looking into the Options framework. There you get interesting ideas like intra-option updates, which may allow you to also perform learning whilst larger macro-actions (or options) are still in progress.
Last time I looked there hasn't been a lot of work involving the combination of eligibility traces and options, but there has been some: Eligibility Traces for Options. This paper doesn't specifically apply the algorithm you mentioned ($Q(\lambda)$), but it does discuss a bunch of other -- much more recent, and likely better -- off-policy algorithms with eligibility traces.

Dear @Dennis. Can I know if the only differences will be the following: 1) Take action $a$, observe $r, s^{\prime}, \tau$. 2) Updating the error: $\delta \leftarrow r+\gamma^{\tau}Q(s^{\prime}, a^{\ast})-Q(s,a)$. 3) Updating eligibility trace $e(s,a) \leftarrow \lambda \gamma^{\tau} e(s,a)$

@AminSh Yes that looks correct to me, assuming that $\tau$ is the duration of the last "action" $a$ you've taken (where $a$ may be a "macro-action" that takes more than a single primitive time step). I assume that the $r$ you observe there is also a sum of rewards collected during execution of $a$, with appropriate discounting by $\gamma$ "inside" that sum as well, rather than $r$ being just a single primitive reward.

Dear @Dennis I really appreciate you.


text:
What should the range of the output layer be when performing classification?
I am working on a MLP neural networks, using supervised learning (2 classes and multi-class classification problems). For the hidden layers, I am using $\tanh$ (which produces an output in the range $[-1, 1]$) and for the output layer a softmax (which gives the probability distribution between $0$ and $1$). As I am working with supervised learning, should be my targets output between 0 and 1, or $-1$ and $1$ (because of the $\tanh$ function), or it does not matter?
The loss function is quadratic (MSE).

For this particular classification problem, I would recommend you using a softmax function whose output range is [0,1].
The sum of all outputs should be 1, so an advantage of using a softmax function is that you get a percentage of how confident the network is in this classification.
Side note: As DuttaA has commented, cross entropy loss is a better loss function than the quadratic mean squared error.

your targets should be in the same range as your output functions other wise your loss function wont be accurate, with supervised learning your trying to reduce the loss of your output against your targets so in this case your targets should be the true/optimal probability distribution for that set of input data. Im from the midwest so obligatory "cant compare apples to oranges" here ;)


text:
Is there a way to run other platforms (other than Atari) in an OpenAI's Gym-like environment?
Is there a way to run C64, Nintendo Gameboy, or other platforms in an OpenAI's Gym-like environment? It seems that we can run only Atari games.

Ok, I found
https://blog.openai.com/gym-retro/
but there may be other platforms?

https://github.com/openai/retro
Current list of machines is

Atari
Atari2600 (via Stella)
TurboGrafx-16/PC Engine (via Mednafen/Beetle PCE Fast)
Game Boy/Game Boy Color (via gambatte)
Game Boy Advance (via mGBA)
Nintendo Entertainment System (via FCEUmm)
Super Nintendo Entertainment System (via Snes9x)
GameGear (via Genesis Plus GX)
Genesis/Mega Drive (via Genesis Plus GX)
Master System (via Genesis Plus GX)

There is a vague tutorial here for adding other systems
https://github.com/openai/retro/issues/169


text:
Is Intelligence a naturally occurring function of Information Technology?
Here intelligence is defined as any analytic or decision making process, regardless of strength (utility), and, potentially, any process of computation that produces output, regardless of the medium.
The idea that AI is part of an evolutionary process, with humans as merely a vehicle of the next dominant species, has been a staple of Science Fiction for many decades.  It informs our most persistent mythologies related to AI.  (Recent examples include Terminator/Skynet, Westworld, Ex Machina, and Alien:Covenant).
What I'm driving at here is that, although the concept of Neural Networks have been around since the 1940's, they have only recently demonstrated strong utility, so it's not an unreasonable assumption to identify Moore's Law as the limiting factor.  (i.e. it is only recently that we have had sufficient processing power and memory to achieve this utility.)
But the idea of AI is ingrained into information technology once automation becomes possible.  Babbage's Difference Engine led to the idea of an Analytic Engine, and the game of Tic-Tac-Toe was proposed as a means of demonstrating intelligence.
What I'm driving at here is that the idea of analysis and decision making are so fundamental in regard to information technology, that it is difficult to see functions that don't involve them.  And, if the strength of analysis and decision making is largely a function of computing power:
Can intelligence be understood as a naturally occurring function of information technology?

Maybe, but it depends to a very large degree on the choice of definition.
One of the biggest challenges for AI researchers, neuroscientists, philosophers, and psychologists, has been that the layperson's understanding of intelligence does not appear to correspond to a well-defined concept. This point was most famously exploited by John R. Searle in his paper Minds Brains and Programs.
Consider your definition again carefully, and notice that decision is unbound. Does a rock rolling down a hill make a decision when it 'decides' to roll to the left? If not, Searle and his ilk would argue that a computer isn't making a decision when it decides to execute the next instruction. It is simply obeying deterministic laws of physics.
To escape this problem and still claim that machines are intelligent, you can either chose to believe that rocks make decisions (a view called panpsychism), or that we should behave as though rocks make decisions because doing so gives us a lot of explanatory power (the intentional stance), or claim that humans are no different from the rock or the computer (eliminative materialism). The last option leaves you with the problem of explaining subjective experiences though.
An excellent book that covers the mainstream arguments in this vein is Haugeland's Mind Design II. All the arguments I've outlined here are covered in great detail within. Among modern AI researchers, I'd say there is a split between those who embrace something like the intentional stance and those who embrace eliminative materialism.

The rock rolling down the hill doesn't qualify, nor water clocks (which can be viewed as algorithms.)  Here I'm going by what I've inferred to be the Russell & Norvig definition, where intelligence is a relative measure of problem solving strength in relation to a problem or set of problems.  This definition is entirely based in utility, which seems to be the only meaningful metric.

@DukeZhou Well, again, that definition is highly contentious. A rock can solve a problem by rolling down a hill, if we properly interpret its movements, and arrange the hill so that its bounces correspond to some form of computation. If you don't accept that the rock is thus 'intelligent' and solving a problem, then the issue becomes why you would be willing to say that computer (where electrical charges bounce around in a carefully arranged way to 'solve' a problem) can be.

This is a very good point.  The rock rolling down the hill is a form of computation.  The rock takes input and responds based on the rules of the system, in this case the laws of physics.  The rock has intelligence in this one problem, and the complete computation of rolling down the hill has a greater degree of intelligence.  If this method is infinitely expanded, the process could result in much stronger intelligence.  I guess the key distinction here is that this is more of a puzzle than a game being "solved" to whatever degree.

I guess the reason I focus on utility is that process can be seen as corollary--we can only measure intelligence by the result of any given process. Thus, one can take any of the listed views as to the nature of intelligence, but, in the final analysis, "what is the outcome" is the only truly relevant question. In other words, reducing it to the point of suggesting these arguments are interesting but immaterial.  The increasing utility of AI, despite not fully understanding the process of our own human intelligence, would seem to affirm this view


text:
What is the difference between a problem representation and problem modelling?
As far as I know, a problem representation is the formulation of the problem in a way that it can be programmed and therefore solved (for example, you can represent the $N$-queens problem by using an array of $N \times N$).
What does problem modelling mean? What is the difference between a problem representation and problem modelling?

I would say these terms are often used interchangeably in AI. When they differ, I would say that problem modeling means finding a mathematical description of the problem, while problem representation means finding a particular way to represent that mathematical formalism.
For example, a list of numbers can be stored (represented) with a linked list, and array list, a hash table, or a self-balancing tree. All of them can produce a faithful model of the list, but if what you want to do is find the order that elements were entered in, the array list or linked list is far faster and more natural. If what you want to do is determine whether certain pieces of information are present in the list, the hash table is fastest. If what you want to do is find ranges of similar elements, the tree is fastest. Essentially, representational choices are engineering problems, while modelling choices are scientific problems.

What do you mean by "mathematical description". If I formulate the problem as a CSP, I have to use mathematical operations to express the constraints. How else could I formulate the problem in a mathematical way? @JohnDoucette

@JayCritch To model a CSP, you write down the constraints. To represent a CSP, you might use a list of strings that state logical conditions; a function that checks consistency of some representation of an assignment by using loops over known indices; or even a compact representation in terms of binary digits and a sequence of bitwise operators. These different instantiations correspond to different representational choices. You can actually write all of these down mathematically, but usually this isn't worthwhile. Where it pays off is when writing down the program that solves the CSP.


text:
How do I choose the activation function of the output layer of a neural network (based on theoretical motivations)?
Is there a method (not a table of recommendations!) that could tell me what activation function to choose if the outputs of the neural network have some interpretation? For example, these can be the mean of some normal distribution, probabilities in multinomial distribution, parameters of the exponential distribution, and so on.

This is an old question, but I think you should clarify what you really mean by "interpretation". Do you mean that the outputs are just "the mean of some normal distribution, probabilities in multinomial distribution, parameters of the exponential distribution", i.e. related to distributions, or maybe something else?

Yes, please, clarify. NNs are just powerful aproximators. You can interpret the network outputs as you want (and treat them accordingly in the loss_fn). The act_fn are more an architectural design question rather than a information coding/interpretation question.


text:
What is the best machine learning algorithm to select best 3 variable combinations?
I have 10 variables as like below

$V_1=1$, $V_2=2$, $V_3=3$, $V_4=4$, $V_5=5$, $V_6=6$, $V_7=7$, $V_8=8$, $V_9=9$ and $V_{10}=10$

Note: Each variable can have any value
Now I want to select the best 3 variables combination as like below

$V_1V_3V_4$ or $V_{10}V_1V_7$ or $V_5V_3V_9$ etc.

The best combination is nothing but the sum of the values of 3 variables in the combination.
Example:

Combination 1($V_1V_2V_3$) : 1+2+3 $\Rightarrow$ 6
Combination 2($V_8V_9V_{10}$) : 8+9+10 $\Rightarrow$ 27

In the above example Combination 2($V_8V_9V_{10}$) has the highest sum value. So the Combination 2($V_8V_9V_{10}$) is the best combination here.
Like this, if I have a large number of variables means which machine learning algorithm selects the best combination in all the sense.
Suggest me the best machine learning algorithm for selecting the best variable combinations. Thanks in advance.

If you exactly know the rule to pick these 3, why using a machine learning approach? You could just find the 3 biggest values. Even if the rule was other one, if you know how to score these, you could just make all the combinations posible (which are 120), and find out the best one

What you have is an instance of a local search or black-box optimization problem.
There are many AI algorithms to do this, though as commenters have noted, with only 10 binary variables, you're happier to just look at all $2^{10}$ of them.
If you had, say, 60, then looking at them all is no longer a good idea ($2^{60}$ is very large).
The simplest BBO technique is called hill climbing. It looks like this:

Pick a random subset of the variables to be true.
For each variable, determine the effect of flipping that variable from true to false in the current assignment.
Select the 'flip' that leads to the largest value of your function.
If the 'flip' has a larger value of your function than the current assignment, GOTO 2.
Otherwise, return the current assignment.

This amounts to: 'Pick an assignment at random, then keep making small changes that improve it until you can't anymore. Claim that this is the best solution.'
Clearly, this algorithm will not recover the globally best solution with certainty, but you could run it many times, and eventually, it will find it with high probability.
A complete algorithm, that will always uncover the global optimum, is simulated annealing. This algorithm has a temperature parameter, usually denoted T, which changes according to a temperature schedule. If you start T large 'enough', and decrease it slowly 'enough', then it will find the global optimum. Unfortunately, 'enough' depends a lot on the shape of the function you are optimizing, which you generally cannot observe. There are some common guesses for T, like multiplying it by, e.g. 0.9999 at each step in the search, that seem to work okay in practice.
Both of these algorithms are widely available in packages.


text:
What is a graph neural network?
What is a graph neural network (GNN)?
Here are some sub-questions

How is a GNN different from a NN?
How exactly is a GNN related to graphs?
What are the components of a GNN? What are the inputs and outputs of GNNs?
How can GNNs be trained? Can we also use gradient descent with back-propagation to train GNNs?

If someone is interested in answering to this question, have a look at this article A Gentle Introduction to Graph Neural Networks (Basics, DeepWalk, and GraphSage) and the research article A New Model for Learning in Graph Domains (which the paper that introduced GNN).

Graph Neural Networks
The term Graph Neural Network, in its broadest sense, refers to any Neural Network designed to take graph structured data as its input:

To cover a broader range of methods, this survey considers GNNs as all deep learning approaches for graph data.

A Comprehensive Survey on Graph Neural Networks, Wu et al (2019)

However the original paper to propose the term specifically referred to recursive neural networks adapted to take graph-structured data as their input:

This paper presents a new neural model, called graph neural network (GNN), capable of directly processing graphs. GNNs extends recursive neural networks and can be applied on most of the practically useful kinds of graphs, including directed, undirected, labelled and cyclic graphs.

A new model for learning in graph domains (2005)

Subtypes
Note, Wu et al propose a taxonomy dividing GNN's into four subgroups:

Recurrent graph neural networks (RecGNN)
Convolutional graph neural networks (ConvGNN)
Graph autoencoders (GAE)
Spatial-temporal graph neural networks (STGNN)

ConvGNN's can themselves be classified by whether they use Spectral methods or Spatial methods, and GAE's by whether they are designed for Network embedding or Graph generation.

Of course, after all this time, I am already familiar with GNNs. However, I think that one of the unclearest things when it comes to GNNs is how they are trained. Are they still trained with GD? What are the labels? What are the loss functions? You could talk a little bit about that (then I would accept your answer).


text:
What do the vectors of the center and outside word look like in word2vec?
In word2vec, the task is to learn to predict which words are most likely to be near each other in some long corpus of text. For each word $c$ in the corpus, the model outputs the probability distribution $P(O=o|C=c)$ of how likely each other word $o$ in the vocabulary is to be within a certain number of words away from $c$. We call $c$ the "center word" and $o$ the "outside word".
We choose the softmax distribution as the output of our model: $$P(O=o|C=c) = \frac{\exp(\textbf{u}_{0}^{T} \textbf{v}_{c})}{\sum_{w \in \text{Vocab}} \exp(\textbf{u}_{w}^{T} \textbf{v}_c)}$$
where $\textbf{u}_0$ and $\textbf{v}_c$ are vectors that represent the outside and center words respectively.

Question. What do the vectors  $\textbf{u}_0$ and $\textbf{v}_c$ look like? Are they just one-hot-encodings? Do we need to learn them
  too? Why is this useful?

No, the word vectors are not one-hot encodings. Yes, they are learned.
The purpose of the word2vec model is actually to learn dense, semantically meaningful encodings for words. That is, if your words are $d$-dimensional vectors, then each word's position in this vector space says something about what that word means. This is because word2vec learns to represent words in similar ways if they are frequently close together in your corpus. It implements the idea of distributional similarity.
The task of predicting an "outside word" given a "center word" accomplishes all of this in an indirect way.
A naive objective function to maximize for word2vec is
$$J = \prod_{t=1}^L \prod_{-m \leq j \leq m\\ \quad j\neq 0} p(\textbf{u}_{t+j}|\textbf{v}_t)$$
where $L$ is the length of your corpus, $m$ is the "radius" from each center word you want to consider, $\textbf{u}_{t+j}$ is an outside word, and $\textbf{v}_t$ is a center word.
If we let $p(\textbf{u}_{t+j}|\textbf{v}_t)$ be the softmax distribution, then maximizing $J$ means maximizing the inner product $\textbf{u}_{t+j}^T\textbf{v}_t$ in the softmax's numerator. Maximizing that inner product means making center words as close as possible to their neighboring words, giving you some semantically meaningful word vectors to use in your downstream NLP tasks.
This lecture from Stanford's CS224N goes into more detail.

They're asking if training the model involves learning the "word vectors" u and v. These vectors are the parameters of the model. We don't do inference from word2vec, but the resulting word vectors are used to encode input text for some other NLP tasks. I'll edit the question to add more context.

Word2vec is like an autoencoder in that it learns useful encodings of inputs in an unsupervised way. It is not, however, a neural network. I also meant that we are not interested, ultimately, in the final output distribution of word2vec. We do not infer outputs of the model (distributions over likely nearby words) on new data. As in autoencoders, we may infer from a model that makes use of the learned encodings, but we do not typically infer from the model itself. Maybe I'm thinking of inference the wrong way, though. This is roughly my idea of inference.


text:
What would be a good comprehensive source about the different forms of classical learning in mammals?
I am looking for a source that really discusses the classic rules of learning in depth. So classical conditioning, operant conditioning, imitation learning... I have found an infinite number of books that supposedly discuss these topics, but have not yet found anything that summarizes all the important findings on this topic. I am familiar with the basics, but I am explicitly interested in a detailed presentation of the topic.
Can anyone tell me a good source about the different forms of classical learning in mammals? I consider "Reinforcement Learning: An Introduction" comprehensive and detailed regarding RL. A comparable book on biological learning systems would be great. Sources in German and English would fit.

I recommend Animal Learning and Cognition: An Introduction by John Pearce.

As an introduction to the field, it's comprehensive. It covers all the topics you mentioned and more. Here is a free preview of the book.
It's well-regarded, having over 500 citations on Google Scholar.
It's relatively modern. Published in 2008, it discusses experimental findings that you won't be able to find in some of the other classics published in the 70s or 80s.

Although it does not specifically focus on mammals, all of the ideas extend to mammals.

I wouldn't call it casual, but it does have less math than Sutton and Barto's RL book. Keep in mind that we have no complete mathematical model of cognition. This book, along with the rest of the field, is less algorithmic than RL. Instead, it focuses more on experiment and observation.


text:
Are there communities dealing with costs-vs-accuracy tradeoffs in Machine Learning?
My group is working on a ML model that can work with little data (and bad accuracy) as long as there actually is little data available but can easily be extended as soon as said data becomes available (Think of interpolating existing models and then creating an individual predictor). This is due to a business requirement of the relevant application (new plants get installed over time but need to be integrated seamlessly). Transforming from a coarse yet cheap prediction to an accurate but expensive prediction takes labor effort that the company can invest as desired.
Are there research areas that take into account such "evolutionary transformation processes" which enable a tradeoff between costs and accuracy? What are the right keywords to look for?
I am looking for keywords/papers/communities.

Welcome to AI.SE ks.and1.
What you're describing could be related to several areas, but since you are looking only for some keywords, I'll suggest some:

You might be interested in anytime learning, which might better be called 'learning that can stop at any time'. A good example of this approach in machine learning is anytime learning for decision trees, as summarized in Esmeir & Markovitch's 2007 JMLR paper
Apart from this, it sounds like you are just describing the regular process of retraining a model when you get more data. For most algorithms, this just amounts to changing the input to your program so that it points at a larger file, containing more records than last time, and maybe buying a faster GPU or CPU to train the model with if it's taking too long with the extra data.

Hopefully, this can help you get started. If you have some more details about your problem, I might be able to suggest some more specific resources.


text:
How many parameter would there be in a logistic regression model used to classify reviews into "good" or "bad"?
Suppose we want to classify a review as good ($1$) or bad ($0$). We have a training data set of $10,000$ reviews. Also, suppose we have a vocabulary of $100,000$ words $w_1, \dots, w_{100,000}$. So the data is a matrix of dimension $100,000 \times 10,000$. Let's represent each of the words in the reviews using a bag-of-words approach over tf-idf values. Also, we normalize the rows such that they sum to $1$.
In a logistic regression approach, would we have $10,000$ different logistic regression models as follows:
$$ \log \left(\frac{p}{1-p} \right)_{1} = \beta_{0_{1}} + \beta_{1_{1}}w_{11} + \dots + \beta_{100,000_{1}}w_{100,000} \\ \vdots \\ \log \left(\frac{p}{1-p} \right)_{10,000} = \beta_{0_{10,000}} + \beta_{1_{10,000}}w_{11} + \dots + \beta_{100,000_{10,000}}w_{100,000}$$
So are we estimating $100,000 \times 10,000$ coefficients?

Nope! Our number of coefficients will be driven by the vocabulary, and we'll use each of those 10K samples to estimate values for those coefficients - so, 'just'  100K samples.  However, word frequency in human languages follows a Zipf distribution => most of those words will be rare, seen in only a few samples (=> won't even be able to determine whether they are useful or not, let alone get good values for coefficients).  For an application like this one, you would probably find most of the value from a few hundred words.


text:
How create an AI that continuously adapts to different users?
I want to make an AI with deep learning which can adapt itself from user to user.
Let's say we have food combiner AI which suggests a food to eat with another food as you give as input. This is the most personalized case I found to ask here. For example the AI suggested a food for me. However, the food AI suggested for me might not be good choice for another person. So another person will let the AI know like "I don't like that food to eat with this. Etc. When the user let the AI know that, It should affect AI's further combination food suggestions.
How can I build that AI? Where should I start from? which area or topics should I research?

Maybe have a look at active learning and online machine learning, but maybe this isn't exactly what you are looking for. Maybe you want some kind of AI which is aware of different users. For example, have a look at recommender systems.

This link might help.

Assuming you have enough training data and representational capacity, you can give each user a unique identifier and concatenate that with the other inputs to the neural net. As users give more feedback, the network will learn in the usual way since each situation (e.g. food-user pair) is represented by a unique input.
I'd consider this a general, brute-force approach, and it may not be suited to your application. Others' links to recommendation systems might be more useful. It depends on what your task is and what your constraints are.

Firstly thx for your reply.I think it would be really huge cost that I can't afford to concatenate each users information with other inputs and also it wouldnt work well for a new user, because those who affected the deep learning model before,would also affect the new user's combinations. So It might be produce combinations weirdly.If I had real statistics of what the user liked or did not and encode this info to an array with an algorithm,your method would be really useful.However, I have only images in my real situation(not food comb).So I can't encode how the user likes or not ina good way

I have a new idea. what if I do the AI thing in users' phones? So I can ask them a few pictures (combinations) if they liked or not. Then I take those and give to the Deep learning model. So each users can have different model. However, there is a big concern that I ve been thinking that if I do that, I wont be able to update the model via remotely. For example, I improved the model by training a lot data, but users have their own models and they are old models. So I want to change my new model with their old models. I just cant do that.

What I can do is training each model in each user's phone with "my new big data" and this would be really bad idea. So I got stuck


text:
Why is $M_t$ (the emphasis) helping in correcting for the state distribution in the Emphatic TD algorithm?
The book by Sutton and Barto discusses in section 11.8 that the convergence of off-policy TD function approximation can be improved by correcting for the distribution of states encountered. The section seems to be written in haste and doesn't do a good job in explaining why will $M_t$, the emphasis, help in getting a state distribution closer to the target policy.
My understanding of on-policy distribution is not clear at the moment. I think it is the distribution of states encountered under the target policy (the policy for which we want to state-action/state values).
The importance sampling ratio corrects for update distribution (by multiplying the correction term with the ratio), but how is $M_t$ helping in correcting for the state distribution?

I don't think the section was written in haste. I think they just didn't have space to include the whole proof. It's a bit involved, so they just gave concepts.
An Emphatic Approach to the Problem
of Off-policy Temporal-Difference Learning gives a proof of stability.
At least parts of it should seem familiar if you've read Sutton and Barto's proof of the convergence of linear TD(0) on page 206 of their 2nd edition RL book.
On Convergence of Emphatic Temporal-Difference Learning gives a proof of convergence.
I confess that I don't understand these papers well enough to give a summary. If you eventually do, I would greatly appreciate an update.

Hi, after skimming through all cites of these two papers. I found it seems that there is no paper on how to combine Deep Networks with Emphatic TD. Is it possible to use it in DQN, TRPO or PPO?


text:
Why does the BERT encoder have an intermediate layer between the attention and neural network layers with a bigger output?
I am reading the BERT paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
As I look at the attention mechanism, I don't understand why in the BERT encoder we have an intermediate layer between the attention and neural network layers with a bigger output ($4*H$, where $H$ is the hidden size).
Perhaps it is the layer normalization, but, by looking at the code, I'm not certain.

I am thinking about the same. did you find out why is it there

The paper Undivided Attention: Are Intermediate Layers Necessary for BERT? should answer it.
In the abstract, they write

All BERT-based architectures have a self-attention block followed by a block of intermediate layers as the basic building component. However, a strong justification for the inclusion of these intermediate layers remains missing in the literature.

In the conclusion, they write

In this work we proposed a modification to the BERT architecture focusing on reducing the number of intermediate layers in the network. With the modified BERTBASE network we show that the network complexity can be significantly decreased while preserving accuracy on fine-tuning tasks.

So, why are "these intermediate layers" used? You're quoting the abstract that says that an explanation in the literature is missing, but do the authors of this paper provide it?

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.


text:
Why would adding all the possible embeddings be "worse" than using 1D-convolutions?
Suppose we are using word2vec and have embeddings of individual words $w_1, \dots, w_{10}$. Let's say we wanted to analyze $2$ grams or $3$ grams.
Why would adding all the possible embeddings, $\binom{10}{2}$ or $\binom{10}{3}$, be "worse" than using 1D-convolutions?

N-grams are defined as sets of n contiguous words. We use n-grams because they are more useful than random combinations of words across the sentence. Intuitively, combinations of nearby words have more semantic meaning than combinations of distant words.
Also, using all possible combinations of n embeddings would take much longer, especially since (1D) convolutions are such efficient operations.


text:
Which fields of AI are actively researching consciousness?
Max Tegmark discusses the topic of consciousness in his book Life 3.0 and comes to the conclusion, that consciousness is substrate independent. If his analysis is correct, it should be possible to create artificial consciousness. The integrated information theory (IIT), while currently only just a theory, also points in this direction.
This leads me to the question: which fields of AI research, if any, are currently actively engaged in this domain?
So far, I've only found research concerning consciousness in neuroscience and discussions of experts in philosophy.
Are there any projects publicly known concerning artificial consciousness or organizations that are active in this regard?

Yes, the RefPerSys project is related to your question. Pitrat's book Artificial Beings, the conscience of a conscious machine is explaining in details a possible approach, and is about an experimental software.

I think we still have a long way to go before any progress is made on artificial consciousness. However, researchers are taking inspiration from traits of human consciousness. One relevant paper is Machine Theory of Mind by DeepMind. They show that their model can (at least to some extent) represent the desires, beliefs, and intentions of agents that it observes. It even passes a form of the Sally-Anne test, showing that it can represent the false beliefs of an agent.