text:
Monte Carlo Exploring Starts broke for 2048 game AI
I implemented a MCES for 2048 (the game), with a quality function implemented as a neural net of a single layer.
The starts are created with 6 cells filled with values between 64 and 1024, two cells are 1024 an ther other 8 cells are filled with 0. The game is then progressed until the AI loses or wins and another start is created.
After 10 wins the max cell created in the start is reduced in half. Thus, after the first 10 wins, the max cell created in the start is 512.
The issue I am having is that after the first 10 wins, the AI gets stuck, it can run around 3 million steps but doesn't get any more wins.
How should I create the starts for it to actually learn?
Code for reward (complete code here):
        ArrayList<DataSet> dataSets = new ArrayList<>();
        double gain = 0;

        for(int i = rewards.size()-1; i >= 0; i--) {
            gain = gamma * gain + rewards.get(i);

            double lerpGain = reward(gain);
            INDArray correctOut = output.get(i).putScalar(actions.get(i).ordinal(), lerpGain);
            dataSets.add(new DataSet(input.get(i), correctOut));
        }

        Qnetwork.fit(DataSet.merge(dataSets));

Code:
public class SimpleAgent {
    private static final Random random = new Random(SEED);

    private static final MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
            .seed(SEED)
            .weightInit(WeightInit.XAVIER)
            .updater(new AdaGrad(0.5))
            .activation(Activation.RELU)
            .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
            .weightDecay(0.0001)
            .list()
            .layer(new DenseLayer.Builder()
                    .nIn(16).nOut(4)
                    .build())
            .layer(new OutputLayer.Builder()
                    .nIn(4).nOut(4)
                    .lossFunction(LossFunctions.LossFunction.SQUARED_LOSS)
                    .build())
            .build();

    public SimpleAgent() {
        Qnetwork.init();
        ui();
    }

    private static final double gamma = 0.02;

    private final ArrayList<INDArray> input = new ArrayList<>();
    private final ArrayList<INDArray> output = new ArrayList<>();
    private final ArrayList<Double> rewards = new ArrayList<>();
    private final ArrayList<GameAction> actions = new ArrayList<>();

    private MultiLayerNetwork Qnetwork = new MultiLayerNetwork(conf);
    private GameEnvironment oldState;
    private GameEnvironment currentState;
    private INDArray oldQuality;
    private double epsilon = 1;

    public void setCurrentState(GameEnvironment currentState) {
        this.currentState = currentState;
    }

    public GameAction act() {
        if(oldState != null) {
            double reward = currentState.points - oldState.points;

            if (currentState.lost) {
                reward = 0;
            }

            input.add(oldState.boardState);
            output.add(oldQuality);
            rewards.add(reward);

            epsilon -= (1 - 0.01) / 1000000.;
        }

        oldState = currentState;
        oldQuality = Qnetwork.output(currentState.boardState);

        GameAction action;

        if(random.nextDouble() < 1-epsilon) {
            action = GameAction.values()[oldQuality.argMax(1).getInt()];
        } else {
            action = GameAction.values()[new Random().nextInt(GameAction.values().length)];
        }

        actions.add(action);

        return action;
    }

    private final int WINS_TO_NORMAL_GAME = 100;
    private int wonTimes = 0;

    public void setHasWon(boolean won) {
        if(won) {
            wonTimes++;
        }
    }

    public boolean playNormal() {
        return wonTimes > WINS_TO_NORMAL_GAME;
    }

    public boolean shouldRestart() {
        if (currentState.lost || input.size() == 20) {
            ArrayList<DataSet> dataSets = new ArrayList<>();
            double gain = 0;

            for(int i = rewards.size()-1; i >= 0; i--) {
                gain = gamma * gain + rewards.get(i);

                double lerpGain = reward(gain);
                INDArray correctOut = output.get(i).putScalar(actions.get(i).ordinal(), lerpGain);
                dataSets.add(new DataSet(input.get(i), correctOut));
            }

            Qnetwork.fit(DataSet.merge(dataSets));

            input.clear();
            output.clear();
            rewards.clear();
            actions.clear();

            return true;
        }

        return false;
    }

    public Game2048.Tile[] generateState() {
        double lerped = lerp(wonTimes, WINS_TO_NORMAL_GAME);
        int filledTiles = 8;

        List<Integer> values = new ArrayList<>(16);

        for (int i = 0; i < 16-filledTiles; i++) {
            values.add(0);
        }

        for (int i = 16-filledTiles; i < 14; i++) {
            values.add((int) (7-7*lerped) + random.nextInt((int) (2- 2*lerped)));
        }

        values.add((int) ceil(10-10*lerped));
        values.add((int) ceil(10-10*lerped));

        Collections.shuffle(values);

        return values
                .stream()
                .map((value) -> (value == 0? 0: 1 << value))
                .map(Game2048.Tile::new)
                .toArray(Game2048.Tile[]::new);
    }

    private static double reward(double x) {
        return x/ 2048;
    }

    private static double lerp(double x, int maxVal) {
        return x/maxVal;
    }

    private void ui() {
        UIServer uiServer = UIServer.getInstance();
        StatsStorage statsStorage = new InMemoryStatsStorage();
        uiServer.attach(statsStorage);
        Qnetwork.setListeners(new StatsListener(statsStorage));
    }
}

Could you add more details, like what policy iteration are you going to use, what are rewards, when an episode ends ?

The implementation is the standard Monte Carlo exploring starts, but with an NN to save the expected rewards.

Episode ends when the AI losses or wins.


text:
How to optimize neural network parameters with REINFORCE
I've seen a few mentions in papers that neural network parameters can be found using REINFORCE algorithm. It was mentioned in the context of nondifferentiable operations involving e.g. step function which appears in "hard attention" or weight pruning. Unfortunately, I haven't seen how to really do this. In Markov Decision Process we have, states (S), actions (A) and rewards (R) so what is what in case of neural net? I don't see how can we find parameters or neural net if the gradient is not well defined.
Any code sample or explanation?

The term REINFORCE actually corresponds to a method of estimating gradients, it is not particular to reinforcement learning. The paper you linked doesn't appear to deal with RL at all, so the issue they're describing is not one that you should expect to find in a policy gradient application.
If you're using REINFORCE to estimate policy gradients in RL (this is the common use case), your parameters are parameterizing a policy function. The inputs are states and the outputs are actions. The issue with estimating the gradient of the parameters is as follows. In RL the objective is to maximize expected rewards. This is an expectation with respect to the policy of the cumulative reward of a trajectory. If you take the gradient of this wrt the parameters, since the expectation is wrt the policy which depends on the parameters, you can't move the gradient inside the expectation so you can't estimate the gradient from samples. Using REINFORCE however, you use the "log-derivative" trick to rewrite the objective as an expected gradient, which can be estimated from samples.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Do we have to use the IOB format on labels in the NER dataset? If so, why?
Do we have to use the IOB format on labels in the NER dataset (such as B-PERSON, I-PERSON, etc.) instead of using the usual format (PERSON, ORGANIZATION, etc.)? If so, why? How will it affect the performance of the model?

There's nothing stopping you from training a model with whatever tags you want.
Using what you describe as "usual" format means you would have approx half as many tags as using the IOB format. In theory this means your model will develop higher accuracy faster and with less training data. On the downside, you will need to do more work when interpreting the results in order to be confident where one named entity ends and another one begins.
I made a notebook to do some empirical tests on this which uses 17 tags in the IOB format and 9 in the "usual" format.
TL;DR using the "usual" format did not produce a noticeable improvement in model quality.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How can a learning rate that is too large cause the output of the network (and the error) to go to infinity?
It happened to my neural network, when I use a learning rate of <0.2 everything works fine, but when I try something above 0.4 I start getting "nan" errors because the output of my network keeps increasing.
From what I understand, what happens is that if I choose a learning rate that is too large, I overshoot the local minimum. But still, I am getting somewhere, and from there I'm moving in the correct direction. At worst my output should be random, I don't understand what is the scenario that causes my output and error to approach infinity every time I run my NN with a learning rate that is too large (and it's not even that large)

How does the red line go to infinity ever? I kind of understand it could happen if we choose a crazy high learning rate, but if the NN works for 0.2 and doesn't for 0.4, I don't understand that

think of it in terms of weights. the weights would find a drastic change for example, from 1 to 1.001 for lr=0.001 or from 1 to 1.4 for lr=0.4. multiplied with any input, the latter would give a more drastic change/output (consider relu activated where inputs can be large). these accumulate over layers and increase the final output errors for next batch/sample after being updated by backprop and so, the backpropogated loss will also be high and the cycle continues until you get nan


text:
How can I change observation states' values in OpenAI gym's cartpole environment?
I am learning with the OpenAI gym's cart pole environment.
I want to make the observation states discrete (with small stepsize) and for that purpose, I need to change two of the observations from [$
-\infty, \infty$] to some finite upper and lower limits. (By the way, these states are velocity and pole velocity at the tip).
How can I change these limits in the actual gym's environment?
Any other suggestions are also welcome.

I don't recommend changing the rules of the environment.
What you could do:
Perform a method called bucketing i.e. take a value from a continuous state space see which discrete bucket it should go into and then let your agent use the bucket number as the observation.
e.g. Say I do have a continuous state space with one variable in range $[-\infty,\infty]$
The buckets can be as follows:
0). x < -1000
1). -1000 $\le$ x $<$ -500
2). -500 $\le$ x $<$ -100
3). -100 $\le$ x $<$ -50
4). -50 $\le$ x $<$ 0
5). 0 $\le$ x $<$ 50
6). 50 $\le$ x $<$ 100
7). 100 $\le$ x $<$ 500
8). 500 $\le$ x $<$ 1000
9). x > 1000
Therefore in this example scenario there are 9 buckets. Hence, the observations can be in range [0, 9] discretely.


text:
What is the purpose of a Neural Network in Reinforcement Learning when we have a Q-learning update rule?
I'm confused as to the purpose of training a neural network (NN) for reinforcement learning (RL) tasks such as Gridworld. In RL tasks, namely q-learning, we have a q-learning update rule, which is designed to take some state and action and compute the value of that state-action pair.
Performing this process several times will eventually produce a table of states and what action will likely lead to a high reward.
In RL examples, I've seen them train a neural network to output q-values and a loss function like MSE to compute the loss between the q-learning update rule q-value and the NN's q value.
So:
(a) Q-learning update rule-> outputs target Q-values
(b) NN -> outputs Q values
MSE to compute the loss between (a) and (b)
So, given we already know what the target Q-value is from a, why do we need to train a NN?

Can you please link us to the algorithm that you have seen where the target Q values are not produced by a NN but an NN is still used in the algorithm?

Hello, what I mean is the target is produced by a Q-learning update rule, and the NN is what is trained to mimic the behaviour of the Q-learning update rule. The example is from a book called "Deep RL in Action" and the example can be found on pages 54-68.

I don't think people generally do use neural nets for grid world. As long as the state and action spaces are small enough, you should be able to store Q values in a table like you suggested. Neural nets come in handy when the state space is very large (or even continuous), so you can't afford to store a table of Q values. Also, neural nets have the ability to generalize across "similar" states -- for instance, if two states are very similar the neural net would likely produce similar values for those states, whereas a tabular implementation might not have seen enough data to accurately estimate the Q values of both.

But surely for large state space problems, the same method would apply? There's an example in a book called "Deep Reinforcement Learning in Action" that has an example of a NN being trained to learn to play Gridworld, which is what I'm basing my question off. So I suppose what I'm asking is: do we use NNs because they're much more efficient at handling the task?

In general, as stated above, nowadays, NNs are the preferred choice (over traditional, tabular approaches to RL) because of the reasons mentioned in the answer above. But if you apply NN to very simple problems, I see two main reasons: 1) It's super convenient to use them because super-efficient, scalable implementations are available for free on the internet; 2) often applying a NN-based  RL-algorithm to a simple task can be seen as some kind of proof of concept (and for illustration). If you can show that it works nicely on small problems, then it will theoretically also work on larger ones.

Besides that, even for games that look super simple (superficially), the number of possible states can grow tremendously large pretty quickly. So, even if you could (still) fit all those states in a table, the required training time could be immense. Just because there is just no generalization in the training outcome when training a tabular RL agent.


text:
What kind of policy evaluation and policy improvement AlphaGo, AlphaGo Zero and AlphaZero are using
I'm trying to find out what kind of policy improvement and policy evaluation AlphaGo, AlphaGo Zero, and AlphaZero are using. By looking into their respective paper and SI, I can conclude that it is a kind of policy gradient actor-critic approach, where the policy is evaluated by a critic and is improved by and actor. Yet still can't fit it to any of the known policy gradient algorithms.


text:
Forcing a neural network to be close to a previous model - Regularization through given model
I'm wondering, has anyone seen any paper where one trains a network but biases it to produce similar outputs to a given model (such as one given from expert opinion or it being a previously trained network).
Formally, I'm looking for a paper doing the following:
Let $g:\mathbb{R}^d\rightarrow \mathbb{R}^D$ be a model (not necessarily, but possibly, a neural network) trained on some input/output data pairs $\{(x_n,y_n)\}_{n=1}^N$ and train a neural network $f_{\theta}(\cdot)$ on
$$
\underset{\theta}{\operatorname{argmin}}\sum_{n=1}^N \left\|
f_{\theta}(x_n) - y_n
\right\| + \lambda \left\|
f_{\theta}(x_n) - g(x_n)
\right\|,
$$
where $\theta$ represents all the trainable weight and bias parameters of the network $f_{\theta}(\cdot)$.
So put another way...$f_{\theta}(\cdot)$ is being regularized by the outputs of another model...

This reminds me of the ELBO objective function, which is composed of two terms: one is the likelihood and the other the KL divergence between the variational density and the prior. Why does your problem/question remind me of the ELBO? Because, essentially, the prior here would be your previously trained model while the variational density is what you are looking. You might want to have a look at it for inspiration (at least). But what's the problem with your current approach of using some kind of norm to regularize?

@nbro I'll have a look at the ELBO objective function; thanks I also had some type of pre-training idea in mind.  To answer your question, basically I was able to show universality of certain models arising from this type of regularization so I'm looking to connect it to pre-existing literature.

Well, actually, this also reminds me of the GAN. See https://deepgenerativemodels.github.io/notes/gan/.

@nbro I gave this some more thought and is this related to transfer learning, where, the second term dictates how for a NN departs from a previously trained model?

This is vaguely related to transfer learning: in transfer learning (at least what people usually mean by that term) you don't jointly optimize one or two models, but you first train a model with dataset 1, then you train it with dataset 2. As I said above, I think you're looking more for something like a GAN or along the lines of a GAN. In the GAN, you will see there are two models and they are conceptually trained jointly, which is what I think you want, if I understand correctly.

@nbro Yes but in the GAN formulations I've seen there's always a minimax problem to solve where here's theres only a min involving an exjogenous model.  Have you seen a GAN formulation where there is no minimax (This would be perfect).

No, I have not seen it, but I just wanted to point you the idea behind the GAN, where there are two models. Of course, you will have to change the objective function to do what you want.


text:
Classification or regression for deep Q learning
DQN implemented at https://github.com/PacktPublishing/PyTorch-1.x-Reinforcement-Learning-Cookbook/blob/master/Chapter07/chapter7/dqn.py uses the mean square error loss function for the neural network to learn the state -> action mapping :
self.criterion=torch.nn.MSELoss()

Could cross-entropy be used instead as the loss function? Cross entropy is typically used for classification, and mean squared error for regression.
As the actions are discrete (the example utilises the mountain car environment - https://github.com/openai/gym/wiki/MountainCar-v0) and map to [0,1,2] can cross-entropy loss be used instead of mean squared error? Why use regression as the state -> action function approximator for deep Q learning instead of classification?
Entire DQN src from https://github.com/PacktPublishing/PyTorch-1.x-Reinforcement-Learning-Cookbook/blob/master/Chapter07/chapter7/dqn.py :
'''
Source codes for PyTorch 1.0 Reinforcement Learning (Packt Publishing)
Chapter 7: Deep Q-Networks in Action
Author: Yuxi (Hayden) Liu
'''

import gym
import torch

from torch.autograd import Variable
import random

env = gym.envs.make("MountainCar-v0")

class DQN():
    def __init__(self, n_state, n_action, n_hidden=50, lr=0.05):
        self.criterion = torch.nn.MSELoss()
        self.model = torch.nn.Sequential(
                        torch.nn.Linear(n_state, n_hidden),
                        torch.nn.ReLU(),
                        torch.nn.Linear(n_hidden, n_action)
                )
        self.optimizer = torch.optim.Adam(self.model.parameters(), lr)

    def update(self, s, y):
        """
        Update the weights of the DQN given a training sample
        @param s: state
        @param y: target value
        """
        y_pred = self.model(torch.Tensor(s))
        loss = self.criterion(y_pred, Variable(torch.Tensor(y)))
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

    def predict(self, s):
        """
        Compute the Q values of the state for all actions using the learning model
        @param s: input state
        @return: Q values of the state for all actions
        """
        with torch.no_grad():
            return self.model(torch.Tensor(s))

def gen_epsilon_greedy_policy(estimator, epsilon, n_action):
    def policy_function(state):
        if random.random() < epsilon:
            return random.randint(0, n_action - 1)
        else:
            q_values = estimator.predict(state)
            return torch.argmax(q_values).item()
    return policy_function

def q_learning(env, estimator, n_episode, gamma=1.0, epsilon=0.1, epsilon_decay=.99):
    """
    Deep Q-Learning using DQN
    @param env: Gym environment
    @param estimator: DQN object
    @param n_episode: number of episodes
    @param gamma: the discount factor
    @param epsilon: parameter for epsilon_greedy
    @param epsilon_decay: epsilon decreasing factor
    """
    for episode in range(n_episode):
        policy = gen_epsilon_greedy_policy(estimator, epsilon, n_action)
        state = env.reset()
        is_done = False

        while not is_done:
            action = policy(state)
            next_state, reward, is_done, _ = env.step(action)
            total_reward_episode[episode] += reward

            modified_reward = next_state[0] + 0.5

            if next_state[0] >= 0.5:
                modified_reward += 100
            elif next_state[0] >= 0.25:
                modified_reward += 20
            elif next_state[0] >= 0.1:
                modified_reward += 10
            elif next_state[0] >= 0:
                modified_reward += 5

            q_values = estimator.predict(state).tolist()

            if is_done:
                q_values[action] = modified_reward
                estimator.update(state, q_values)
                break

            q_values_next = estimator.predict(next_state)

            q_values[action] = modified_reward + gamma * torch.max(q_values_next).item()

            estimator.update(state, q_values)

            state = next_state

        print('Episode: {}, total reward: {}, epsilon: {}'.format(episode, total_reward_episode[episode], epsilon))

        epsilon = max(epsilon * epsilon_decay, 0.01)

n_state = env.observation_space.shape[0]
n_action = env.action_space.n
n_hidden = 50
lr = 0.001
dqn = DQN(n_state, n_action, n_hidden, lr)

n_episode = 1000

total_reward_episode = [0] * n_episode

q_learning(env, dqn, n_episode, gamma=.9, epsilon=.3)

import matplotlib.pyplot as plt
plt.plot(total_reward_episode)
plt.title('Episode reward over time')
plt.xlabel('Episode')
plt.ylabel('Total reward')
plt.show()

I don't want to answer because I'm not 100% but there are some technical details that mean Q-learning function approximators must be trained according to the MSE loss.


text:
Is the Bellman equation that uses sampling weighted by the Q values (instead of max) a contraction?
It is proved that the Bellman update is a contraction (1).
Here is the Bellman update that is used for Q-Learning:
$$Q_{t+1}(s, a) = Q_{t}(s, a) + \alpha*(r(s, a, s') + \gamma  \max_{a^*} (Q_{t}(s',
   a^*)) - Q_t(s,a)) \tag{1} \label{1}$$
The proof of (\ref{1}) being contraction comes from one of the facts (the relevant one for the question) that max operation is non expansive; that is:
$$\lvert \max_a f(a)- \max_a g(a) \rvert \leq \max_a \lvert f(a) -  g(a) \rvert \tag{2}\label{2}$$
This is also proved in a lot of places and it is pretty intuitive.
Consider the following Bellman update:
$$ Q_{t+1}(s, a) = Q_{t}(s, a) + \alpha*(r(s, a, s') + \gamma  SAMPLE_{a^*} (Q_{t}(s', a^*)) - Q_t(s,a)) \tag{3}\label{3}$$
where $SAMPLE_a(Q(s, a))$ samples an action with respect to the Q values (weighted by their Q values) of each action in that state.
Is this new Bellman operation still a contraction?
Is the SAMPLE operation non-expansive? It is, of course, possible to generate samples that will not satisfy equation (\ref{2}). I ask is it non-expansive in expectation?
My approach is:
$$\lvert\,\mathbb{E}_{a \sim Q}[f(a)] - \mathbb{E}_{a \sim Q}[g(a)]\, \rvert \leq \,\,\mathbb{E}_{a \sim Q}\lvert\,\,[f(a) - g(a)]\,\,\rvert \tag{4} \label{4} $$
Equivalently:
$$\lvert\,\mathbb{E}_{a \sim Q}[f(a) - g(a)] \, \rvert \leq \,\,\mathbb{E}_{a \sim Q}\lvert\,\,[f(a) - g(a)]\,\,\rvert$$
(\ref{4}) is true since:
$$\lvert\,\mathbb{E}[X] \, \rvert \leq \,\,\mathbb{E} \,\,\lvert\,\,[X]\,\,\rvert $$
But, I am not sure if proving (\ref{4}) proves the theorem. Do you think that this is a legit proof that (\ref{3}) is a contraction.
(If so; this would mean that stochastic policy q learning theoretically converges and we can have stochastic policies with regular q learning; and this is why I am interested.)
Both intuitive answers and mathematical proofs are welcome.

(1) is a bellman update; it is a copy paste error that rhs has t+1 (sorry about that) thanks for noticing; I fixed the error now.


text:
Extending patch based image classification into image classification
I am trying to classify tampered, pristine images from set of images, in that I have built a network in which I would divide the image into multiple overlapping patches and then classify them into pristine or fake(based on the probability outputs), but now I want extend the same to Image level. That is I want to build some model or some rule over output probabilities of patches of each image to get probability that the image is fake or pristine.
ways I am thinking to do is -

Build a shallow network over the probabilities of the patch probabilities. In this case problem is all images are of different shape
Apply a ML classifier (something like Logistic Regression), with output probabilities by appending zeros to the output probability vector generated so that all image has same sized probability vector as input
generate a mask using patches and then build  a simple classification network over the masks using original image labels.

I can't really say which among the above three is better or worse, I don't even know the possibility of the above three. (Kind of hit a roadblock in thinking)
Now the question am I thinking in right direction, what would be better among the ideas I am considering and why. Is there anything better than what I am thinking. It would helpful in suggesting some resources.


text:
What is a multi channel supervised classifier?
I came across a paper that describes its model architecture in the following way.

Our TRIL network is a two-channel network jointly trained to predict the expert’s action given state and the system’s next state transition given state and expert action. The training procedure of TRIL is similar to that of a multi- channel supervised classifier with regularization. Let $\theta_{π_0}$ be the parameters of TRIL and $L_{ce}$ be the cross entropy loss for predicting expert’s action and $L_{mse}$ be the mean squared error loss on predicting next state given current state and the expert’s action

The loss function is given in the following manner
$$L(\theta_{\pi_0}) = L_{ce}(a, \pi_0(s)) + \lambda L_{mse}(T_{\pi_0}(s,a),s')$$

TRIL is a dual- channel network that shares certain hidden layers and jointly predicts expert action(a) and state transitions(s’)

I am not sure what a dual channel network means and what does it mean when it is able to jointly predict two outputs ? It seems something similar to a multi-task learning since there is shared hidden layers and different "task" prediction but i am not too sure of that either.


text:
When to apply reward for time series data?
Reading the paper 'Reinforcement Learning for FX trading 'at https://stanford.edu/class/msande448/2019/Final_reports/gr2.pdf it states:

While our end goal is to be able to make decisions on a universal time
scale, in order to apply a reinforcement learning approach to this
problem with rewards that do not occur at each step, we formulate the
problem with a series of episodes. In each episode, which we designate
to be one hour long, the agent will learn the make decisions to
maximize the reward (return) in that episode, given the time series
features we have.

This may be a question for the authors but is it not better in RL to apply rewards at each time step instead of "rewards that do not occur at each step"? If apply rewards at each time step then the RL algorithm will achieve better convergence properties as a result of learning at smaller time intervals rather than waiting for "one hour". Why not apply rewards at each time step?

authors: Шевченко Сергій


text:
Why can we use a network to estimate $Q_\pi(s, a)$ in Actor-Critic Method?
According to deep Q learning, we want to learn $Q^*(s,a)$, which is the optimal action-value function. It does make sense because we assume there is only one optimal function so the algorithm will converge supposedly.
But when it comes to actor-critic method, we use critic network (also called  value network) to estimate $Q_\pi(s, a)$. This is what confused me. Since our policy $\pi$ will change through time, the target $Q_\pi(s, a)$ of value network will also change. What will happen for a network to learn a changing function?

The policy doesn't change over time. That is, the values will change, otherwise we would not be learning anything, but our rules for action selection don't. I.e. we always take action according to the distribution postulated to our current estimate of the policy $\pi_\theta(a|s)$, we don't suddenly start taking $\max_a \pi_\theta(a|s)$, which would be a true change in policy and would make learning both the actor and the critic unstable.
Because NN's are able to handle noisy target distributions, this is how they can deal with the changing data. If you think of how Actor-Critic methods work, you would initially start to shift your NN to some unfeasible values (due to random initialisation and the Actor-Critic not having any information about the environment), but as you start to interact with the environment you will start to update the agent towards the 'true' policy.
An analogy in supervised learning would be to have some noisy data which is incorrect and some true data. If you trained your network on the noisy data for a small number of epochs and then never showed it to the network again and trained it solely on the correct data, it would forget it has ever seen the noisy data and only represent the new, true data.

The rules for derivation of the target policy do not change. However, the target policy does change over time in both cases (Q learning and actor-critic) as estimates are refined. In turn that means the value functions being learned also change over time in both cases the OP is interested in. The reason it works is that NNs can deal with non-stationary target distributions, forgetting the old values.

@NeilSlater I meant more that the policy doesn't change in the sense that we take actions according to the distribution (which will change) consistently, we don't suddenly start taking e.g. $\max_a \pi_{\theta}(a|s)$.

Yes, I was justclarifying that in my comment. OP seems to have conflated derivation rule not changing and policy not changing though, so it is important to be clear IMO

right -- I will edit my answer to reflect this.


text:
Is it good practice to save NLP Transformer based pre-trained models into file system in production environment
I have developed a multi label classifier using BERT. I'm leveraging Hugging Face Pytorch implementation for transformers.
I have saved the pretrained model into the file directory in dev environment. Now, the application is ready to be moved the production environment.
Is it a good practice to save the models into file system in prod ?
Can I serialize the model files and word embeddings into any DB and read again ?


text:
Why does not the deepAR model of Amazon require the time series being stationary, as opposed to ARMA model?
As what the title said. Does not deepAR require the time series being stationary?

I have have been part of a project where we implemented Amazon Forecast service in production. As per my practical experience, we need not worry about stationarity while applying DeepAR.
As per my understanding as in ARIMA the prediction function is a function around the time series moving average, but in DeepAR we are more relying on the backtest window and leg-horizon length.
DeepAR uses recurrent neural Networks (RNN) and gated recurrent unit (GRU) without any assumption on their probability distribution to learn the sequence for prediction.
For a detailed explanation, you can refer. https://aws.amazon.com/blogs/machine-learning/forecasting-time-series-with-dynamic-deep-learning-on-aws/

in the official video of aws, it says that data MUST be stationary. https://youtu.be/g8UYGh0tlK0?t=1406


text:
Combining clustering and deep learning for computer vision
Is there any recent work on combining clustering approaches (k-means, or gaussian mixture or PGM) with deep learning for computer vision?
In particular I'm interested in if anyone has used the first few layers of a deep learning network as feature extractors in conjunction with clustering algorithms which have been engineered to induce things like translation and rotation invariance while preserving basic object structure?
Taking the max value out of the output of each feature and fitting them to a gaussian mixture is the easiest approach but I'm interested in seeing other ways you could structure the clustering algorithm. For example I'm interested in seeing how you might learn structure between features that includes position information.

My interpretation is that you want to take non-deep learning based clustering algorithms and use deep learning to improve on them in a collective way.  If so, why not just use deep learning on feature data?  There are several deep learning clustering algorithms.

Deep learning can be unreliable for consistent detection and sometimes produces completely wrong results without ability to interpret them.

Is your statement that "deep learning can be unreliable" a general one or is it in regards to results you have seen using DL-based clustering on your data?

My premise, wrong or right is that classic clustering algorithms are less prone to catastrophic failure that sometimes is seen neural networks so I'm interested in seeing how clustering algorithms could be used in conjunction with low level feature extractors.


text:
OpenAI gym's CartPole problem system does not learn
My OpenAI CartPole-v0 problem's implementation using basic Q-learning does not learn at all.
I am a beginner and have implemented my first ever Q-learning from scratch after learning from tutorials.
Can anyone suggest what is going wrong?
I have seen through testing that the problem may be that most of the states are remain unvisited even after 10,000 runs. Hence, Q-table remains mostly unchanged at the end of all episodes. I have seen other things in the implementation and they all seem fine to me, at least. Any tip where I should start looking at?
The reward is -200 flat, for all the episodes! which suggests that the improvement is NILL/NADDA/NONE!
Some relevant images are given at the end.
The q-learning part of code is given below:
env.reset()
while not done:
    current_state = current_state_to_string(assign_obs_to_bins(obs, bins))

    if np.random.uniform() < EPSILON:
        act = env.action_space.sample()
        best_q_value = return_max_from_dict(q[current_state], action = act)
    else:
        act, best_q_value = return_max_from_dict(q[current_state])

    obs, reward, done, _  = env.step(act)
    q[current_state][act] += LEARNING_RATE * (reward + DISCOUNT_FACTOR * best_q_value - q[current_state][act])
    cnt+=1
    total_reward += reward

I would like to add one more thing that I tried epsilon-greed approach, greedy approach, and random approach. All gave me the same result in terms of reward and in all cases, most of the states remained untouched.

Why are you using Q learning rather than DQN?

because I am a beginner and do not want to ride a sports bike before properly learning to balance on the bicycle :)

try to test your implementation on a simple grid world and see if it works. After that you need to properly discretize state space. Try using simple case first, use only 2 states, one state when pole is to the left of the 90 degrees, and second when its right to 90 degrees. Then keep refining your discretization if your performance is not good enough.

CartPole will not be solvable with tabular DQN without some tuning of the state space as @brale mentioned. Just do some of the exercises from the Sutton and Barto book if you want to practice Tabular Q-Learning.


text:
Does Algorithmic Mechanism Design come under the field of AI?
I see many papers in AAMAS talk about artificial intelligence and mechanism design simultaneously. I was wondering, for the sake of being pedantic, is mechanism design could be classified under AI.

It seems to be an interdisciplinary field. If in the end you tend to have a model to predict something, which could be the case, then you can consider it as AI subfield as well.


text:
Multilabel stratified split for images/object detection
I am working on an object detection model and have thought of looking into stratified splits for the dataset.
Now since I am doing object detection I have a variable number of "labels" for every image because in each image there is variable number of occurrences for each object I am looking for (car, truck, motorbike, etc.).
Obviously single-label stratification does not apply.
From what I understand multi-label stratification is only applicable if there are basically label "features" that we know are always present, which does not seem the case here.
My question is... is there a way to perform stratified split in this case so that in each split there is roughly the same number of cars/trucks/bikes/etc.? (Or is it going to actually improve the results at all?)

group them on the basis of the feature of the multilabel

@RithikBanerjee can you please explain in more detail? The very definition of multilabel seems to be that there are multiple features which can be considered labels.

@RaresDima Did you find an optimal solution?

@SadraNaddaf no. There isn't one. The best you can do is consider this as a partitioning problem and approach it in a greedy way because of the massive amount of data.

@sadranaddaf  If you explain and clarify your question I can post my experience/code as an answer here. If you have roughly split each label to train/dev/test with 70%/10%/15% so then you have ended up with a different number of images in each set but they roughly have an equal % of each class label. Is it the same as what you want? I did a similar thing on my OD task.

@RaresDima I should have mentioned you in previous comment


text:
How to train an LSTM with varying length input?
I have a dataset where each of the training instances is different in the length and the data is sequential. So, I design an LSTM but I am thinking about how to train the LSTM. In fixed-length data, we just keep all of the input in an array and pass it to the network, but here the case is different. I can not store varying length data in an array and I do not want to use padding to make it fixed length. Now, should I train the LSTM where each training instance are varying in length?

Are you predicting a sequence e.g. like in machine translation?

I am performing prediction but not like the machine translation

You could sequentially pass in each element of your sequential data and save the hidden and cell states in a separate buffer. In a typical LSTM implementation, you input the entire sequence and the hidden and cell states are propagated internally. In the end, the final hidden and cell states returned as the output. This works if your input is all the same length. Instead, you can handle sequentially giving the next element to the LSTM as well as the hidden and cell state yourself. To keep it efficient you can batch your inputs by the batch dimension (batch_first=True in the pytorch LSTM implementation).
For example, propose you have 5 sequences of length 5, 4, 3, 2, and 1. You initialize your hidden and cell state buffers for each of the sequences and pass the first batch containing the first element of all 5 sequences. You save the output hidden and cell states in the buffers for each sequence. Next, you input the batch of the second element of the 4 sequences with length > 1, and save the states in the respective sequence buffers and so and so forth until you exhaust the sequence of greatest length.

What if all of the sequences are different length, so I can not batch the same length sequence together?

You don't need to batch the sequences, you just need to batch the elements of the sequences. Batch all the first elements of your sequences into one vector (or matrix if your elements are multidimensional), all the second elements in another etc.

I do not want to pass the first batch containing the first element of all 5 sequences like this, I am also curious about why one should pass input like this?

Because this avoids having to pad your input to efficiently process variable length sequences. AFAIK the LSTM implementations in all typical python packages receive input of size Batch x Sequence x Hidden (or a permutation). Since your sequence dimension is not the same for every sequence, you either need to pad your sequences. Or as suggested in the answer, you can change the sequence dimension to 1, but need to handle the hidden and cell states of the LSTM yourself.

This question might be deep learning framework specific, so my answer is intended for Keras.
In keras, an LSTM cell accept 3D tensor, which is (num of samples, num of timesteps, num of features). The question is about varying num of timesteps.
My naive approach will be like this:

Define model architecture in a function that accept num of timestep as parameter that returning model object
Prepare next sample
Instantiate a model with num of timestep based on current sample
Load the latest weights if exist
Train one epoch
Save the latest weights
Back to number 2

So, the code  will be looks like this:
from keras.models import Model
from keras.layers import LSTM, Input, Dense
import numpy as np

def my_robust_lstm_model(num_of_timesteps=512, num_of_features=1):
    input_layer = Input(shape=(num_of_timesteps, num_of_features))
    hidden_layer = LSTM(64)(input_layer)
    output_layer = Dense(1)(hidden_layer)  # Example output layer
    model = Model(inputs=input_layer, outputs=output_layer)
    return model

def array_generator(my_list_pickle_path):
    # Implement how one sample X_train, y_train looks like that has varying length
    # Example implementation: (Replace this with actual data generation logic)
    while True:
        X_train = np.random.random((1, 10, 1))  # Dummy data with varying lengths
        y_train = np.random.random((1, 1))
        yield X_train, y_train

generator = array_generator('path/to/pickle/file')
EPOCHS = 10

for _ in range(EPOCHS):
    current_X, current_y = next(generator)
    num_of_timesteps = current_X.shape[1]
    num_of_features = current_X.shape[2]
    current_model = my_robust_lstm_model(num_of_timesteps=num_of_timesteps, num_of_features=num_of_features)

    try:
        current_model.load_weights('latest.weights.h5')
    except Exception as e:
        print(f'Weights don\'t exist or cannot be loaded: {e}, skipping')

    current_model.compile(optimizer='adam', loss='mse')  # Example compile parameters
    current_model.fit(current_X, current_y, batch_size=1, epochs=1)
    current_model.save_weights('latest.weights.h5')


text:
Strange behavior of Q-learning agent after being trained
I built a simple X*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment.
Environment:

A 4x4 grid world with episode starting at (0,0) and terminal state (3,3)
Four actions: Left, Up, Right, Down
The reward of -1 for moving into new state from the previous state to a new state. The reward of 0 when reaching the terminal state. The reward of -2 for bouncing off of the boundary
Epsilon-greedy scheme for action selection.

All works fine, and the following are the learning results of the agent.

Later I ran a test run of my TRAINED QL-agent where I used greedy action selection. All I could see in every episode was that my agent start from (0,0), take right to move to (1,0), and then take left to move back to (0,0) again and this goes on and on and on... I check the Q table and it makes sense because the Q-values for these actions justifies such behaviour. But this is not a practical agent should be doing.

I don't think you will get an answer due to the data you have provided. The simple conclusion is that you have a mistake somewhere in the implementation. I would suspect something in how you have stored/interpreted the Q table or re-implemented the environment when moving from training to testing.

Agreed with Neil Slater, and I also have another question. The return that you converge to is around -20, which seems very low to me. Shouldn't your agent be able to solve this in like 6 steps, achieving a return of -5?

@harwiltz I checked and found it reach min. a score of -9, with a Toral reward of -8. But mostly the number of steps in 10s or 20s for a trained agent.

So is it possible that in your evaluation episodes the agent goes back and forth around 15 times before finally going to the goal?

Still, this is very strange behavior, especially since there's a negative reward at each timestep. Sounds like your learning algorithm might not be working correctly, since it shouldn't be too hard for it to converge to an optimal (or at least nearly optimal) policy.

I get your point. I have been looking into why it is doing so. Convergence shouldn't be a big issue because the environment is simple, the agent's logic is simple but I am looking into it. Thank you, the discussion helped in better understanding of RL developmental issues for me.


text:
Looking for the proper algorithm to compress many lowres images of nearby locations
I have an optimization problem that I'm looking for the right algorithm to solve.
What I have: A large set of low-res 360 images that were taken on a regular grid within a certain area. each of these images is quite sparsely sampled and each of these 360 images has an accurate XYZ position assigned of its center. There are millions of these small images, clusters of close-by images obviously share a lot of information while images farther apart can be completely different.
What I want to do is to compress these small 360 images.
If two 360 images are close by each other, they can be 'warped' into each other by projecting it onto a sphere of finite distance and then moving that sphere (so a closeby 360 image can be a good aproximation of another 360 image when it has been warped that way).
Based on this idea, I want to compress these small low-res 360 images by replacing each of them with:

N (N being something like 2-5) indices into an archive of M (M being something like 50-500) different 'prototype' images (of possibly higher resolution than the low res 360 images), each of which has an XZY location assigned plus a radius
N blend weights

Such that if I want to reconstruct one of the small, sparsely sampled 360 images I take the N indices stored for this image, look-up the corresponding prototype images from the archive, warp them based on the radius of the archive image and the delta vector of archive XZY and compressed image XYZ location, and then blend the N prototype images based on the N blend weights (and possibly scale down in the prototype images are higher res)
I guess this goes into the direction of Eigen Faces, but with Eigen faces each compressed face has a weight stored for each eigen-face, whereas I want that each compressed sphere only has N non-zero weights.
So my input is:
a lot of small 360 images plus a XYZ location each
my output should be:

an archive of M "prototype" images, each assigned an XYZ location and a projection radius
all compressed spheres, with each sphere compressed to N indices and N weights

This seems to be some non-linear least squares problem, but I wonder if someone can point me into the right direction on how to solve this?
As a completely alternative approach I also looked into spherical harmonics, but with those I only get enough high-frequency details at l=6 which takes 36 coefficients which is too much and also too slow to decompress.


text:
What is the best way to make a deep reinforcement learning environment with a continuous 2D action space?
I understand that the actor-critic method is probably where I want to start because of how it works with continuous action spaces.
However, the problem I am trying to solve would require the action be a vector of 11 continuous values. When I go to design my training environment and the architecture of my DRL network, I am not sure how to map a vector of values to the state for the state-action pairs.
I am trying to use this article as a jumping off point, but I am not sure where to go: https://medium.com/@asteinbach/actor-critic-using-deep-rl-continuous-mountain-car-in-tensorflow-4c1fb2110f7c


text:
NLP: What is expected from the output of a perfect coreference system?
For instance, consider the following piece of text:
'The father of Richard is a very nice guy. He was born in a poor family. Because of that, Richard learnt very good values. Richard is also a very nice guy. However, Richard's mother embarasses the family. She was born rich and she does not know the real value of the money. She did not have to be a hard worker to succeed in life."
How should a perfect coreference should work? Is this solution the perfect solution?
Cluster 1:
'The father of Richard' (first sentence) <-> 'He' (second sentence)

Cluster 2:
'Richard' (third sentence) <-> 'Richard' (forth sentence)

Cluster 3:
'Richard\'s mother' (fifth sentence) <-> She (sixth sentence) <-> she (sixth sentence) <-> She (seventh sentence)

If I use the coreference library of spacy (neuralcoref), I get these clusters:
Clusters:
[Richard: [The father of Richard, He, Richard, Richard, Richard], a poor family: [a poor family, the family], Richard's mother: [Richard's mother, She, she, She]]

Note that this output says that "Richard" is the same for sentences that is true. However, "He" in the second sentence is not related to "Richard", but to his father. Also "Richard" and the "Father of Richard" are together in the same cluster. Furthermore, "poor family" and "family" should not come together. However, this is realy difficulty since in this case there is some level of ambiguity.'
I know that this is a very difficult problem. The point is not criticize this fantastic library. I am just trying to understand what I should expect as perfect result.
If I change a little the text:
'The mother of Richard is a very nice woman. She was born in a poor family. Because of that, Richard learnt very good values. Richard is a very nice guy. However, Richard's father embarasses the family. He was born rich and he does not know the real value of the money. He did not have to be a hard worker to succeed in life.'
The clusters are:
[Richard: [The mother of Richard, She, Richard, Richard, Richard, He, he, He], a poor family: [a poor family, the family]]

In this case, the clusters become stranger, since "She" and "Richard" are in the same cluster. Furthermore, the "He" related to the "father of Richard" belongs to the cluster, but not "Richard's father".
So, my question is:
What is the perfect result that I should expect from a "perfect" coreference system?

The clusters you mentioned at the beginning seem right. A "perfect" coreference system should be able to find all words/phrases that refer to the same object.
However, in language, there will always be some ambiguity. For example, consider the sentences "Bob and John like building. He has fun doing it.". It is ambiguous whether "He" refers to "Bob" or "John". In that sense, a perfect coreference system may not even exist theoretically. If it did, it would either avoid classifying "He" or recognize that it could go with either "Bob" or "John".

"Richard" and "the father of Richard" belong to the same cluster is not ambiguity, is it?

They definitely should not belong to the same cluster.


text:
How to make sense of label propagation formula in graph neural networks?
In the label propagation algorithm in section 3.2.3, we know the label of some nodes and we want to predict the label for the rest of the nodes whose labels we don't know. The update formula for this is the following:
$$F(t+1) = \alpha SF(t) + (1-\alpha)Y $$
where $F(t)$ is predicted label from timestep t and $S$ can be considered as an adjacency matrix, $Y$ is the label for both the unlabeled data and labeled data. In the case of labeled data, we initialize $Y$ with ground truth and for the unlabeled data, we randomly initialize their label and assign it to $Y$.
Now, the most problematic part is I think the $Y$ matrix. Since I do not know the label of some nodes, so we initialize with some random value and keep Y as a constant throughout this iterative process.
We can calculate the optimal value of F directly using:
$$F^{*} = (I - \alpha S)^{-1}Y$$
But my question is, if we keep Y as a constant ( assign random numbers to unknown nodes as labels) what kind of sense does it make?

Comments are not for extended discussion; this conversation has been moved to chat.

One question is: why is $S$ considered to be an adjacency matrix when the formula $\Delta = I - S$ implies that $S = D^{-1} A $ or a symmetric-normalized version? Is it just because we don't care about the constants?

Sorry, one other question is: why did we drop the $ (1 - \alpha) $ from the front of the expression for $F^{*}$? Is it just because it is a constant?


text:
Can I use ML to discover via videos the best place to shoot in foosball?
I am a programmer, but just now attempting to enter the world of ML. I'm eyeballing a potential project/problem related to foosball.
Pro foosball is a thing believe it or not and I'm wondering if I can use decades worth of game footage to determine where defensive holes are most likely to be.
The way shooting works in pro foosball is you front pin the ball and walk it back and forth in front of the goal. The defense meanwhile is attempting to randomly move two men in front of the goal. Of course, our human brains are not truly random and this I'd like ML to help me understand and exploit.
Questions like, if I walk the ball left, then step right, historically where is the open hole likely to be?
If you'd like to better understand the nature of shooting, here is video of pro foosball: https://www.youtube.com/watch?v=uOdnqmwOQhA&t=16s
So what ML topics should I research and what strategies and tools do you recommend for creating such a model?


text:
What is the state-of-the-art algorithm for neural style transfer?
I've read the paper A Neural Algorithm of Artistic Style by Gatys et. al. and I find the application of neural style transfer very fun.
I also read that Exploring the structure of a real-time, arbitrary neuralartistic stylization network by Ghiasi et. al. is a more modern approach to NST.
My question is whether the above paper by Ghiasi et. al. is still the state-of-the-art method in NST, or maybe new algorithms perform even more efficiently.
I shall precise that my goal is to deploy some NST algorithm on a web page as a fun project to apply some deep learning and learn about backend-frontend interactions.

you can find all the state-of-the-art papers related to all field in machine learning via this community: https://paperswithcode.com/

So for neural style transfer, using the particular method described in Gatys paper, nobody has done better than using VGG net. This is seemingly due to VGGs inherent stability and inability to learn non-robust features of images. More on this here: https://reiinakano.com/2019/06/21/robust-neural-style-transfer.html
That being said, GANs have had huge success in the field of style transfer, getting much better results than the neural style transfer method described in the paper you mentioned. Cyclegan is one of the best in this respect: https://machinelearningmastery.com/what-is-cyclegan/


text:
How to overfit GANs with a single image
When designing CNN for image recogition a commonly used sainty check to see if a model is working/designed fine is to see if we are able to overfit the model with a very small subset of images.
I am trying out GANs. While designing GAN I took a dataset with just one image(full black image). I used the DCGAN implementation in pytorch websitecode link.
I tried training the model with this just one black image and even after training for 100s-1000 epochs. I am not able to overfit the model ie generate a black (or something close). All what is generated are random noise image as below
However the model does work well for celeba dataset(the one used in the tutorial). Which means the model is good. Can anybody help me why overfitting is very difficult/impossible when using a single image.

Complete guess here, but it might be the loss function. If the desired image is completely black perhaps all the 0 values are not giving any cost, and so the GAN never actually learns anything.

hi @Recessive , Thanks for the response this does make a point. To validate on this I took an image from MNIST dataset and I tired overfitting the model and tried training for 1000 iterations. I was still getting Random Noise. Then I had a thought that the number of epochs (1000) were a lot less as the original MNIST had 60000 images. I trained the model with mnist image for 10000 iterations and within almost 4000 iterations it started to overfit and started mimicing my input image.  Then I went ahead and tried with the black image and it was difficult for it to mimic the black colour

but however after about 10000+ epochs the noise was still present but I got an increase in pixels that were close to black.

Hi, I am working on the same problem now. Did you manage to figure out the problem? Thanks

Training more epochs was the solution


text:
How to run a Monte Carlo Tree Search MCTS for stochastic environment?
For MCTS there is an expansion phase where we make a move and list down all the next states. But this is complicated by the fact that for some games, after making the move, there is a stochastic change to the environment. Consider the game 2048, after I make a move, random tile is generated. So the state of the world after my next move is a mix of possibilities!
How does MCTS work in a stochastic environment? I am having trouble understanding how to keep track of the expansion, do I expand all stochastic possibilities and weight the return via their chance of happening?

MCTS has been used for two-player games like Go, so maybe you could consider the random move as simply the move of the opponent.

That is something I don't understand. In Go the player can have hundreds of different responses but you can use a metric to choose a branch (essentially, assuming your opponent plays optimally, but u r guess optimality with a heuristic). with random play you don't know which move will be played so you have to track through all possible moves.

Yeah, I see your point. Maybe you could use random sequences (which are fixed for each tree search) of new tile positions? Of course, you would run into the complication of a new tile overlapping an existing tile in some iterations, so some way of countering that would need to be put in place.

I am having trouble understanding how to keep track of the expansion, do I expand all stochastic possibilities and weight the return via their chance of happening?

This is indeed one option you can take. This would be very similar in spirit to the idea of "Expectimax" as a variant of minimax for non-deterministic games, in the sense that you'll include explicit "chance nodes" in your tree. When running into such a chance node later on again during a Selection phase, of a later MCTS iteration, you can just select a path of the tree to follow based on a "dice roll". Importantly, note that this option is only actually available if you have explicit knowledge of exactly when chance events occur, which states they can lead to, and with which probabilities they lead to different states. We also assume that this is feasible, i.e. that you don't have a crazy high (or infinite) number of slightly different game states you could reach.
An alternative option is to use an "open-loop" variant of MCTS. Your nodes would no longer represent game states, but only be representative of the sequence of actions leading to them. You would no longer store any game states in any nodes, but always regenerate them from scratch when traversing the tree, starting from the root node. You would no longer have any explicit chance nodes, but instead have states being representative of larger sets of states that could possibly be reached by following the corresponding path from the root node. For more on this, see my answer to this other question. The advantage of this approach is that it does not require explicit knowledge of all the possible states you can reach due to chance events, do not need explicit knowledge of the probabilities, and can just sample instead of explicitly enumerating every possible outcome.


text:
How does DQN convergence work in reinforcement learning
In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case
The network predicts its own target value, now how exactly does it converge if the network predicts its target value
Can someone explain this to me ??

Initially, the target will be totally wrong but as the process goes on and on it will become closer to  the unbiased estimate, then it will eventually converge

Also, remember that this target is an unbiased estimate of the optimal Q value.

@Swakshar Deb Why does it get closer to the unbiased estimate, what exactly makes it that way

@Swakshar Deb why does it get closer to the unbiased estimate, what in that formula makes it the unbiased estimate

$q^{}=E[R+\gamma \max_{a}Q(s_{t+1},a_{t+1})]$, that means $R+\gamma \max_{a} Q(s_{t+1},a_{t+1})$ is the unbiased estimate of $q^{}$. So, if the iteration is large enough the target will eventually converge to the true estimate.

@Swakshar Deb I would have liked the answer as a post for other users, but this is fine , Thanks a lot , I appreciate

@ChukwudiOgbonna Can you please describe in your own words your problem without taking screenshots? Can you edit your post to do this? If we take the time to read your posts and try to help, you should also take the time to write a question/post properly and show that you have put some effort into writing.

@SwaksharDeb I suggest that you a formal answer to the question (if you have one: I didn't read the comments above), given that comments are meant to be temporary.

Sorry, I don't have the mathematical proof for the convergence. I have to look for it. If I get one I'll post it.


text:
Can I resize my images after labeling them?
Is it okay if I label my images with their original size and then resize them, or should I first resize them and then label them?
I mean do I need to recalibrate my labels if I resized my images?

If resizing does not change the contents of the images, then you probably can first label them and later resize them. However, resizing can actually change the $x$ and $y$ dimensions differently and make certain objects look different (e.g. stretched) and maybe no more really recognizable. I would say that if resizing changes both dimensions by the same amount, then it should be safe to label the images first, but, to be honest, I have never really studied this issue seriously.

@nbro Thanks for the response, I will consider that.

your labels are related to x,y positions into the image? if yes, you must change the x,y in labels by the same factor you use on resize your photo.

@AliKHalili In my comment above, I assumed you're doing image classification. However, as Rogelio wrote above, it's true that, if you're trying to solve a task like image segmentation (i.e. dense classification), then you'll probably need to change the labels after resizing.


text:
Is there a problem for "Sound Source Identification in Video Footage"?
I've been considering starting a project for some time on sound source identification.
To be more specific, my goal is to be able to identify the "sources" for sound in videos. Moving parts clanging, lips speaking, hands clapping etc. I'd like to think a model trained to be able to do this might be helpful for:

Identifying who is saying what in a crowd
Discovering noise sources caught on video (ex. a carpenter's saw as he is talking to someone)
Extending this to design a model for reading lips, to discern speech in silent video.

(Taken from https://www.youtube.com/watch?v=8ch_H8pt9M8)
You might think of this task like grounding in NLP, expect for sound/speech instead. I'm sure this has been done before, and I'd like to conduct a literature review. So is there a name for this kind of sound-source identification?
I've tried Googling "Sound Source Identification", but it only returns Speech Classification results (Is this sound a car or a truck etc.)

There is "speaker identification", which is your first bullet point, although I don't know about the others. I think the model for reading lips would be separate from the rest of what you are trying to do however, since the rest of it can utilize the sound in the video.

Well to my knowledge, "speaker identification" is solely an audio problem: "Is the person currently talking someone I know, and if so who?".


text:
Alternatives to Hierarchical RL for centralized control tasks?
Consider a problem where the agent must learn to control a hierarchy of agents acting against another such agent in a competitive environment.  The agents on each team need to learn cooperate in order to compete with the other agents.
A hierarchical RL algorithm would seem to be ideal for such a problem, learning a policy that includes sub-policies for sub-agents.  But are there are other types of algorithms that could be used for this kind of task, perhaps ones that are involved centralized cooperation but aren't considered hierarchical RL?


text:
experiences on using genetic algorithms as a way to improve neural networks?
I wonder if there is research, patents, or libraries using Genetic algorithms (GA) to improve Neural Networks. I don't find anything in the subject. For example:

use GA to find better parameters in a NN. So the chromosome will be [learning rate, activation function, layers number, layers size, dropout factor] and the fit function minimize computational cost to reach NN 95% accuracy.
use GA to mix your NN input data and generate new data to adjust.
use GA to mix several small NN, different types, and find the perfect mix for better predictions.

Are you aware of neuro-evolution?


text:
Understanding graphs of the mean square error: relationships between val loss and train loss
I am currently working with some models aimed at predicting time series (89 days for training, 22 for testing), including a CNN LSTM and a convLSTM.
When training these models, I had the following scenario:

In the first case, it is possible to see the val loss moving more sharply away from the train loss. In the second case, it seems to me that this also happens, but in a much smoother way.
What do these graphs mean? What causes these situations to occur? If they are problematic situations, is it possible to correct them? If so, how?


text:
Where can I find short videos of examples of RL being used?
I would like to add a short ~1-3 minute video to a presentation, to demonstrate how Reinforcement Learning is used to solve problems. I am thinking something like a short gif of an agent playing an Atari game, but for my audience it would probably be better to have something more manufacturing/industry based.
Does anyone know any good sources where I could find some stuff like this?

Does your presentation need to be stand-alone? Google Slides can embed YouTube videos very simply. I did just that for a presentation on RL recently, and simply used YouTube's search. The advanced version lets you select short clips. DeepMind, OpenAI and others upload clips there.

@NeilSlater I'm not sure what the format has to be actually -- I was using Overleaf but if my supervisor says it is okay then I can check out Google slides. Thanks!

Please, have a look at this, this, this and this question. If any of those answers answer your question, let me know

@Arpit-Gole no, these look like sources to lectures etc. I'm not looking for lectures/resources for learning, I want a video demonstration e.g. of a robotic hand that was trained by RL.

You can take some ideas from this YouTube video .
In addition, you should consider that page which is about Deep Reinforcement Learning used in a game (Pong from Pixels) .


text:
Customized food for persons based on their profile using Reinforcement learning
I am newbie to Reinforcement Learning, this is my idea - Agent(food provider) has to select a food based on the environment(based on the user profile). Here the reward will be given to the agent based on the user's feedback. This is for single person, what if I wanted to make it for multiple persons. I wanted the system to learn on its own and create a policy such that it can identify certain group of people based on their profile and what type of food will be suitable for them.

Is this possible to implement in Reinforcement learning?
If so what type of problem is this and what type of solution I can use to solve this.

I too need an approach like the above question, for my similar problem set

Please specify the shape and structure of your dataset more clearly.


text:
What is the scope of real-world deep learning applications in 2020?
2015 was a milestone year for AI--"deep learning" was validated in a very public way with AlphaGo.  However, at the time, the question was raised: "What else is deep learning good for?"
5 years later, I want to gauge:

How is deep learning applied to real world problems in 2020?  What real world applications is it currently used for?

Deep learning is used to perform language translation in Google Translate [1]. Specifically, Google Translate now uses transformers and RNNs rather than the original GNMT system (proposed in 2016), which was also based on neural networks. Deep learning is also used in DeepL (though I cannot find a good resource to cite apart from Wikipedia [2] given that the system is closed-source), a strong alternative to Google Translate. However, note that, in general, machine translation is still far from perfect and it is probably not adopted to perform serious translations.
Tesla's autopilot also uses neural networks [3].

Built on a deep neural network, Tesla Vision deconstructs the car's environment at greater levels of reliability than those achievable with classical vision processing techniques.

DeepFakes are also developed using deep learning techniques [4].
Neural networks are also being used for chords and beat detection of songs [5].

I will be updating this answer, as I gather more information.

Another application:https://en.wikipedia.org/wiki/DeepFace


text:
How can I implement 2D CNN filter with channelwise-bound kernel weights?
I would like to bind kernel parameters through channels/feature-maps for each filter. In a conv2d operation, each filter consists of HxWxC parameters I would like to have filters that have HxW parameters, but the same (HxWxC) form.
The scenario I have is that I have 4 gray pictures of bulb samples (yielding similar images from each side), which I overlay as channels, but a possible failure that needs to be detected might only appear on one side (a bulb has 4 images and a single classification). The rotation of the object when the picture is taken is arbitrary. Now I solve this by shuffling the channels at training, but it would be more efficient if I could just bind the kernel parameters. Pytorch and Tensorflow solutions are both welcome.

Assuming you want HxWx1 kernel to perform convolution on hxwxc images.
Here's sample code which uses single channel kernel to operate on multichannel feature: maps
import torch
import torch.nn as nn
import torch.nn.functional as F

class model(nn.Module):
    def __init__(self, in_ch=4):
        super().__init__()
        self.in_ch  = in_ch
        # single channel kernel initialization
        self.kernel = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=1, stride=1)

    def forward(self, x):
        bs, ch, w, h = x.shape
        x   = x.view((bs, ch, 1, w, h))
        out = self.kernel(x[:, 0])
        # reusing of same kernel
        for i in range(1, self.in_ch):
            out = torch.cat((out, self.kernel(x[:, i])), 1)
        return out

net = model(4)
print(net)
inp = torch.randn((10, 4, 100, 100))
out = net(inp)
print(out.shape)

(The main hack is in the forward function)

In PyTorch implementation of convolution modules, you can just set the out_channels and groups argument to be the number of your input channels. In your case, if your input has 4 channels, you can simply construct the layer as
torch.nn.Conv2d(in_channels=4, out_channels=4, kernel_size=3, groups=4)

This Keras solution uses a shared sub-model, and then applies it to each of the input image's channels separately. The architecture is very similar to a Siamese network.
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.layers import Input, InputLayer, Dense, Flatten
from tensorflow.keras.layers import Conv2D, MaxPooling2D

import tensorflow.keras.backend as K

res = 128

sub_model = Sequential([
    InputLayer((res,res,1)),
    Conv2D(16, 5, activation='relu'),
    MaxPooling2D(2),
    Conv2D(16, 5, activation='relu'),
    MaxPooling2D(2),
    Conv2D(16, 5, activation='relu'),
    MaxPooling2D(2),
    Flatten(),
    Dense(512, activation='relu'),
    Dense(128, activation='relu'),
])

sub_model.summary()

Summary:
Model: "sequential_8"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_19 (Conv2D)           (None, 124, 124, 16)      416
_________________________________________________________________
max_pooling2d_17 (MaxPooling (None, 62, 62, 16)        0
_________________________________________________________________
conv2d_20 (Conv2D)           (None, 58, 58, 16)        6416
_________________________________________________________________
max_pooling2d_18 (MaxPooling (None, 29, 29, 16)        0
_________________________________________________________________
conv2d_21 (Conv2D)           (None, 25, 25, 16)        6416
_________________________________________________________________
max_pooling2d_19 (MaxPooling (None, 12, 12, 16)        0
_________________________________________________________________
flatten_7 (Flatten)          (None, 2304)              0
_________________________________________________________________
dense_4 (Dense)              (None, 512)               1180160
_________________________________________________________________
dense_5 (Dense)              (None, 128)               65664
=================================================================
Total params: 1,259,072
Trainable params: 1,259,072
Non-trainable params: 0
_________________________________________________________________

Then the combined model:
n_channels = 4
inp = Input((res,res,n_channels))
x = K.concatenate([sub_model(inp[:,:,:,i:i+1])[:,:,None] for i in range(n_channels)])

model = Model(inp, x)

x.shape
# TensorShape([None, 128, 4])

Naturally it is up to you whether to have those Dense layers in the sub_model or not, and which extra operations you'll do to the concatenated x.


text:
Best/quickest approach for tuning the hyperparameters of a restricted boltzmann machine
I have an RBM model which takes extremely long to train and evaluate because of the large number of free parameters and the large amount of input data. What would be the most efficient way of tuning its hyperparameters (batch size, number of hidden units, learning rate, momentum and weight decay)?


text:
Need some reviews in PEAS descriptions
Here is the Question:
Describe the PEAS descriptions for the following agents:
a) A grocery store scanner that digitally scans a fruit or vegetable and
identifies it.
b) A GPS system for an automobile. Assume that the destination has been
preprogrammed and that there is no ongoing interaction with the driver.
However, the agent might need to update the route if the driver misses a turn.
c) A credit card fraud detection agent that monitors an individual’s transactions
and reports suspicious activity.
d) A voice activated mobile-phone assistant
For each of the agents described above, categorize it with respect to the six dimensions
of task environments as described on pages 41-45 (Section 2.3.2 of AIMA). Be sure
that your choices accurately reflect the way you have specified your environment,
especially the sensors and actuators. Give a short justification for each property
Here is what i thinks that the answers of above questions might be this. Can you guyz correct me if i answered wrong at any point.

Can you clarify what is "PEAS" supposed to mean?

PEAS stands for (Performance, Environment, Actuators and Sensors), when you are asked to give the peas of a AI then you should describe it as follows:
Example:
PEAS for refinery controller:
•   Performance measure: maximize purity, yield, safety
•   Environment: refinery, operators
•   Actuators: valves, pumps, heaters, displays
•   Sensors: temperature, pressure, chemical sensors
Please refer to the following link where I found precise information:
http://www.cs.bilkent.edu.tr/~duygulu/Courses/CS461/Notes/Agents.pdf


text:
How are neural networks built in practice?
I am curious to know how neural networks are built in practice.
Are they hand coded using weight matrices, activation functions etc OR are there ways to build the NN by mentioning the number of layers, number of neurons in each layer, activation to be used, etc as parameters?
Similar question on training, once built is there a ‘fit’ method or does the training need to be hand coded?
Any reference for understanding these basics will be of great help.

You know about a "fit" method and you're asking how neural networks are implemented in practice? This is strange because it seems like you are aware of common libraries to build NNs, but, at the same time, you're asking this very basic question. It's not clear to me the purpose of this question then.

A "software library" is a codebase that includes many commonly used functions/algorithms, you do not need to write the same function again if you "import" the library.
In practice, practitioners/researchers don't often code the nitty gritty details of well-used algorithms, they just use existing libraries such as sklearn or TensorFlow that already has an existing implementation.
In the case of sklearn, ignoring some details, you can create a muti-layer perceptron in this line of code:
>>> clf = MLPClassifier(activation = 'relu', alpha=1e-5, hidden_layer_sizes((5,2))

...in which clf is an "object" that refers to the neural network, MLPClassifier is a function provided by sklearn that "creates" a neural net depending on the parameters provided.
In this case, the parameters provided are that the activation function is ReLU, where the network has two layers, with the first having 5 nodes, and the second has 2 nodes.
All other parameters that are not explicitly provided are set to a default value, in this case, the learning rate, among others.
It should be noted that this is not complete yet, we still have not completed training. Luckily, this is as simple as:
>>> clf.fit(X, y)

Where X and y are some array-shaped data/target set.
I recommend sklearn as i think its more beginner friendly.
https://scikit-learn.org/stable/getting_started.html


text:
How to train the images of various sizes?
I am practicing with an image dataset which is having different dimensions.
If I simply crop and pad them to 1024X1024(the original images having smallest width is around 300 and largest is around 2400 and widths and heights of the images are not the same) I am not getting good val_accuracy. It's just giving 49% accuracy.
How to do image processing to these images because the brightness of the images is also changing. My task is to classify them into 5 classes.

why don‘t you simply rescale the original image to some value, e.g. 300x300. Did you check for this?

trained with images making them to size of 358X358(smallest size is having 358 as the width) and got the val_accuracy around 57%.

Why do not simply perform some bilinear or bicubic interpolation?
Tensorflow and PyTorch deep learning frameworks have dedicated function to do this - https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.Resize and https://www.tensorflow.org/api_docs/python/tf/image/resize .

You can rescale images to same size based on the classification model you are using (preferably 300x300). Also for preprocessing you can try some morphological operations and some brightness removal techniques from OpenCV-Python. One more factor that could have affected your accuracy might be the number of images, if you are having less number of images you can augment your dataset to come up with enough samples for training.

I would use the ImageDataGenerator.flow_from_directory. Documentation is here.  Make a train directory. Within it create 5 subdirectories one for each class and give them the desired class name. Place the associated images into the 5 class sub directories. Then use something like the code below. Set the image size to something standard particularly is you are using transfer learning. For MobileNet standard size is 224 X 224. You can use the Image Data Generator to augment your data for example set horizontal flip =True. You can also vary the brightness and rotation of the images.
tf.keras.preprocessing.image.ImageDataGenerator(
    featurewise_center=False,
    samplewise_center=False,
    featurewise_std_normalization=False,
    samplewise_std_normalization=False,
    zca_whitening=False,
    zca_epsilon=1e-06,
    rotation_range=0,
    width_shift_range=0.0,
    height_shift_range=0.0,
    brightness_range=None,
    shear_range=0.0,
    zoom_range=0.0,
    channel_shift_range=0.0,
    fill_mode="nearest",
    cval=0.0,
    horizontal_flip=True,
    vertical_flip=False,
    rescale=None,
    preprocessing_function=None,
    data_format=None,
    validation_split=0.0,
    dtype=None)
#for flow from directory
.flow_from_directory(
    directory,
    target_size=(256, 256),
    color_mode="rgb",
    classes=None,
    class_mode="categorical",
    batch_size=32,
    shuffle=True,
    seed=None,
    save_to_dir=None,
    save_prefix="",
    save_format="png",
    follow_links=False,
    subset=None,
    interpolation="nearest",
)

authors: عبدالله احمد


text:
Choosing an AI method to recreate a given binary 2D image
If the title wan not very clear, I want a method to take an input image like this,
[[0, 0, 0, 0],
 [1, 1, 1, 0],
 [1, 1, 1, 0],
 [0, 1, 1, 0]]

and output the 2D coordinates of the 1s of the image (So that I can recreate the image)
The application is a robot creating the image using some kind of building blocks, placing one block after the other
I want the output to be sequential because I need to reconstruct the input image pixel by pixel and there are some conditions on the image construction order (e.g. You cannot place a 1 somewhere when it is surrounded by 1s)
The image can change and the number of 1s in the image too.

What is an appropriate AI method to apply in this case?
How should I feed the image to the network? (Will flattening it to 2D affect my need of an output order?)
Should I get the output coordinates one by one or as an ordered 2xN matrix?
If one by one, should I feed the same image for each output or the image without the 1s already filled?

I have tried to apply "NeuroEvolution of Augmenting Topologies" for this using neat-python but was unsuccessful. I am currently looking at RNNs but I am not sure if it is the best choice either.

https://cs.stackexchange.com/q/129468/755

This sounds like a problem that's best solved with a simple non-AI algorithm. If you just enumerate the coordinates in a regular order (rows, colums, zig-zag, hilbert curve) and emit the coordinates where the image has a '1' you're meeting your requirements. Is there any specific reason you want to use an AI algorithm which is most likely worse than this?

Yes the algorithmic solution is pretty simple. But I need to extend this problem further and use multiple agents to recreate the image with coordination. This is my first step and I want to identify a good approach for the extended problem. And in that, the order of placing ones on the image matters (The application is robots creating the image using some kind of building blocks, placing one block after the other)

Multiple agents can still execute a centrally generated plan, no need to involve AI. If recreating the original image faithfully is the actual goal, emergent swarm behavior is probably not suitable.


text:
How is weighted average computed in Deep Q networks
I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns
I.e
New estimate= old estimate +alpha*[returns- old estimate]
So by the law of large numbers this will converge to the optimal true q value
Now when we go to Deep Q networks,how exactly is the weighted average computed, all they simply did was try to reduce the error between the target and the estimate, and keep in mind this isn’t the true target, it’s just an unbiased estimate,since it’s an unbiased estimate how is the weighted average computed , which is the expectation?
Can someone help me out here??
Thanks in advance

Let's say $Q$ is the old estimate, $Q'$ the new estimate, and $R$ is the return.
We have
$$
Q' = Q + \alpha(R-Q)
$$
This can be rewritten as
$$
Q' = (1-\alpha)Q + \alpha R
$$
When $\alpha$ is a constant, this is an exponential weighted average of returns. If $n$ is the number of samples we get and $\alpha=1/n$ (so it decreases with each sample), we get
$$
Q' = \frac{n-1}{n}Q + \frac{1}{n}R
$$
This simply represents the average return. So, playing with $\alpha$ tunes the weighting of the estimate.

You’re completely missing me, I understand the weighted average in TD learning for normal Q learning, my question is how is this computed with deep Q networks ???

Firstly, please don't leave me comments to check your comments.

In deep Q learning, you do function approximation. You make estimates of the target returns and current Q values from samples (which is an unbiased estimator of the mean). Then you simply use SGD to minimize the MSE between your current state estimate and your target. As you mentioned, this is in fact an approximation. In general there are no convergence guarantees in deep RL. However, in the limit of infinite data, you can suppose your target network will converge to the value function (assuming the network is wide enough).

I still don’t get it,I know the Q values are unbiased estimates of the optimal values, the expectation which is the mean is the weighted average, which is the update formula in normal Q learning, in Deep Q learning, how is that weighted average implemented?,that’s my question , do you understand ?

What I mean exactly is, in Q learning the update rule implements the Weighted average which is the expectation,  in deep q learning how exactly is the weighted average implemented

I see. I believe the answer to this is that it isn't in fact implemented in deep Q learning. In the weighted average equation you mentioned, it's simple to implement with tabular methods because you update the Q values state/action-wise. With uncountable state space, you can't really do this. Instead you make updates in parameter space, so you're never directly updating particular state/action values.

Wow, so if that’s the case, how are the expected values computed in Deep Q learning , because expectation is basically the mean right, so in Deep Q learning, we’re using a target which is the unbiased estimate of the true target Value, so how is the expectation computed in DQNs

What I mean exactly is the target we use is the unbiased estimate of the actual target, meaning the expectation of the target gives the true target value, so my question is in DQNs how is that expectation computed, in normal Q leaning it’s the weighted average, in DQNs how is that computed

In DQN it's computed so as to minimize the expected MSE.


text:
What's the nearest neighbor algorithm used for upsampling?

Additionally, by default, the UpSampling2D layer will use a nearest neighbor algorithm to fill in the new rows and columns. This has the effect of simply doubling rows and columns, as described and is specified by the ‘interpolation‘ argument set to ‘nearest‘. Alternately, a bilinear interpolation method can be used which draws upon multiple surrounding points. This can be specified via setting the ‘interpolation‘ argument to ‘bilinear‘.

How exactly does the nearest neighbor algorithm mentioned above work? Also, what does interpolation mean in this context (nearest and bilinear)?
Source: Section on Upsampling2D layer

Interpolation typically means that you compute the value for a position/unit $x$ based on its "surrounding" positions/units. I would need to read that article to give a formal answer.


text:
Human intuition behind SVD in case of recommendation system
This does not answer my question. I struggled very hard to understand the SVD from a linear-algebra point of view. But in some cases I failed to connect the dots. So, I started to see all the application of SVD. Like movie recommendation system, Google page ranking system, etc.
Now in the case of movie recommendation system, what I had as a mental picture is...
The SVD is a technique that falls under collaborative filtering. And what the SVD does is factor a big data matrix into two smaller matrix. And as an input to the SVD we give an incomplete data matrix. And SVD gives us a probable complete data matrix. Here, in the case of a movie recommendation system we try to predict ratings of users. Incomplete input data matrix means some users didn't give ratings to certain movies. So the SVD will help to predict users' ratings. I still don't know how the SVD breaks down a large matrix to smaller pieces. I don't how the SVD determines the dimensions of the smaller matrices.
It would be helpful if anyone could judge my understanding. And I will very much appreciate any resources which can help me to understand the SVD from scratch to its application to Netflix recommendation systems. Also for the Google Page ranking system or for other applications.
I am looking forward to seeing an explanation more from human-intuition level and from a linear-algebra point of view. Because I am interested in using this algorithm in my research, I need to understand as soon as possible: how does the SVD work deep down from the core?


text:
How homographs is an NLP task can be treated?

A homograph - is a word that shares  the same written form as another word but has a different meaning.

They can be even different parts of speech. For example:

close(verb) - close(adverb)
lead(verb) - lead(noun)
wind(noun) - wind(verb)

And there is rather a big list https://en.wikipedia.org/wiki/List_of_English_homographs.
As far as I understand, after processing the text data in any conventional way, lemmatization, building an embedding, these words, despite having different meaning, and appearing in different contexts, would be absolutely the same for the algorithm, and in the end we would get some averaged context between two or more meainings of the word. And this embedding would be meaningless.
How is this problem treated or these words are regarded to be too rare to have a significant impact on the quality of resulting embedding?
I would appreciate comments and references to the papers or sources

I have exactly the same question and wonder if after these 7 months with no replies the original questioner has found some reference elsewhere. Thanks.

Actually, contextual embeddings - https://stackoverflow.com/questions/62272056/what-are-the-differences-between-contextual-embedding-and-word-embedding, https://arxiv.org/pdf/2003.07278.pdf - take each token not by itself alone, but in the surrounding context. These words are likely appear in surrounding of rather different words - so the resulting embedding for them would be different.


text:
Why isn't medical imaging improving faster with AI?
Researcher here. I just read this piece about medical imaging ai with object recognition and it left me wondering why there are still 100,000+ deaths a year in the US due to misdiagnosis - anyone out there working on these problems? Vinod Khosla famously said that he'd rather get surgery from AI than from a human - so where are we at with that?

"so where are we at with that?" That's a very, very broad question. I also don't think it's just limited by AI, but by other high-precision mechatronics not being economically viable at this time.

Welcome to SE:AI.  "Improving faster" is subjective--this question might be better served with historical stats on diagnoses errors.  (Short answer--tech takes time to develop & implement, and prove itself in the field.  Also, new tech tends to be very expensive initially, especially in the medical sector.  We've only had strong-narrow statistical AI for about 5 years now.)

AI in healthcare is already playing a big role in diagnosing the diseases, assisting patients or helping the medical staff in supplying various things or performing actions.
The medical imaging helps AI to diagnosis the diseases without help radiologist. And to develop the AI model using medical images to diagnosis various disease need training datasets for machine learning algorithm helping to detect with accuracy.
But just like AI, medical imaging is not improving faster making sometimes difficult or more time taking to diagnosis the diseases. Actually, while developing such highly sensitive models, high-quality training data is required and there is lack of such annotated data to train the models.
The more training data is feed into the model, it will learn the detection process with more accuracy resulting predicting the faster diagnosis. And to improve the quality and quantity of data, more dedicated and fast labeling process is required.
Hence, data annotation companies now use the AI-assisted labeling process to annotate the medical images at faster speed with better accuracy.  Compare to manual annotation, AI companies can get multiple times faster-annotated data for machine learning algorithms.
Once the training data will start available in large quantity for medical imaging analysis process through AI will also improve. The AI-assisted data annotation process can only help to produce the medical imaging datasets in large quantity for better predictions in healthcare sector.


text:
How to combine specific CNN models that work better at slightly different tasks?
I'm not sure how to describe this in the most accurate way but I'll give it a shot.
I've developed a Inception-Resnet V2 model for detecting audio signals via spectrogram.  It does a pretty good job but is not exactly the way I'd like it to be.
Some details:  I use 5 sets of data to evaluate my model during training.  They are all similar but slightly different.  Once I get to a certain threshold of F1 Scores for each training set I stop training.  My overall threshold is pretty hard to get to.  Every time training develops a model that produces a "best yet" of one of these data sets I save the model.
What I've noticed is that, during training, some round will produce a high F1 Score for one particular set while the other sets languish as mediocre.  Then, several dozen rounds later, another data set will peak while the others are mediocre.  Overall the entire model gets better but there are always some models that work better for some data sets.
What I would like to know is, given I might have 5 different models that each work better for a particular subset of data, is there a way that I can combine these models (either as a whole or better yet their particular layers) to produce a single model that works the best for all my data validation subsets?
Thank you.  Mecho

What you're describing are "Ensemble Models" -- where multiple models are trained in parallel, and then combined at inference time to squeeze out better performance.
This article gives a decent overview: https://towardsdatascience.com/ensemble-models-5a62d4f4cb0c

A single algorithm may not make the perfect prediction for a given dataset. Machine learning algorithms have their limitations and producing a model with high accuracy is challenging. If we build and combine multiple models, the overall accuracy could get boosted.

And they're also covered in Stanford's 2017 deep learning computer vision course, lecture 7: https://youtu.be/_JB0AO7QxSA?t=3098.

Rather than having just one model, we’ll train 10 different models independently from different initial random restart. At test time, we’ll run our data through all of the 10 models and average the predictions.

I recommend you check out this course, since it also delves quite a bit into convolutional networks too.


text:
How classification neural nets are different from simple dimension reduction + clustering?
I know the training of neural nets involves some sort of dimension manipulation to separate classes of different features.
If there is no variation of features, no matter for neural nets or simple dimension reduction methods (e.g. PCA, LDA) + clustering, neither of them are going to distinguish different classes.
In such sense, I would like to know the true power of neural nets:
How classification neural nets are different from simple dimension reduction + clustering?
or rephrase the question:
What value do neural nets add to solving classification problems in terms of its algorithmic architecture compared with simple dimension reduction + clustering?

PCA is linear, NN are nonlinear, more generally it's a universal function approximator.
That said basic NN are not terribly usefull, the real value of NN is for structured data which is obvious for us but hard to describe analytically. Basically NN architecture has been designed to learn this structure using operations like convolution and maxpooling.
There are still problems of course, and in some ways I think clustering algorithms still have value for things like anomaly detection.

authors: محمد علي عبدالله العسيلي