text:
Genetic Algorithm Python Snake not improving
So, i have created Snake game using Pygame and Python. Then i wanted to create an AI with Genetic algorithm and a simple NN to play it. Seems pretty fun, but things aren't working out.
This is my genetic algorithm:
def calculate_fitness(population):
    """Calculate the fitness value for the entire population of the generation."""
    # First we create all_fit, an empty array, at the start. Then we proceed to start the chromosome x and we will
    # calculate his fit_value. Then we will insert, inside the all_fit array, all the fit_values for each chromosome
    # of the population and return the array
    all_fit = []
    for i in range(len(population)):
        fit_value = Fitness().fitness(population[i])
        all_fit.append(fit_value)
    return all_fit

def select_best_individuals(population, fitness):
    """Select X number of best parents based on their fitness score."""
    # Create an empty array of the size of number_parents_crossover and the shape of the weights
    # after that we need to create an array with x number of the best parents, where x is NUMBER_PARENTS_CROSSOVER
    # inside config file. Then we search for the fittest parents inside the fitness array created by the
    # calculate_fitness function. Numpy.where return (array([], dtype=int64),) that satisfy the query, so we
    # take only the first element of the array and then it's value (the index inside fitness array). After we have
    # the index of the element we just need to take all the weights of that chromosome and insert them as a new
    # parent. Finally we change the fitness value of the fitness value of that chromosome inside the fitness
    # array in order to have all different parents and not only the fittest
    parents = numpy.empty((config.NUMBER_PARENTS_CROSSOVER, population.shape[1]))
    for parent_num in range(config.NUMBER_PARENTS_CROSSOVER):
        index_fittest = numpy.where(fitness == numpy.max(fitness))
        index_fittest = index_fittest[0][0]
        parents[parent_num, :] = population[index_fittest, :]
        fitness[index_fittest] = -99999
    return parents

def crossover(parents, offspring_size):
    """Create a crossover of the best parents."""
    # First we start by creating and empty array with the size equal to offspring_size we want. The type of the
    # array is [ [Index, Weights[]] ]. If the parents size is only 1 than we can't make crossover and we return
    # the parent itself, otherwise we select 2 random parents and then mix their weights based on a probability
    offspring = numpy.empty(offspring_size)
    if parents.shape[0] == 1:
        offspring = parents
    else:
        for offspring_index in range(offspring_size[0]):
            while True:
                index_parent_1 = random.randint(0, parents.shape[0] - 1)
                index_parent_2 = random.randint(0, parents.shape[0] - 1)
                if index_parent_1 != index_parent_2:
                    for weight_index in range(offspring_size[1]):
                        if random.uniform(0, 1) < 0.5:
                            offspring[offspring_index, weight_index] = parents[index_parent_1, weight_index]
                        else:
                            offspring[offspring_index, weight_index] = parents[index_parent_2, weight_index]
                    break
    return offspring

def mutation(offspring_crossover):
    """Mutating the offsprings generated from crossover to maintain variation in the population."""
    # We cycle though the offspring_crossover population and we change x random weights, where x is a parameter
    # inside the config file. We select a random index, generate a random value between -1 and 1 and then
    # we sum the original weight with the random_value, so that we have a variation inside the population
    for offspring_index in range(offspring_crossover.shape[0]):
        for _ in range(offspring_crossover.shape[1]):
            if random.uniform(0, 1) == config.MUTATION_PERCENTAGE:
                index = random.randint(0, offspring_crossover.shape[1] - 1)
                random_value = numpy.random.choice(numpy.arange(-1, 1, step=0.001), size=1, replace=False)
                offspring_crossover[offspring_index, index] = offspring_crossover[offspring_index, index] + random_value
    return offspring_crossover

My neural network is formed using 7 inputs:
is_left_blocked, is_front_blocked, is_right_blocked, apple_direction_vector_normalized_x,
snake_direction_vector_normalized_x, apple_direction_vector_normalized_y,snake_direction_vector_normalized_y

Basically if you can go left, front, right, direction to the apple and snake direction.
Then i have an hidden layer with 8 neurons and finally 3 output that indicate left, keep going or right.
The Neural Network forward() is calculate like this:
self.get_weights_from_encoded()
Z1 = numpy.matmul(self.__W1, self.__input_values.T)
A1 = numpy.tanh(Z1)
Z2 = numpy.matmul(self.__W2, A1)
A2 = self.sigmoid(Z2)
A2 = self.softmax(A2)
return A2

where self.__W1 and self.__W2 are the weights from input to hidden layer and then the weights from hidden layer to the output. Softmax(A2) return the index of the matrix[1,3] where the value is the biggest, then i use that index to indicate the direction that my neural network choose.
This is the config file that contains the parameters:
# GENETIC ALGORITHM
NUMBER_OF_POPULATION = 500
NUMBER_OF_GENERATION = 200
NUMBER_PARENTS_CROSSOVER = 50
MUTATION_PERCENTAGE = 0.2

# NEURAL NETWORK
INPUT = 7
NEURONS_HIDDEN_1 = 8
OUTPUT = 3
NUMBER_WEIGHTS = INPUT * NEURONS_HIDDEN_1 + NEURONS_HIDDEN_1 * OUTPUT

And this is the main:
for generation in range(config.NUMBER_OF_GENERATION):

    snakes_fitness = genetic_algorithm.calculate_fitness(population)

    # Selecting the best parents in the population.
    parents = genetic_algorithm.select_best_individuals(population, snakes_fitness)

    # Generating next generation using crossover.
    offspring_crossover = genetic_algorithm.crossover(parents,
                                                      offspring_size=(pop_size[0] - parents.shape[0], config.NUMBER_WEIGHTS))

    # Adding some variations to the offspring using mutation.
    offspring_mutation = genetic_algorithm.mutation(offspring_crossover)

    # Creating the new population based on the parents and offspring.
    population[0:parents.shape[0], :] = parents
    population[parents.shape[0]:, :] = offspring_mutation

I have 2 problems:
1) I don't see an improvement over the new generations
2) I'm actually running the game inside the for loop, but waiting for all the snake of a generation to die and repeat with the new one is really time consuming. Isn't there a way to launch all or, atleast, more than 1 instance of the game and keep filling the array with the result?
This is Fitness().fitness(population[i])
def fitness(self, weights):
    game_manager = GameManager(weights)
    self.__score = game_manager.play_game()
    return self.__score

This is where it's called inside the for loop
def calculate_fitness(population):
    """Calculate the fitness value for the entire population of the generation."""
    # First we create all_fit, an empty array, at the start. Then we proceed to start the chromosome x and we will
    # calculate his fit_value. Then we will insert, inside the all_fit array, all the fit_values for each chromosome
    # of the population and return the array
    all_fit = []
    for i in range(len(population)):
        fit_value = Fitness().fitness(population[i])
        all_fit.append(fit_value)
    return all_fit

This the function that launch the game (GameManager(weights)) and return the score of the snake.
This is my first time on AI so this code could be all a mess, don't worry about pointing out what i did wrong, just please don't say "It's all wrong" because i won't be able to learn otherwise.

Perhaps it could be due to the lack of inputs? Many videos of Snake AIs that use NN and GA to learn seem to have more than double or triple the amount of inputs you're feeding to your neural network (See here and here). I would recommend that you add more inputs by giving the NN the distance to the snake part and the wall for every direction you're looking in.


text:
What does the notation $\mathcal{N}(z; \mu, \sigma)$ stand for in statistics?
I know that the notation $\mathcal{N}(\mu, \sigma)$ stands for a normal distribution.
But I'm reading the book "An Introduction to Variational Autoencoders" and in it, there is this notation:
$$\mathcal{N}(z; 0, I)$$
What does it mean?
picture of the book:

Comments are not for extended discussion; this conversation has been moved to chat.

It means that $z$ has a (multivariate) normal distribution with 0 mean and identity covariance matrix. This essentially means each individual element of the vector $z$ has a standard normal distribution.

Comments are not for extended discussion; this conversation has been moved to chat.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What type of model should I fit to increase accuracy?
Currently, I'm working on 6-axis IMU(Inertial Measurment Unit) dataset. This dataset contain 6 axis IMU data of 7 different drivers. The Imu sensor attached on vehicle. The drivers drives same path. So, the dataset include 6 feature columns and a label column.
I tried multiple neural network models.The sensor data is a sequential data so I tried LSTM(Long Short Term Memory) & classical fully-connected layers.
Some of my architecture(in keras framework):

Layer (type)                 Output Shape              Param #

lstm_4 (LSTM)                (None, 1, 128)            69120
_________________________________________________________________
lstm_5 (LSTM)                (None, 1, 64)             49408
_________________________________________________________________
lstm_6 (LSTM)                (None, 1, 32)             12416
_________________________________________________________________
dense_8 (Dense)              (None, 1, 64)             2112
_________________________________________________________________
dropout_2 (Dropout)          (None, 1, 64)             0
_________________________________________________________________
dense_9 (Dense)              (None, 1, 7)              455

2nd Architecture:

=================================================================
dense_10 (Dense)             (None, 32)                224
_________________________________________________________________
dense_11 (Dense)             (None, 64)                2112
_________________________________________________________________
dense_12 (Dense)             (None, 128)               8320
_________________________________________________________________
dense_13 (Dense)             (None, 256)               33024
_________________________________________________________________
dropout_3 (Dropout)          (None, 256)               0
_________________________________________________________________
dense_14 (Dense)             (None, 512)               131584
_________________________________________________________________
dense_15 (Dense)             (None, 256)               131328
_________________________________________________________________
dense_16 (Dense)             (None, 128)               32896
_________________________________________________________________
dense_17 (Dense)             (None, 64)                8256
_________________________________________________________________
dropout_4 (Dropout)          (None, 64)                0
_________________________________________________________________
dense_18 (Dense)             (None, 128)               8320
_________________________________________________________________
dense_19 (Dense)             (None, 7)                 903

The best accuracy in my models was %70 which is not good. How style of layers should I use to handle this data? Or, which type of model would increase accuracy?

after reading some literature in the area I'd recommend the following:

Try using Convolution Neural Networks (CNNs) this paper outlines some really good points on why should you use CNNs.
Try a Combination of the different layers in the same model. Start with some Convolution Layers, then some LSTMs and then a couple of Dense Layers followed by Dropout

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is there a logical method of deducing an optimal batch size when training a Deep Q-learning agent with experience replay?
I am training an RL agent using Deep-Q learning with experience replay. At each frame, I am currently sampling 32 random transitions from a queue which stores a maximum of 20000 and training as described in the Atari with Deep RL paper. All is working fine, but I was wondering whether there is any logical way to select the proper batch size for training, or if simply using a grid search is best. At the moment, I’m simply using 32, for its small enough that I can render the gameplay throughout training at a stunning rate of 0.5fps. However, I’m wondering how much of an effect batch size has, and if there is any criteria we could generalize across all Deep Q-learning tasks.

There is no special calculation you can do to determine the optimal batch size for any situation, so you kinda have to do a bit of testing to determine what batch size will work best. But there are some common trends you can take into account to make your testing easier.

How to choose your batch size
According to the paper Accelerated Methods for Deep Reinforcement Learning you get the best performance from DQNs (on average) with a batch size of 512. The problem with this is that is is much slower than the usual batch size of 32 and most of the time the performance improvement doesn't warrant it.

If you are just trying to test out your agents it is generally best to stick with a batch size of 32 or 64 so that you can train the agent quickly yet still get an idea of what it is capable of. But if getting the best performance is your top priority and waiting longer isn't a problem, then you should go for a batch size of 512 (higher can actually lead to worse performance) or something near that.

In my experience, exploration is always the challenge with RL, and I think about batch size a way of increasing exploration.
What happens during training of an RL agent, is the policies entropy often reduces as the agent receives rewards, leading to reduced exploration.  If the rewards in the environment are sparse, the agent can sometimes only discover the first reward, and it can take time to discover other rewards.. as it just beelines to the reward it knows about, and doesn't explore enough.
Increasing batch size, (assuming your not using a single environment, but many parallel environments) has the effect of ensuring that the agent can get multiple reward signals from more diverse state transitions... allowing a more accurate value function and therefore a more complex policy that explores more.   The challenge is that finding the correct learning rate can be tricky, as well as the values of other hyperparameters.
Overall, I would say for something like Atari, which is a very high dimensional state space, aim for the largest batch size you can get the agent to learn at.
For reference, I'm running a batch size of 4096 a gridworld maze exploration task (using PPO), and I'm getting very good results at that size.  I would go larger but I don't have the memory to do so, however I had to run at least 40 hyperparameter sets to find the settings that worked at that batch size.
Finally, I would point out that advantage based algos are a lot more stable, as the values are implicitly normalized, which probably is why I'm able to reach such a big batch size with so few hyperparameter adjustments.  You may find DeepQ to be a lot harder to stabilize.


text:
Effect of adding an Independent Variable in Multiple Linear Regression
I am new in machine learning and learning linear regression concept. Please help with answers to below queries.
I want to understand effect on existing independent variable(X1) if I add a new independent variable(X2) in my model.
This new variable is highly correlated with dependent variable(Y)

Will it have any effect on beta coefficient of X1?
Will relationship between X1 and Y become insignificant?
Can adjusted R-square value decrease?

Yes, the addition of the independent variable ($X_2$) will have an impact on the beta coefficient of $X_1$.
Maybe, it depends on the relationship between $X_1$ and $Y$, and whether you are using any regularization or not.
If the independent variable $X_2$ is more correlated than $X_1$, then the value of $r^2$ should be higher (but I am not sure on this)


text:
Why does (not) the distribution of states depend on the policy parameters that induce it?
I came across the following proof of what's commonly referred to as the log-derivative trick in policy-gradient algorithms, and I have a question -

While transitioning from the first line to the second, the gradient with respect to policy parameters $\theta$ was pushed into the summation. What bothers me is how it skipped over $\mu (s)$, the distribution of states - which (the way I understand it), is induced by the policy $\pi_\theta$ itself! Why then does it not depend on $\theta$?
Let me know what's going wrong! Thank you!

The proof you are given in the above post is not wrong. It's just they skip some of the steps and directly written the final answer. Let me go through those steps:
I will simplify some of the things to avoid complication but the generosity remains the same. Like I will think of the reward as only dependent on the current state, $s$, and current action, $a$. So, $r = r(s,a)$
First, we will define the average reward as:
$$r(\pi) = \sum_s \mu(s)\sum_a \pi(a|s)\sum_{s^{\prime}} P_{ss'}^{a} r $$
We can further simplify average reward as:
$$r(\pi) = \sum_s \mu(s)\sum_a \pi(a|s)r(s,a) $$
My notation may be slightly different than the aforementioned slides since I'm only following Sutton's book on RL. Our objective function is:
$$ J(\theta) = r(\pi) $$
We want to prove that:
$$ \nabla_{\theta} J(\theta) = \nabla_{\theta}r(\pi) = \sum_s \mu(s) \sum_a \nabla_{\theta}\pi(a|s) Q(s,a)$$
Now let's start the proof:
$$\nabla_{\theta}V(s) = \nabla_{\theta} \sum_{a} \pi(a|s) Q(s,a)$$
$$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s)  + \pi(a|s) \nabla_{\theta}Q(s,a)]$$
$$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s)  + \pi(a|s) \nabla_{\theta}[R(s,a) - r(\pi) + \sum_{s^{\prime}}P_{ss^{\prime}}^{a}V(s^{\prime})]]$$
$$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s)  + \pi(a|s) [- \nabla_{\theta}r(\pi) + \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})]]$$
$$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s)  + \pi(a|s) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})] - \nabla_{\theta}r(\pi)\sum_{a}\pi(a|s)$$
Now we will rearrange this:
$$\nabla_{\theta}r(\pi) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s)  + \pi(a|s) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})] - \nabla_{\theta}V(s)$$
Multiplying both sides by $\mu(s)$ and summing over $s$:
$$\nabla_{\theta}r(\pi) \sum_{s}\mu(s)= \sum_{s}\mu(s) \sum_{a} Q(s,a) \nabla_{\theta} \pi(a|s)  + \sum_{s}\mu(s) \sum_a \pi(a|s) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime}) - \sum_{s}\mu(s) \nabla_{\theta}V(s)$$
$$\nabla_{\theta}r(\pi) = \sum_{s}\mu(s) \sum_{a} Q(s,a) \nabla_{\theta} \pi(a|s)  + \sum_{s^{\prime}}\mu(s^{\prime})\nabla_{\theta}V(s^{\prime}) - \sum_{s}\mu(s) \nabla_{\theta}V(s)$$
Now we are there:
$$\nabla_{\theta}r(\pi) = \sum_{s}\mu(s) \sum_{a} Q(s,a) \nabla_{\theta} \pi(a|s)$$
This is the policy gradient theoram for average reward formulation (ref. Policy gradient).

Do you mean $\pi (a|s)$ instead of $\pi (s|a)$? Correct me if I'm misinterpreting!

Also a little confused with the notation - why is $Q(s,a) = R(s,a) - r(\pi) + \sum_{s^{\prime}}P_{ss^{\prime}}^{a}V(s^{\prime})$ ?

@cogito_ai I followed Sutton's notation. So, denoted policy as $\pi(a|s)$. For your case, $\pi(a|s) = \pi(s|a)$. But I think this is a mistake in this slide, it should be $\pi(a|s)$.

I think you've denoted the policy by $\pi(s|a)$ in your answer in multiple places - or is that something else? The slide shows $a|s$

For average reward setting $Q(s,a) = r(s,a) - r(\pi) + \sum P_{ss'}v(s')$. See Sutton's book chapter on policy funciton approximation section average reward.

Sorry, it should be $\pi(a|s)$. It was my mistake.

How is $\sum_{s^{\prime}}\mu(s^{\prime})\nabla_{\theta}V(s^{\prime}) = \sum_{s}\mu(s) \sum_a \pi(s,a) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})$? Also which edition of Sutton's book are you using?

Because this is the product rule of probability. This is not in the Sutton's book. I give the reference, check it for more details.

If you open the summation you can see clearly this is exactly telling us what is the probability of the agent to go to the state $s'$ from the state $s$.

Looks good! Lastly, in places where you've written $\pi(a,s)$, did you mean $\pi(a|s)$ there too?

Yes, I make a mistake, it should be $\pi(a|s)$.

Your formulation ends with $Q(s,a)$, while the slide has $R(s,a)$ - why the difference? I understand the rest! It'd be great if you could elaborate on this!

I do not follow this lecture for RL, So, I can not tell it now, first I should watch Deep RL Bootcamp. But I hope, if you think about it you may find the reason.

Okay, I watched their lecture. At the time 40.55 they, replace $\sum r(s,a)$ with $Q(s,a)$.

$Q(s,a)$ is an unbiased estimate of $R(s,a)$ — ie they are the same in expectation. @cogito_ai

The reason you are confused is because this is not the full derivation of the Policy Gradient Theorem. You are correct in thinking that $\mu(s)$ depends on the policy $\pi$ which in turn depends on the policy parameters $\theta$, and so there should be a derivative of $\mu$ wrt $\theta$, however the Policy Gradient Theorem doesn't require you to take this derivative.
In fact, the great thing about the Policy Gradient Theorem is that the final result does not require you to take a derivative of the state distribution with respect to the policy parameters. I would encourage you to read and go through the derivation of the Policy Gradient Theorem from e.g. Sutton and Barto to see why you don't need to take the derivative.

Above is an image of the Policy Gradient Theorem proof from the Sutton and Barto book. If you carefully go through this line by line you will see that you are not required to take a derivative of the state distribution anywhere in the proof.

Could you please elaborate and possibly provide mathematical details? I've gone through the derivation in Sutton and Barto and I'm still unclear why the derivation above works.

There’s not much to elaborate on. If you followed the Sutton and Barto proof you would see that you don’t need to take a derivative of the state distribution. I can’t really explain any clearer than just writing out the full proof which I am not going to do since that is not the point of this site. If you have a particular question about the proof that you find unclear then you should post that and I’d be happy to answer it.

Also, the above isn't a derivation. It is an extremely poor example, especially if they have not previously gone through the policy gradient theorem proof. What they have written (first two lines) gets the right answer but goes about it the wrong way.

I understand the proof written in Sutton and Barto's text. My question was - why does the wrong proof (in my post) work? I apologize if you've already explained this, but I still don't see why the wrong proof works. Sutton and Barto's proof doesn't require us to take the derivative of the state distribution anywhere, but that is no reason for me to overlook the same in the proof I've attached in my post. Please explain. Thanks!

What you have posted is not a proof. They have started with the final result of the Policy Gradient Theorem, and then they have derived the update that is used in REINFORCE. As I say, the source you are using does not look like something I would recommend. The equality on the first line should be replaced directly with the equality on the second line to avoid confusion. Regardless, if you want good source material for Reinforcement Learning then I would recommend you read Sutton and Barto to prepare you for more advanced topics you will find in papers.

@cogito_ai to answer your edit, what you have posted is not a proof of the policy gradient theorem. The author has started with the final result in the policy gradient theorem, and has derived something entirely separate.


text:
Can I use one-hot vectors for text classification?
For an upcoming project I'm trying to write a text classifier for the IMDb sentiment analysis dataset. This needs to vectorize words using an embedding layer and then reduce the dimensions of the output with global average pooling. This is proving however to be very difficult for my low experience level, and I am struggling to wrap my head around the dimensionality involved, bearing in mind I must avoid libraries such as tensorflow that would make it very basic exercise. I am hoping that I could make it easier by encoding each word in the reviews as a one-hot vector, and passing it through a few regular dense layers. Would this work and yield decent results?

One hot encoding is a good strategy to apply with categorical variables that assume few possible values. The problem with text data is that you easily end up with corpora with a really large vocabulary. If I remember correctly the IMDb dataset contains around 130.000 unique words, which means that you should create a network with an input matrix of size 130.000 x max_length where max_length is the fixed maximum length allowed for each review. Apart from the huge size, this matrix would also be extremely sparse, and that's another big issue in using one-hot encoding with text.
For these reasons, I really doubt you would achieve any good results with a simple one-hot encoding. Embeddings where actually designed precisely to overcome all these issues, they have fixed reasonable size, they assume continue values between 0 and 1, which is desirable for deep neural networks, and they can be treated as "extra" trainable weights of a network.
If you really want to avoid embeddings I would suggest you to use (or implement, I don't think it will be so hard) a term frequency–inverse document frequency vectoriser. It is closer to one-hot encoding in the fact that it is based on the creation of a huge co-occurances matrix between words, but at least the values are continuous and not dichotomous. Nevertheless I would not expect high performances with the tf-idf either, simply because this type of encoding works best with shallow models like the Naive Bayes rather than deep models.


text:
Why is sampling non-uniformly from the replay memory an issue? (Prioritized experience replay)
I can't seem to understand why we need importance sampling in prioritized experience replay (PER). The authors of the paper write on page 5:

The estimation of the expected value with stochastic updates relies on those updates corresponding to the same distribution as its expectation. Prioritized replay introduces bias because it changes this distribution in an uncontrolled fashion, and therefore changes the solution that the estimates will converge to (even if the policy and state distribution are fixed).

My understanding of this statement is that sampling non-uniformly from the replay memory is an issue.
So, my question is: Since we are working 1-step off-policy, why is it an issue? I thought that in an off-policy setting we don't care how transitions are sampled (at least in the 1-step case).
The one possibility for an issue that came to my mind is that in the particular case of PER, we are sampling transitions according to the errors and rewards, which does seem a little fishy.
A somewhat related question was asked here, but I don't think it answers my question.

The problem is not that we need importance sampling because the learning is off-policy -- you are correct in that for one step off-policy algorithms such as $Q$-learning we don't need importance sampling, see e.g. here for an explanation why. The reason we need the importance sampling is due to the loss used to train the network.
In the original DQN paper, the loss is defined as
$$L_i(\theta_a) = \mathbb{E}_{(s,a,r,s') \sim \mbox{U}(D)} \left[ \left( r + \gamma \max_{a'} Q(s',a' ; \theta_i^-) - Q(s,a;\theta_i) \right)^2 \right ]\;.$$
You can see here the expectation over the loss is taken according to a uniform distribution over the replayed buffer $D$. If we started randomly sampling non-uniformly, as is the case in PER, then the expectation wouldn't be satisfied and would introduce bias. Importance sampling is used to correct this bias. Note that the main reason that we sample uniformly at random is that gradient descent usually assumes that the data is i.i.d. You can argue that in an RL problem the data will never be fully i.i.d, but you can probably tell that sampling uniformly at random from a large buffer of experience is 'closer' to being i.i.d, whereas using a priority without importance sampling will likely lead to non-i.i.d. data (e.g. there is probably high correlation between experience that has high TD error, if the priority is chosen according to this).
Note that in the paper they mention that the bias isn't as much of an issue at the start of learning and hence they use a decaying $\beta$ that only makes the importance sampling weights the 'correct' weights to use at the end of learning - this means that the estimate of the loss is asymptotically unbiased.


text:
Which type of feature extractor do you suggest to classify sensor data?
I have IMU (Inertial Measurment Unit- 6 axis) sensor data. The sensor attached on a car and 7 different drivers wipe on same path. I want to extract features and classify drivers. Which type of feature extractor do you guys suggest? I am planning to use PCA and Autoencoders but what do you think about classical signal properties to classify drivers?

There could be multiple possible ways to extract the features. One would be to use RNNs for a temporal relationship as the input data is time-series.


text:
What's a good neural network for this problem?
I am very new to the field of AI so please bear with me.
Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The input variables are numerous but not that many, maybe 7-10.
These input variables are certain formulae that involve calculations to do with wind, time, angle, momentum etc, and each formula returns which side it thinks the dice will like roll. Let's say that intuitively, by looking at these variables, I can make a very good guess at which side the dice lands on. If for example 6 out of 7 input variables say it likely that the dice will land on 1 but the 7th input suggests that it will land on 0, I would guess it lands on 1. As a human, I'm essentially consulting these inputs as a kind of "brains trust", and I act as a judge to make the final decision based on the brains trust. Of course in that example, my logic as a judge was simply majority rules, but what if some other more complicated non-linear method of judging was needed?
I essentially want my neural network to take this role as a judge. I have read that feedforward nns have limitations regarding control flow and loops, so I'm not sure if that structure will be appropriate. I'm not sure if recurrent nn will be appropriate either as I don't care what the previous inputs were.
Thanks

A simple feed-forward neural network with at least one hidden layer would suffice in your problem, and can deal with arbitrary non-linear relationships between input and output. If you expect relationships to be highly non-linear then additional layers might be required, but from your description of the problem, I would be surprised if you needed more than few layers, and a relatively small network.
However, I note that:

The input variables are numerous but not that many, maybe 7-10.

This gives you $3^{10} = 59049$ possible inputs. That's not much in terms of amount of data needed for ML statistical models. Assuming that even the best predictions are still probabilistic, then you may only need a million or so examples to create a reasonably accurate lookup table, not needing a neural network at all.
The strength of a neural network is to be able to generalise well from less examples than that. Of course, this is not perfect, but it would be able to do things such as notice if inputs 1,2 and 3 agree then that is always the most likely answer. If that turns out to be true (and not an accident of having low numbers of samples), then the NN could learn that useful pattern using far less data than a table-based approach.

I have read that feedforward nns have limitations regarding control flow and loops, so I'm not sure if that structure will be appropriate.

This is true, but does not impact your situation, because there is no control flow or loops involved. You have described a simple function. Whilst you or I might inspect the data and look backwards and forwards across it before coming to a decision, a neural network approximating a function does not need to do that, and in simple cases there is usually no benefit to doing so - a statistical summary of the correct mapping from input to output is more than sufficient and likely the best that can be done.

I'm not sure if recurrent nn will be appropriate either as I don't care what the previous inputs were.

As all your inputs represent the same kind of thing, you could implement as a RNN with a single input, -1, 0 or +1, always feeding in the predictions by type in the same order. It might resemble how you are thinking about the problem as a human (at least a better analogy than the direct statistical match in a feed-forward network), especially if you implemented a variant of attention. However, I don't think there would be any benefit to that in improved accuracy, and it would be a significant challenge to build that if you are new to AI.

Thanks for the answer. What if the inputs were fuzzy? Say instead of -1,0,1 it predicts more like -0.9, 0.1, 0.8 ? Does this change anything? What if the input values were higher than 1, would the nn normalize that automatically? Should I use relu?

@minusatwelfth If the inputs were fuzzy then you could not use a tabular approach as suggested in this answer, because there would be too many input variations. The NN would still work just fine though, and would treat e.g. -0.8 as similar to -0.7 (you probably want that). Neural networks do not automatically normalize inputs. An input range between -2 and +2 would be safe enough, up to you whether you want to clean them up before using. If the values could be really high you would want to pre-process before feeding to the NN, because NNs cope badly with large magnitude inputs in general.

Thanks, that's pretty much everyt-... I read somewhere that I should try the weights to be low numbers first (-1 to 1). Is this true? With the platform im working with, i cant afford to optimize for every single weight value as it would take too long to compute.

@minusatwelfth: The best numbers depend on size of layers. Most neural network libraries will use sane weight initialisation amounts. If you are implementing your own NN routines (using e.g. NumPy) then you will need to research what to use for initialisation, alongside how to do backpropagation etc. Otherwise if you are using a modern framework like Keras or PyTorch, the defaults will work fine for you

Ah I see. Sadly I'm not using any of those softwares you mentioned, I have to initialise the weights and the steps manually

@minusatwelfth: It is not possible to give you rules for initialisation in the space of a comment. I suggest look it up. Keywords to search "Glorot initialisation" or "Xavier initialisation". It's not too complicated. Easier than understanding backprop in any case


text:
Should image augmentation be applied before or after image resizing?
For the purposes of training a Convolutional Neural Network Classifier, should image augmentation be done before or after resizing the training images?
To reduce file size and speed up training time, developers often resize training images to a set height and width using something like PIL (Python Imaging Library).
If the images are augmented (to increase training set size), should it be done before or after resizing the members of the set?
For simplicity sake, it would probably be faster to augment the images after resizing, but I am wondering if any useful data is lost in this process. I assume it may depend on the method used to resize the images (cropping, scaling technique, etc.)

It is beneficial to check resize function built-in PIL here: https://pillow.readthedocs.io/en/stable/_modules/PIL/Image.html#Image.resize

As it is told in PIL documentation
It uses some filters to resize images.And those filters are explained here
uses mostly numerical methods as I see. So it is approximating the image data input. Which means you are right about data loss. But here might be the question would it change the data so much If it is done after augmentation or before?
Since in numerical methodic approaches the more values means more valid approximation in general.
So it might be beneficial to augment and then resize.
But one should look for its real answer. Mine is just a thought experiment. You will get better proven answers

Awesome, thanks! I'm going to look around for similar studies done, and perhaps perform my own.


text:
What are the rules behind vector product in gradient?
Let's suppose we have calculated the gradient and it came out to be $f(WX)(1-f(W X))X$, where $f()$ is the sigmoid function, $W$ of order $2\times2$ is the weight matrix, and $X$ is an input vector of order $2\times 1$. For ease let $f(WX)(1-f(W X))=\Bigg[
\begin{array}{c}
0.3 \\
0.8 \\
\end{array}\Bigg]$ and $X=\Bigg[
\begin{array}{c}
1 \\
0 \\
\end{array}\Bigg]$. When we multiply these vectors we will multiply them as $f(WX)(1-f(W X))\times X^T$ i.e $\Bigg[
\begin{array}{c}
0.3 \\
0.8 \\
\end{array}\Bigg]\times[1 \quad0]$. I do this because I know that we need this gradient to update a $2\times 2$ weight matrix, hence, the gradient should have size $2\times 2$. But, I don't know the law/rule behind this, if I was just given the values and had no knowledge that we need the solution to update the weight matrix, then, I might have done something like $[0.3 \quad 0.8]\times\Bigg[
\begin{array}{c}
1 \\
0 \\
\end{array}\Bigg]$ which will return a scalar. For a long chain of such operations (multiple derivatives in applying chain rule, resulting in many vectors), how do we know if the multiplication of two vectors should return a vector or matrix (dot or cross product)?

It helps to think of each output dimension separately.
You have $X$ which is an $(2 \times 1)$ vector, and $W_1$ is a $(1 \times 2 )$ vector. Their product is a scalar, of which we then take the sigmoid to get our output $Y_1$.
The gradient of this w.r.t. $W_1$ will be $f(WX) (1 - f(WX)) X^T$, which has the appropriate dimensions.
Then, you just stack these for all your outputs, giving you the shape you got.


text:
What are some suitable positive functions as activations of neural networks?
I am working on a deep Q-learning project. My project is different than normal deep Q-learning. The rewards of my neural network must be positive because I need their values to importance sample actions. I know that I can't use ReLU as the activation function of my neural network. So the only suitable functions which I know are sigmoid, softmax and exponential function. I tried working with sigmoid and softmax but they generate wrong results and the loss function diverges.
There are two terminal states in my model. Their rewards are 1 and 0. All other states don't have any immediate rewards.

First of all: an activation function is usually placed after a linear operation and you can have a lot of them (maybe different) in your nn. That's why it would be better to say $\bf a$ activation function of the neural net and not $\bf the$ activation function. So if you meant by activation function the last operation which is going to make your outputs non-negative, then you're right, ReLU isn't a good choice. Usually when people need to output some positive values, they take exponent as the last operation. I used it several times and everything was just fine.


text:
How do I label images for deep learning classification?
I have roughly 30,000 images of two categories, which are 'crops' and 'weeds.' An example of what I have can be found below:

The goal will use my training images to detect weeds among crops, given an orthomosaic GIS image of a given field. I guess you could say that I'm trying to detect certain objects in the field.
As I'm new to deep learning, how would one go about generating training labels for this task? Can I just label the entire photo as a 'weed' using some type of text file, or do I actually have to draw bounding boxes (around weeds) on each image that will be used for training? If so, is there an easier way than going through all 30,000 of my images?
I'm very new to this, so any specific details would really help a lot!

If each photo is intended to show a photo of weed or crops you should give one label. If your task is different where you also try to localize weed or crops in the image, then you need to label accordingly. My understanding is you are trying to do the first case, therefore, there should be one label for each image.

I am trying to pinpoint the location of the weeds in the field. Therefore, should I using something like the labelImg software to create Pascal VOC samples of the images?

I think so. The following link discusses about preprocessing training images for YOLO , you can follow the same. It will be really interesting to see how well your model works for this task. https://www.arunponnusamy.com/preparing-custom-dataset-for-training-yolo-object-detector.html

This is really a semantic segmentation problem if OP wants to pinpoint the weeds.
If OP wants to have such a segmentation he will need to hand segment every.single picture.

Maybe you should explicitly say that "semantic segmentation" is about classifying pixels (maybe provide an image that illustrates it) and consequently "segmenting" the objects in the image. Of course, labelling each pixel will be a lot of labor, so maybe you should also bring this up.


text:
Can a neural network be trained on a dataset containing only values for true output for a classification problem?
I am using a dataset from Google which contains 1,27,000 data points on simulated concentrations of the atmosphere of exoplanets which can sustain life. So, the output label of all these data points is 1 i.e, probability of life existing there is 1. If I train my neural network on this data, and test it on data points with concentrations other than these, can I expect to get probability values at the output? Asking because the model knows no false labelled value.

Yes, you can and the answer is One-Class Classification. A well-written resource to understand is this.

Thank you for your reply. I will check it out.


text:
What is the “Hello World” problem of Unsupervised Learning?
As a followup to this question, I'm interested in what the typical "Hello World" problem (first easy example problem) is for unsupervised learning.
A quick Google search didn't find any obvious answers for me.

Probably knmeans clustering or gaussian clustering

@FourierFlux Both seem to be solution methods, not problems. Do I understand correctly that problems would be: cluster $n$ training observations in a $d$-dimentsional space, then apply those clusters to new observations?

I disagree with the context that MNIST is the "hello world" of supervised learning. It is definitely, though, the "hello world" of image classification, which is a very specific sub-field of supervised learning.
I'd consider the Iris dataset a better candidate for the "hello world" of supervised learning, with other close candidates such as the Wine, Wisconsin breast cancer or Pima indians datasets. However, as an even simpler and more alternative choice, a lot of people prefer generating their own 2-dimensional datasets so that can more intuitively understand what the different algorithms are doing. An example of this is TensorFlow playground.
Equivalently, in unsupervised learning there are a lot of different tasks. I personally think that clustering is probably the task that is easier for people to understand and as such the most common intro to unsupervised learning. Here there are, as well, two options:

Using an already established dataset, e.g. Iris (without the labels).
Generating your own synthetic 2-dimensional data, to better understand how the algorithms work. An example is this.


text:
Research paths/areas for improving the performance of CNNs when faced with limited data
I've been reading through the research literature for image processing, computer vision, and convolutional neural networks. For image classification and object recognition, I know that convolutional neural networks deliver state-of-the-art performance when large amounts of data are available. Furthermore, I know that Hinton et al. created "capsule networks" to try and overcome some of the fundamental limitations of CNN architecture (such as them not being rotationally invariant). However, my understanding is that capsule networks have been a failure (so far), and most people expect them to go nowhere. And CNNs have progressively been improved in various ways (Bayesian optimisation for hyper parameter tuning, new convolution kernels, etc.). It seems to me that, at the moment, and for the foreseeable future, CNNs are the best architecture available for image-related stuff.
But, as I said, CNNs, like other Deep Learning architectures, require large amounts of data. So my question is as follows:
What are the research areas/topics for improving CNNs in the sense of making them work more effectively (that is, have greater performance) with less data (working with small datasets)?
I know that there is various research looking at approaches to increasing data (such as data augmentation, generative networks, etc.), but I am primarily interested in fundamental modifications to CNNs themselves, rather than purely focusing on changes to the data itself.
And to expand upon my question, using my above definition of "performance", I am interested in these two categories:

"Computational methods" for increasing CNN performance. This would be the non-mathematical stuff that I've read about, such as just increasing the number of layers and making the CNN deeper/wider (and I think another one had to do with just making the size of the convolution kernel smaller, so that it looks at smaller pieces of the image at any one time, or something like that?).

"Mathematical methods" for increasing CNN performance. This would be the cutting-edge mathematical/statistical stuff that I've read about: things like algorithms (such as Bayesian optimization); I've come across a lot of geometric stuff; and I guess the cutting-edge convolution kernels created by the image processing people would also fall under this category.

Obviously, this "list" is not exhaustive, and it's probably incorrect; I'm a novice to this research, so I'm trying to find my way around.
I am interested in studying both of the above categories, but I will primarily be working from the mathematical/statistical side. And I want to work on research that is still practical and can be put to use in industry for improved performance (even if it might still be "advanced"/complex for most people in industry) – not the the highly theoretical stuff related.
Related (but unanswered): Are there any good research papers on image identification with limited data?

So basically you're wondering about ways to make neural network training more data-efficient?

This question has extremely many answers. Basically, any method that improves the performance on neural networks without adding more data would qualify, which I would say is what the majority of papers in the field are about. Or in other words, if a method improves the performance of a neural network when faced with a lot of data, it will likely also improve the performance of a neural network when faced with little data.

@HelloGoodbye "data-efficient" – I think that's a good term. Yes, I suspected that that would be the case. Are there any areas that are particularly focused on, or effective at, improving "data-efficiency" in CNNs? I'm not looking for an exhaustive list – just broad categories with some examples (in particular, on the mathematics/statistics side). For someone starting in the field, it's difficult to know where to start, so I just need some information to help me along.

Some research areas that come to mind which can be useful when faced with a limited amount of data:

Regularization: Comprises different methods to prevent the network from overfitting, to make it perform better on the validation data but not necessarily on the training data. In general, the less training data you have, the stronger you want to regularize. Common types include:

Injecting noise in the network, e.g., dropout.

Adding regularization terms to the training loss, e.g., L1 and L2 regularization of the weights, but also confident output distributions can be penalized.

Reducing the number of parameters in the network to make it unable to fit the training data completely and thus unable to overfit badly. Interestingly, increasing the number of parameters for large models can also improve the validation performance.

Early stopping of training. For example, if one part of the training set is set aside and not used to update the weights, training can be stopped when the observed loss on this part of the training set is observed to start to increase.

Generating new training data:

Data augmentation: Ways to augment existing training examples without removing the semantics, e.g., slight rotations, crops, translations (shifts) of images.

Data interpolation, e.g., manifold mixup.

Using synthetic data, e.g., frames from video games or other CGI.

Transfer learning: When you take a neural network that has already been trained on another, much larger dataset of the same modality (images, sounds, etc.) as your dataset and fine-tune it on your data.

Multitask learning: Instead of training your network to perform one task, you give it multiple output heads and train it to perform many tasks at once, given that you have that the labels for the additional tasks. While it may seem that this is a more difficult for the network, the extra tasks have a regularizing effect.

Semi-supervised learning: If you have much unlabeled data that labeled data, you can combine supervised learning with unsupervised learning. Much like with multitask learning, the extra task introduced by the unsupervised learning also has a regularizing effect.

Other interesting methods can be found in systems that perform one-shot learning, which inherently implies very little training data. These systems often uses slightly modified network architectures. For example, facial recognition systems can learn to recognize a face from only a single photo, and usually use a triplet loss (or similar) of a vector encoding of the face, instead of cross-entropy loss of the output of a softmax layer normally used for image classification.
Zero-shot learning also exists (e.g., zero-shot machine translation), but this is a completely different type of problem setup and requires multiple data modalities.

Can transfer learning be done with any type/variation of CNN architecture?

Transfer learning can be done with any type of neural network as long as there are large enough commonalities between the dataset the network was originally trained on and your dataset, if your data is structured in the same way, as far as I know. So yes, it works for any type of CNN.


text:
How much can an inclusion of the number of iterations have on the training of an MLP?
My doubt is like this :

Suppose we have an MLP. In an MLP, as per the backprop algorithm (back-propagation algorithm), the correction applied to each weight is :

$$ w_{ij} := -\eta\frac{\partial E}{\partial w_{ij}}$$
($\eta$ = learning rate, $E$ = error in the output, $w_{ij}$ = $i^{\text{th}}$ neuron in the $j^{\text{th}}$ row or layer)

Now, if we put an extra factor in the correction as:

$$ w_{ij} := -k\eta  \frac{\partial E}{\partial w_{ij}}$$ ($k$ denotes the number of iterations at the time of correction)

how much will that factor affect the learning of the network ? Will it affect the convergence of the  network such that it takes time to fit to the data ?

NB : I am only asking this as a doubt. I haven't tried any ML projects recently, so this is not related to anything I am doing.

If anything, you want the learning rate to decrease as the number of iterations increases.
When you're looking for a good spot and you're clueless, take large steps. When you've found a pretty good spot, take small steps, so you don't end up far away.
In other fields of machine learning, there are studies of how the learning rate should scale. For example, in traditional reinforcement learning methods, if $\alpha_i$ is the learning rate at step $i$, then we want to have the following two criteria, to make sure we get convergence to the optimal policy:

$\sum_{i=0}^{\infty} \alpha_i = \infty$. This makes sure that, no matter how bad our initial experience was, we can eventually forget it and replace it with better information.
$\sum_{i=0}^{\infty} \alpha_i^2 < \infty$. This guarantees eventual convergence.

A typical choice here is $\alpha_i = \frac{1}{1+i}$, which fits both criteria.
I am unaware of similar criteria for MLPs, but if you're going to modify the step sizes, I would follow a similar approach. Make the step sizes decrease, but not too fast.

I didn't aim anything - I just wanted to know what happens when an additional factor is introduced, @RobbyGoetschalckx

I tried to view it from the mathematical side .

Also, I only wanted to know what happens when the factor introduced is the number of iterations at the time of correction.

Yes, and from the mathematical side, you want to decrease the step size as you're going along. When you've already learned a lot, you want step size to be small, so you don't overshoot the mark. And when you're far away, you want step sizes to be large.

I answered exactly what happens if you do that. If you scale it by the number of iterations, you will get a very unstable process.

Oh... sorry for my misunderstanding - I am just 15.

authors: игорь баламутин


text:
Does it make sense to train images (for object detection algorithms) with cameras that will not be used to collect future data?
I am training an algorithm to identify weeds within crops using the YOLOv5 algorithm. This algorithm will  be used in the future to identify weeds in images collected by unmanned aircraft (drones) after making an orthomosaic images. Using the open-source LabelImg software, I am labeling images for object detection that were collected with both UAV and hand-held digital cameras. Using both platforms, I collected many images of weeds that will need to be identified.
My question is this: Does it make sense to collect training samples from the hand-held digital camera, since it will be of much higher resolution than the UAV imagery (and thus not used for future imagery collections after the model is trained)? My initial thought is that it would be best to only use the UAV imagery, since it will be the most similar to what will be collected in the future. However, I do not want to throw out the hand-held digital imagery if it could help in the image classification process.

I think this can only be used for pretraining/some kind of transfer learning. This would be useful if the ratio of real training data to pretraining data is really low. You could then pretrain on digital data, and fine-tune on your UAV data.
How much of this is really useful I can't really say, this depends on how close the digital data is to UAV data. If it is significantly different, you are training on a different distribution and sample space than your UAV images, which is pointless.

That was my thought. It would be most useful to train on data that would be the most similar to the output of the end goal, so I chose to use the UAV data. Thanks.


text:
Which AI algorithm is great for mapping between two XML files
My work colleague got a project with a lot of work that is not hard or complicated.  The problem is simple but it is a lot of work.
We have two XML files with a lot of variables in it. Not only is the XML files flatten but you would have classes with classes in them that can reach an absurd amount of depth.
The problem comes in that the one file is a request that is received and the other is a response. The request needs to map its variables to the response variables. A simple tool could be built to solve this solution but the problem comes in that there are rules for certain variables. Some variables in the request have arithmetic involve, sometimes with other variables, or some don't get mapped if other variables are present.
I was thinking about Genetic Programming when I heard about this problem. If all the rules could be defined then it should be able to build a tree that would represent the desired output which is the XML response.
Will it work and if not do you think there is an AI algorithm that can solve this?

The standard tool to work with XML files is XSLT. You may not need AI to solve this problem. But.. you have to learn how to program with XSLT ;)
On Windows you can use MSXML, if you work from C++ - msxsl.exe, C# has internal supoort for XSLT. That is what I know about. There are also non-MS tools.


text:
Formal definition of the Object Detection problem
For many problems in computer science, there is a formal, mathematical problem defition.
Something like: Given ..., the problem is to ...
How can the Object Detection problem (i.e. detecting objects on an image) be formally defined?
Given a set of pixels, the task is to decide

which pixels belong to an object at all,
which pixels belong to the same object.

How can this be put into a formula?

For the first question, a typical way to formalize it might be defining a characteristic function: $$\chi_o (p) =
\begin{cases}
1, & \text{if pixel $p$ belongs to object $o$,}\
0, & \text{else }
\end{cases}$$

That formulation would be ok for image (or object) segmentation (where you need to classify individual pixels), but, in an object detection problem (which is a different problem than image segmentation), you do not need to classify individual pixels. You only need to find if the image contains an object of class $c$, where is that object (i.e. locate it), and maybe draw a bounding box around it. Maybe you are interested in image (or instance) segmentation. If that's the case, please, edit your post to say that.

Thank you, @nbro. I am interested in object detection, not image segmentation. I just thought that using pixel-based information could be a workaround to find a formula for object detection.
Actually, image segmentation could be the first step of the object detection task:

classify single pixels and assign probabilities to them of belonging to a specific
object,
merge pixels to objects.

The problem with that approach is that you need labelling information for all pixels, which may be expensive to acquire. That's why pixel-level classification to perform object detection may be overkill, but, of course, it's possible. In fact, image/instance segmentation can be thought of as a form of object detection (but a fine-grained one, let's say).

This is just an idea
Given a set of pixels, the task is to decide:

Which pixel is the center of an object?
What is the size of the bounding boxes with the center is the pixel in part 1?

Formula, consider this is a 2D image, call $(x,y)$ is the horizontal and vertical coordinate and $(w_i,h_i)$ is the size of bouding box of object $i$:
$\text{For }m \in[x,x+w_i] \text{ and } n\in[y,y+h_i]$
$c_i(m,n) = \begin{cases}
    1, \text{if pixel at position (m,n) is belongs to object i,}\\
    0, \text{else}
\end{cases}$

It's important to note that classifying pixels is known as image (or instance) segmentation, which is a different task than object detection (finding objects in images, maybe by providing the bounding box around the found objects). So, you do not need to classify individual pixels in the object detection task. You just need to find where the objects are (e.g. the center of the bounding box around them).

I knew what you mean, but the question of @JavAlex asked for the formula, so I thought about that equation based on his comment.

Thank you very much @Toby. That sounds like a good first idea


text:
How does one know that a problem is "model-free" in reinforcement learning?
Consider this slide from a Stanford lecture on reinforcement learning. It states that a model is

the agent's representation of how the world changes in response to the agent's action.

I've been experimenting with Q-learning for simple problems such as OpenAI's FrozenLake and Mountain Car, which both are amenable to the Q-learning framework (the latter upon discretization). I consider the topologies of the lake and the mountain to be the "worlds" (aka. environments) in the two cases, respectively.
Q-learning is said to be "model-free". Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken?

Let me know if the answers to this question answer your question. If your question is not "What does model-free mean?", then, please, edit your post to clarify what your question really is.

@nbro there is some overlap, but my question is much more concrete and dumbed-down: E.g., keeping in mind a specific example such as the car in the MountainCar problem, how can we tell that it is model-free? What would be the model in this case (e.g., a representation of the gravitational potential of the mountain)? As far as I see it, the Q-table that the agent (i.e. car) populates is an (implicit) representation of how the world changes, thereby making it model-based as per the definition.

If you understand the answers to the duplicate question, you will understand what a "model" is and, therefore, conclude that a value function is not a model (or what we mean by "model" in RL). The model is the transition matrix (or function) of the MDP. If you have or use it, your algorithm is model-based, otherwise, it is not. For your algorithm to be model-based in MountainCar, you need to learn or know this transition matrix. If you don't know what a transition matrix is, I suggest that you look at the definition of an MDP.

Please, edit your post to clarify your actual question. If you are asking "what is a model in reinforcement learning", then ask that. And please explain how your question is different than the linked one.

Q-learning is said to be "model-free". Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken?

No. That's not why Q-learning is model-free. Q-learning assumes that the underlying environment (FrozenLake or MountainCar, for example) can be modelled as a Markov decision process (MDP), which is a mathematical model that describes problems where decisions/actions can be taken and the outcomes of those decisions are at least partially stochastic (or random). More precisely, an MDP is composed of

A set of actions $A$ (that the RL agent can take); for example, up and down, in some grid world
A set of states $S$ (where the RL agent can be);
A transition function $p(s_{t+1} = s' \mid s_{t} = s , a_t = a)$ (aka the model), which represents the probability of going to state $s'$ at time step $t+1$, given that at time step $t$ the RL agent is in the state $s$ and takes action $a$.
A reward function $r(s, a, s')$ (sometimes also denoted as $r(s)$ or $r(s, s')$, although these can have different semantics); the reward function gives the reward (or reinforcement) to the RL agent when it takes an action in a certain state and moves to another state; the reward function can also be included in the transition function, i.e., often you will also see $p(s_{t+1} = s', r_{t+1} = r \mid s_{t} = s , a_t = a)$, and this is the model: this is what we mean by model in reinforcement learning, it's this $p$ (which is a probability distribution)!

A model-free algorithm is any algorithm that does not use or estimate this $p$. Q-learning, if you look at its pseudocode, does not make use of this model. Q-learning estimates the value function $q(s, a)$ by interacting with the environment (taking actions and receiving rewards), but, meanwhile, it does not know or keep track of the dynamics (i.e. $p$) of the environment, and that's why it's model-free.
And, no, the value function is not what we mean by "model" in reinforcement learning. The value function is, as the name suggests, a function.

How does one know that a problem is "model-free" in reinforcement learning?

A problem is not model-free or model-based. An algorithm is model-free or model-based. Again, a model-free algorithm does not use or estimate $p$, a model-based one uses (and/or estimates) it.

Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken?

No. As stated in the other answer, you could apply the model-based algorithm Dyna-Q to these environments.

A reinforcement learning algorithm is considered model based if it uses estimates of the environments dynamics to help learn. For instance, in the Tabular Dyna-Q algorithm, every time you visit a state action tuple you store in a look-up table the reward received and the next state transitioned to, and after every execution of an action you loop $n$ times to further back up your $Q$ table using these stored model values from the look-up table. I will attach a copy of the pseudo-code for an algorithm at the bottom of this post.
Algorithms like vanilla $Q$-learning are model free because they don't require a model of the environment to learn.

This answer is correct and I would upvote it, but if this question is a duplicate of this one, I think it's better you answer there (although there are already many answers there, but none of them mentions Dyna-Q, apparently), because this will probably be closed as a duplicate of that one (I am just waiting for the OP's feedback).

My question is more about the physical intuition of what the model is, hence my specific reference to the concrete examples of FrozenLake and MountainCar. If a model is "the agent's representation of how the world changes in response to the agent's action", then a Q-learning approach to MountainCar would be considered model-free because the car (i.e. the agent) does not have a representation of what the gravitational potential of the mountain is?


text:
How does dimensionality reduction occur in Self organizing Map (SOM)?
We have n dimension input for SOM and the output 2-D clusters. How does it happen?

SOM (Self-Organinizing Map) is a type of artificial neural network (ANN), introduced by the Finnish professor Teuvo Kohonen in the 1980s, that is trained using unsupervised learning to produce a low-dimensional, discretized representation of the input space of the training samples, called a map, and is therefore a method to do dimensionality reduction.
SOM produces a mapping from a multidimensional input space onto a lattice of clusters, i.e. neurons, in a way that preserves their topology, so that
neighboring neurons respond to “similar” input patterns.
It uses three basic processes:

Competition
Cooperation
Adaptation

In competition, each neuron is assigned a weight vector with the same
dimensionality d as the input space. Any given input pattern is compared to the weight vector of each neuron and the closest neuron is declared the winner.
In cooperation, the activation of the winning neuron is spread to neurons in its immediate neighborhood, and as a result this allows topologically close neurons to become sensitive to similar patterns. The size of the neighborhood is initially large, but shrinks over time, where an initially large neighborhood promotes a topology-preserving mapping and smaller neighborhoods allows neurons to specialize in the latter stages of training.
In adaptation, the winner neuron and its topological neighbors are adapted to make their weight vectors more similar to the input pattern that caused the activation.
So, a Self-Organizing Map (SOM) is to encode a large set of input vectors $\textbf{x}$ by finding a smaller set of “representatives” or
“prototypes” or “code-book vectors” $\textbf{w}$ that provide a good approximation to the original input space. This is the basic idea of vector quantization theory, the motivation of which is dimensionality reduction or data compression. Performing a gradient descent style minimization on SOM's loss function (eg. the sum of the Euclidean distances between the input sample and each neuron) does lead to the SOM weight update algorithm, which confirms that it is generating the best possible discrete low dimensional approximation to the input space (at least assuming it does not get trapped in a local minimum of the error function).
To answer your question, you should take into consideration that the Dimensionality reduction takes place in fields that deal with large numbers of observations and/or large numbers of variables. Thus, SOM helps finding good "prototypes", in a way that each input pattern belongs to exactly one of them. As a result, the training instances are mapped to the training "prototypes" and the whole training set is mapped to a new one with less instances.
In addition, the "prototypes" neurons resulted by SOM can often be used as good centers in RBF networks or to classify patterns with the LVQ family algorithms.

Good explanation about SOM but I could not find where you answered my ques

can you bold those lines


text:
Any comparison between transformer and RNN+Attention on the same dataset?
I am wondering what is believed to be the reason for superiority of transformer?
I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. However, as far as I know, you can use attention also with RNN  architectures as in the famous paper attention is introduced(here)).
I am wondering whether the only reason for the superiority of transformers is because they can be highly parallelized and trained on much more data?
Is there any experiment comparing transformers and RNN+attention trained on the exact same amount of data comparing the two?

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation method:

For example, Deep-Att + PosUnk is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the transformer with self-attention is $2.3 \cdot 10^{19}$ (FLOPs) and $1.0 \cdot 10^{20}$ (FLOPs) for the "Deep-Att + PosUnk" method (the transformer is 4 times faster) on "WMT14 English-to-French" dataset.
Please note that the BLEU is a crucial factor here (not merely training cost). Hence, you can see the BLEU value of the transformer superior to the ByteNet (Neural Machine Translation in Linear Time). Although the ByteNet has not adopted the RNN, you can find the comparison of the ByteNet with other "RNN + Attention" methods in its original paper:

Hence, by transitivity property of the BLEU score, you can find that the transformer has already outperformed other "RNN + Attention" methods in terms of the BLEU score (please check their performance on "WMT14" dataset).

Thanks @OmG! Your answer is what I was looking for! Thank you. By the way, what if the dataset is limited?

My pleasure @miladaghajohari . Due to the translation task's high complexity, the required model's VC dimension will be increased. Hence, based on the computational learning theory, the required sample complexity to train the model will be increased as well. Therefore, a limitted dataset is unlikely to be suitable for the task.


text:
Why is the E step in expectation maximisation algorithm called so?
The E step on the EM algorithm asks us to set the value of the variational lower bound to be equal to the posterior probability of the latent variable, given the data points and parameters. Clearly we are not taking any expectations here, then why is it called the Expectation step? Am I missing something here?

In expectation step, firstly we calculate the posterior of latent variable $Z$ and then the $Q(θ | θ^{(t)})$ is defined as the expected value of the log likelihood of $θ$, with respect to the current conditional contribution of $Z$ given $X$ and the current estimates of $θ^{(t)}$. In maximization step, we update $θ$ using the argmax on $Q$, with respect to $θ$.
$$Q(θ | θ^{(t)}) = E_{Z|X,θ^{(t)}}[logL(θ;Χ,Z)]$$
To be more intuitive, think of k-means as a special case of EM, where in expectation step the $Z$ variables are defined, that is the latent variables indicating the membership in a cluster, and calculated in a hard assignment way. In maximization step the $μ$s of the clusters are updated. If you want to see the corresponding relation for $Q$ in k-means, I suggest you read the chapter 9.3.2 in C.Bishop's book: Pattern Recognition and Machine Learning.

I understood the intuition behind the E step in k means, but is it possible to extend the same intuition to the general EM algorithm?How do we visualise the posterior as an expectation in the derivation of the general EM algorithm? Or equivalently, can we show that the expectation term in the formula given by you for the E step can be expressed as the posterior?

@Manish Kausik Hari Baskar. The Q function uses the posterior calculated in E step as the sum of posteriors * $ln(X,Z|θ)$ over Z. Also, I fixed my post.


text:
How to find the optimal pokemon team
Pokemon is a game where 2 players each select 6 Pokemon (a team) at the beginning of the game without knowing the other player's team. Every Pokemon has one or two types. Every type is either weak, neutral or strong against every other type. This means that every 2 Pokemon matchup will either have a winner or be a tie. This also means that any team can be ranked against any other team based on the number of winning matchups they have.
I want to write a program that can find the optimal Pokemon team out of a set of 70 provided Pokemon.
A team is considered optimal if it has the greatest number of winning matchups against any other team. Basically, I want to calculate which team will have the most amount of favorable matchups if you were to battle it against every other possible team.
What algorithm would be best for doing this? It is not feasible to compute matchups for every possible team. Can I do some sort of A* search with enough pruning to make it computationally feasible?

After my initial comment (where I suggest that it might not be enough info) I believe I actually came up with an idea.
Start with the full set of pokemon. For every possible type, identify the count of pokemon that are strong against that type. For this, you'll end up with a List<(pokemonId, types, List<weakAgainst>)>.
Minimize List<weakAgainst>.Count() and from the possible set of pokemonIds, select one at random. Without knowing anything else besides type, this pokemon is as good as any other with the same weakness count (this is the point of my original comment).
From the list of weaknesses that this selected pokemon has, select a pokemon from your list that is strong against the weakness, minimizing the amount of weaknesses again. Likely more than one will match this criteria, again, select one at random.
Keep repeating this pattern until you obtain the 6 in your team. This is, statistically speaking, one of the best teams that you can gather.
For all the combinations that you might find here, some teams will have less weaknesses, since we're "randomly" walking down a tree of possibilities. This very much sounds like a minimax-prunning algorithm, where each pokemon selection (minimizing your weaknesses) can be met with potential opponents that will maximize your weak points.
Simplified, put together:
input: allPokemon: list<pokemonId, weakAgainst, strongAgainst>

var: teamWeakAgainst: []
var: teamStrongAgainst: []
var: selectedTeam: []

while (size(selectedTeam) < 6)
  goodMatches <- allPokemon.max(p -> size(p.strongAgainst.intersect(teamWeakAgainst)))
  goodMatches <- goodMatches.min(p -> size(p.weakAgainst))
  goodMatches <- goodMatches.max(p -> size(p.strongAgainst))

  selectedPokemon <- goodMatches.random()

  teamWeakAgainst -= selectedPokemon.strongAgainst
  teamWeakAgainst += selectedPokemon.weakAgainst # not counting previously selected pokemon because the current one adds another "weakness", even if it was already accounted for

  selectedTeam += selectedPokemon

output: selectedTeam

From this algorithm is it not obvious where the "max" portion is. We're minimizing our losses (weaknesses) but we're considering all possible opponent teams equally, so there is no real maximization of the opponent choices. For a set of ideas, check below.
Note that this algorithm will give you a set of "teams" that are equally good in the sense that they'll have the same amount of minimized weaknesses and maximized strengths against other possible teams. But even if pokemon are different, the numbers will be the same, just different types.
For a more complex approach, you might want to consider how prevalent some pokemon are (you might not need to optimize against a super rare mythical type, but rather the very common types available in the game), how likely is it that certain pokemon can have better / faster attacks, what is the probability of battle IVs, how frequent can a trainer switch pokemon in battle, etc. Again, I know this is not what you asked for, but for the sake of the example, this will become so complex that instead of a search algorithm, a simulation (Monte Carlo?) approach might be simpler to build teams out of statistical testing.

I'm worried that this algorithm is going to generate an unbalanced team. For example, water is the most common type of Pokemon. Electric is only weak to ground. This means that electric Pokemon are always going to have the most amount of strengths and fewest amount of weaknesses. This algorithm then will generate 6 electric types. This needs to be accounted for somehow. Maybe a weight to someone minimize the value of Pokemon that already have a positive matchup?

I see that it will subtract the pokemon from the weak against list so that accounts for the problem of generating 6 of the same type.                                                    So if I'm understanding this correctly it will find a team that will always have more strengths than weaknesses, but it will not have the maximum possible amount of strengths.

Correct, it minimizes weaknesses as first metric and maximizes strengths as secondary metric. If it adds a water pokemon and that makes you weak against electric, it'll likely pick (for example) a rock type next.


text:
Image Classification for watermarks with poor results
Just starting learning things about tensorflow and NN.
As an exercise I decided to create a dataset of images, watermarked and not, in order to binary classify these. First of all, the dataset ( you can see it here ) was created artificially by me applying some random watermarks.
First doubt, in the dataset I don't have both images one watermarked and one not, would be better to have?
Second, frustrating: model stand on 0.5 accuracy, so it just produce random output :(
Model I tried is this:
model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(16,(1,1), activation='relu', input_shape=(150, 150, 3)),
    tf.keras.layers.MaxPool2D(2,2),
    tf.keras.layers.Conv2D(32,(3,3), activation='relu'),
    tf.keras.layers.MaxPool2D(2,2),
    tf.keras.layers.Conv2D(64,(3,3), activation='relu'),
    tf.keras.layers.MaxPool2D(2,2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation='elu'),
    tf.keras.layers.Dense(64, activation='elu'),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1,activation="sigmoid")

and then compiled as this:
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics = ['accuracy'])

Here below the fit:
history = model.fit(train_data,
                              validation_data=valid_data,
                              steps_per_epoch=100,
                              epochs=15,
                              validation_steps=50,
                              verbose=2)

As for any other details, code is here.
I already checked for technical issues, I'm pretty sure image enter properly, train and validation dataset are 80/20, about 12K images for training. However accuracy bounches up and down around .5 while fitting. How can I improve?

Well probably the response is that previous approach was a little naive.
I managed to fave some interesting result with this kernel that allow me to have an accuracy of 0.969  and a validation accuracy of 0.931.
Model I used is based on ResNet50 with the following additional layers ( and the last one just for binary classification ):
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(32, activation='relu'),
tf.keras.layers.Dense(1,activation="sigmoid")

Even if the net is already trained, I trained each layer again otherwise I did not have any sensible accuracy.
Training history is like this:

Still far to be really good, but progress.


text:
Transfer Learning of Numerical Data
It seems like transfer learning is only applicable to neural networks. Is this a correct assumption?
While I was looking for examples of Transfer Learning, most seemed to be based on image data, audio data, or text data. I was not able to find an example of training a neural network on numerical data.
I want to use transfer learning in this particular scenario: I have a lot of numerical data from an old environment, with binary classification labels, and utilizing this, want to train on a new environment to do the same binary classification.
The dataset would look something like this
Is this possible? What would the model look like?

It seems like transfer learning is only applicable to neural networks. Is this a correct assumption?

No. Wiki page give you pointers of several examples in other methodologies.

While I was looking for examples of Transfer Learning, most seemed to be based on image data, audio data, or text data. I was not able to find an example of training a neural network on numerical data.

All the cases you say are converted to numerical data. Image and audio usually via sampling, text via one-hot encoding.

I want to use transfer learning in this particular scenario: I have a lot of numerical data from an old environment, with binary classification labels, and utilizing this, want to train on a new environment to do the same binary classification.

That is not transfer learning. Transfer learning applies when there are a change in the domain (input features) or in the task (output labels).

The dataset would look something like this Sample Table
Is this possible? What would the model look like?

For a simple case as the one you present, probably a simple network with one hidden layer will be enough. Train it with original pairs of {features,label} or, if not available, use the current predictor to obtain the label from the features.

Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction


text:
Can the hidden layer prior to the ouput layer have less hidden units than the output layer?
I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer.
I remember that the Professor saying that there is no rule for choosing the number of hidden units and that having many of them along with many hidden layers can cause the network to overfit the data and under learn.
However, I still have this question where assuming that we have a network with an input layer of n  input nodes, a first hidden layer of 4 hidden units, a second layer of X hidden units and an output layer of 5 units. Now if I follow the Professor's saying, it would mean that I am allowed to have X = 3 or X = 4 in layer 2.
Is that actually allowed? Won't we have some sort of information gain passing from 4 (or 3) nodes to 5?  The example is illustrated below.

A layer with bigger number of nodes than previous one is something very common. Some examples are:

strategies encoder-decoder (autoencoders) where the encoder typically  has layers with a decreasing number of nodes (until the compressed/encoded data) and the decoder has layers increasing in number of nodes.

bidirectional recurrent networks where in the forward direction number nodes decreases and in the backward increases.

generators, that from a random vector generates, by example, a full image.

As general rule: decrease number of nodes forces the net to filter/resume/abstract/summarize the internal signal information (discarding useless information or noise) while increase number of nodes means apply current information to generate an answer value for a specific question/target.
Allow me a strongly simplified example: assume you want a system that, from a photo of an animal, answers the questions: number of legs? has beak ? flies ? . Net inputs are images of birds and dogs.
The net architecture can have layers of decreasing size until a single node that will decide "is bird or dog ?". From this single item of information (the only one need to answer all the questions) the output layer will have 3 nodes, each one answering one of the specific target questions: number or legs ? 4 if dog, 2 if bird, etc .


text:
Smallest possible network to approximate the $sin$ function
The main goal is: Find the smallest possible neural network to approximate the $sin$ function.
Moreover, I want to find a qualitative reason why this network is the smallest possible network.
I have created 8000 random $x$ values with corresponding target values $sin(x)$. The network, which am currently considering, consists of 1 input neuron, 3 neurons in two hidden layers, and 1 output neuron:
Network architecture:

The neural network can be written as function
$$y = sig(w_3 \cdot sig(w_1 \cdot x) + w_4 \cdot sig(w_2 \cdot x)),$$
where $\text{sig}$ is the sigmoid activation function.
$tanh$ activation function:
When I use $tanh$ as an activation function, the network is able to hit the 2 extrema of the $sin$ function:
$tanh$ activation function" />
Sigmoid activation function:
However, when I use the sigmoid activation function $\text{sig}$, only the first extremum is hit. The network output is not a periodic function but converges:

My questions are now:

Why does one get a better approximation with the $tanh$ activation function? What is a qualitative argument for that?
Why does one need at least 3 hidden neurons? What is the reason that the approximation with $tanh$ does not work anymore, if one uses only 2 hidden neurons?

I really appreciate all your ideas on this problem!

Is the smallest network that can "approximate" the sin function not just $f(x) = \sigma(w\cdot x + b)$ where $w = 1$, $b=0$, and $\sigma(\cdot) = \sin(\cdot)$?

thank you for your comment. But I am looking for the smallest network with a tanh or sigmoid activation function

Is x restricted to some interval ?

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh network had to start from zero).
Generally I would say it's impossible to have a good approximation of sinus with just 3 neurons, but if you want to consider one period of sinus, then you can do something. for clarity look at this picture:

I've write the code for this task in colab and you can find it here, and you can play with it if you want.
If you run the network several times you may get different results (because of different initializations) and you can see some of them at the Results section of the link above. What you showed us in the images above are just two possibilities. But it's interesting that you can get better results with tanh rather than sigmoid and if you want to know why, I highly recommend you to look at this lecture of CS231n. In summary it's because tanh has the negative part and the network can learn better with it.
But actually their power of approximation are almost similar because 2*sigmoid(1.5*x) - 1 almost looks the same as tanh(x) and you can find it by looking the picture below:

So why you can't get the same results as tanh? that's because tanh suits the problem better and if the network wants to get the same result as tanh with sigmoid it should learn their transformation parameters and learning these parameters makes the learning task harder. So It's not impossible to get the same result with sigmoid but it's harder. And to show you that its possible, I have set the parameters of the network using sigmoid manually and got the result below (you can get better results if you have more time):

At last if you want to know why you can't get the same result with 2 neurons instead of 3 neurons, it's better to understand what does the network do with 3 neurons.
If you look at the output of the first layer, you may see something like this (which are outputs of two neurons it has):

Then the next layer gets the difference between the output of these two neurons (which is like sinus) and applies sigmoid or tanh to it, and that's how you get a good result. But when you have just one neuron in the first layer, you can't imagine some scenario like this and approximating one period of sinus is out of it's ability (underfitting).

Thank you very much for your detailed answer. One question: How did you come up with the manually set weights for the sigmoid approximation of the sinus? Did you figure them out just by trial and error/ educated guesses or did you use any system?

Happy if it helped. It was combination of both. The ratio of the weights was trial, but I knew what the network should do and what the output should look like in each layer (as I described in the answer). So it's not hard to find the weights. Feel free to uncomment the related parts in the colab link and play with the weights and biases to see their affect on output of each layer.


text:
Q-learning agent stuck at taking same actions
I have created my own RL environment where I have a 2-dimensional matrix as a state space, the rows represent the users that are asking for a service, and 3 columns representing 3 types of users; so if a user U0 is of type 1 is asking for a service, then the first row would be (0, 1, 0) (first column is type 0, second is type 1...).
The state space values are randomly generated each episode.
I also have an action space, representing which resources were allocated to which users.
The action space is a 2-dimensional matrix, the rows being the resources that the agent has, and the columns represent the users. So, suppose we have 5 users and 6 resources, if user 1 was allocated resource 2, then the 3rd line would be like this: ('Z': a value zero was chosen, 'O': a value one was chosen)
(Z, O, Z, Z, Z)
The possible actions are a list of tuples, the length of the list is equal to the number of users + 1, and the length of each tuple is equal to the number of users.
Each tuple has one column set to 'O', and the rest to 'Z'. (Each resource can be allocated to one user only). So the number of the tuples that have one column = 'O', is equal to the number of users, and then there is one tuple that has all columns set to 'Z', which means that the resource was not allocated to any users.
Now, when the agent chooses the action, for the first resource it picks an action from the full list of possible action, then for the second resource, the action previously chosen is removed from the possible actions, so it chooses from the actions left, and so on and so forth; and that's because each user can be allocated one resource only. The action tuple with all 'Z' can always be chosen.
When the agent allocates a resource to a user that didn't request a service, a penalty is given (varies with the number of users that didn't ask for a service but were allocated a resource), otherwise, a reward is given (also varies depending on the number of users that were satisfied).
The problem is, the agent always tends to pick the same actions, and those actions are the tuple with all 'Z' for all the users. I tried to play with the q_values initial values; q_values is a dictionary with 2 keys: 1st key: the state being a tuple representing each possible state from the state space, meaning (0, 0, 0) & (1, 0, 0) & (0, 1, 0) & (0, 0, 1), combined with each action from the possible actions list.
I also tried different learning_rate values, different penalties and rewards etc. But it always does the same thing.

I am confused. For the initial $Q$-values, you generate one for each possible row $(1, 0, 0), (0,0,0), \ldots$ so you would have 4 states.
However, from the first paragraph it seems that the states themselves are matrices (one row for each user), so the state space is a set of such matrices.
That means that your $Q$-table should have a row for each possible matrix, and a column for each possible total assignment of items to users.

So, suppose we have Q-table accessed as follows: Q[state, action]. So the state is the whole state space matrix and the action is the whole matrix with the different actions for each resource and user ?

I am confused too because from what I gathered from the examples online, is that if we have a state space that's an array of different positions (for example) that the agent can be in, then the state in the Q-table would be each position in that state space. So what I thought I should do is represent each row in the state space as a key for the Q-table.

A state should be the full description of the state. The state space is the set of all possible such states. It appears that in your example, a state is a matrix (with one row for each current user). The state space is a set of such matrices.
In the online examples, each state is just a single position, so there the table is a bit easier.


text:
How to tell a neural network that: "your i-th input is special"
Assume that I have a fully connected network that takes in a vector containing 1025 elements. First 1024 elements are related to the input image of size 32 x 32 x 1, and the last element in the vector (1025-th element) is a control bit that I call it special input.
When this bit is zero, the network should predict if there is a cat in the image or not, and when this bit is one, it should predict if there is a dog in the image or not.
So how can I tell the network that your 1025-th element should be special to you and you should pay more attention to it?
Note that it's just an example and the real problem is more complex than this. So please don't bypass the goal of this question by using tricks special to this example. Any idea is appreciated.

The main benefit of deep learning is that you don't have to manually design your features. The network itself will map it. So, I suppose you can just train your algorithm with your dataset.

Thank you for commenting. My hope of this happening was so low that I did not even bother myself to test a simple scenario like this. But as you say it will happen, I will test it soon and return the result here.

@ArayKarjauv Yes it worked. If you want, you can write your comment as answer and I will accept it

by the way, you can also add one more input layer for your flag. I updated my answer.

The main benefit of deep learning is that you don't have to manually design features.
Classic Machine Learning algorithms always include the Feature engineering step, whereas neural networks are able to extract features automatically during learning. The classic example is CNN. In the first layer, it creates simple features that representing lines, the last layers represent abstract features. Of course, some tasks do require feature engineering (e.g. signal processing).
In your case, if you want to take advantage of the CNN network, you can also add an additional input layer for the flag (e.g. as one-hot vector). Here is an illustration taken from this answer.

Your comments and answer seems a few contradictory. First you propose handle in same way all 1025 inputs, next you talk about a CNN (with 32*32+1 inputs ?) and finally propose an architecture with two separated stages.

there are different approaches. I assume, the authors feeds an image as a NxM vector, as he mentioned that he has 1025-th element. So it is possible to add one extra feature. I have also added some additional information on CCN for generalization

Are neural nets in general able to crate features automatically during learning, or this is a specific characteristic of CNNs?

This applies not only to CNN. As I wrote, CNN is just a classic example.

Assume the image can contain objects of class $C_1 \dots C_c$. Assume a set of additional inputs that has a meaning of questions as "contains the image a C_i or C_j or ... ?".
The main problem for the system is classify the image in classes $C_i$. Second problem is answer the implicit question proposed by the remainder inputs.
Thus, better combine two NNs:

first one an object recognizer, with input the image data.
second one a NN that will answer the question implicit in remainder bits, with inputs the output of previous NN and the "question" bits.

In concrete for the example that the question describe:

a NN to classify dog/cat from 32x32 image. Two outputs for probabilities of dog an cat. Or simply one binary output 0:"is a dog", 1:"is a cat".
a second NN with the "1025" binary input ( 0: "look for dogs", 1: "look for cats") and output from previous one. In this case, if everything is ok, it will infer the logic "a==b": "is a dog" and "look for dogs" or "is a cat" and "look for cats".

Note that if you try to solve the problem directly with a full-connected NN with all inputs (1025 in the example), you loss the possibility of use CNN and max-pooling layers, etc. Moreover, split decreases training times, join subjects increases them "exponentially". Not a promising way.