text:
What's the benefit of repeating an action for a consecutive number of time steps?
What's the benefit of repeating an action for a consecutive number of time steps? Is there a way to tell if an agent in a given environment might perform better from repeated actions?
I came across an action repeat hyperparameter used in the experiments of this paper (Table 11.) and was wondering what the objective of this hyperparameter might be. This happens during training where the action selected by a policy at step $t$ is executed for a repeat number of timesteps $t_r$ after which the policy is then queried for another action at the current state $s_{t + t_r}$.

You mean hold a given action selection for more than one timestep like sticky action?

I assume you're talking about behaviour policies, so during learning. It might a good idea to explicitly state this in your post, even though it seems obvious. I would also recommend that you explain why you're asking this question: just out of curiosity, or do you have something more specific that you can share with us?

Yes @HenDoNR, action selection for more than a signle timestep

I've edited the question to include the suggested changes @nbro

A primary reason to repeat an action of a series of timesteps is that your environment may require more than one timestep to process the timestep.  Said another way, changing the action every timestep might prove to hamper the overall learning process significantly.
As an example, consider using RL with something from OpenAI Gym.  Imagine that you are trying to train an agent to play something like Mario Brothers.  If your agent takes a different random action at every timestep, how long might it take the agent to realize that it needs to keep sending "Right" to improve the x position?  It might be better to configure your agent such that each action is repeated for, say, four timesteps, allowing the action to begin to take on real meaning in the environment.


text:
House price inflation modelling
I have a data set of house prices and their corresponding features (rooms, meter squared, etc). An additional feature is the sold date of the house. The aim is to create a model that can estimate the price of a house as if it was sold today. For example a house with a specific set of features (5 rooms, 100 meters squared) and today's date (28-1-2020), what would it sell for? Time is an important component, because prices increase (inflate over time). I am struggling to find a way to incorporate the sold date as a feature in the gradient boosting model.
I think there are a number of approaches:

Convert the data into an integer, and include it directly in the model as a feature.
Create a separate model for modelling the house price development over time. Let's think of this as some kind of an AR(1) model. I could then adjust all observations for inflation, so that we would get an inflation adjusted price for today. These inflation adjusted prices would be trained on the feature set.

What are your thoughts on these two options? Are there any alternative methods?

The sold date is a feature like any other. You can do this as follow. I am assuming the features are in a pandas data frame  called df where the column date is called date. Easiest way is to use the pandas to_datetime function. Documentation is here.
def encode_dates(df, column):
    df = df.copy()
    df[column] = pd.to_datetime(df[column] )
    df[column + '_year'] = df[column].apply(lambda x: x.year)
    df[column + '_month'] = df[column].apply(lambda x: x.month)
    df[column + '_day'] = df[column].apply(lambda x: x.day)
    df = df.drop(column, axis=1)
    return df
df=encode_dates(df. 'date')

This function will modify the df data frame. It will create 3 new columns labeled
date year, date month and date day and it will remove the date column from the data frame.
Now these new columns along with the other features can be used to train your model.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the dimension of my output of the form (2n + 1, 2n + 1, #filters) after a MaxPooling layer
I'm trying to white board the different mechanisms behind a convolutional neural network.
I have on question regarding the dimension of my volume after using a max pooling layer.
Let's suppose I have a (21,21,#filtres) volume's dimension. If Max Pooling divide by 2 the height and width of my volume, what will be the dimension after the Max Pooling layer ?
If odd numbers are a problem when using max pooling layer, How do I fix it ?
Thank you !

Are you considering the stride parameter (available in most libraries)? If so, what is it set to? The default would be to set stride equal to the pooling size i.e. 2, and therefore divide layer size by 2.

From the lectures I followed, stride was often set to 1. Make it equal to 2 will fix the problem ?

If stride is set to 1 there is no problem, the pooling will not need to be applied partially in order to cover the whole input. Your output would be (20, 20, num_filters) though, it doesn't shrink the feature map as much. It still provides useful generalisation. Stride is often set to 1 for the convolutional layer, but not for pooling layers, so it may be the value you noticed from lectures was referring convolutional layers.

The result from applying a max pooling layer with a stride that does not exactly fit to the input will be dependent on the implementation in your library.
Assuming stride 2, and pool size (2,2), in your case the most likely things are:

The result will round up, so you will have a feature map layer with dimensions (11, 11, num_filters) although the right edge pixels will be a max over 2 pixels in the input, and the right bottom corner will just be a copy of the right bottom corner in the input (counting from top left as $(0, 0)$)

It is an error condition for your libary.

If it is not an error, then the max pooling should still perform the task it was intended to. If important features are often at the right or bottom edge, then they may generalise slightly less well, but you probably won't notice a measurable effect.
You could experiment with different sizes of pooling, different strides, or padding the previous convolutional layer so that the max pooling fits exactly. You will have to do something like this if the library has errors when the pooling does not fit exactly. You can test your experiments using cross validation, to see if there is any measurable difference.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
BERT: After pretraining 880000 step, why fine-tune not work?
I am using pretraining code from https://github.com/NVIDIA/DeepLearningExamples
Pretrain parameters:
 15:47:02,534: INFO tensorflow 140678508230464   init_checkpoint: bertbase3layer-extract-from-google
 15:47:02,534: INFO tensorflow 140678508230464   optimizer_type: lamb
 15:47:02,534: INFO tensorflow 140678508230464   max_seq_length: 64
 15:47:02,534: INFO tensorflow 140678508230464   max_predictions_per_seq: 5
 15:47:02,534: INFO tensorflow 140678508230464   do_train: True
 15:47:02,535: INFO tensorflow 140678508230464   do_eval: False
 15:47:02,535: INFO tensorflow 140678508230464   train_batch_size: 32
 15:47:02,535: INFO tensorflow 140678508230464   eval_batch_size: 8
 15:47:02,535: INFO tensorflow 140678508230464   learning_rate: 5e-05
 15:47:02,535: INFO tensorflow 140678508230464   num_train_steps: 10000000
 15:47:02,535: INFO tensorflow 140678508230464   num_warmup_steps: 10000
 15:47:02,535: INFO tensorflow 140678508230464   save_checkpoints_steps: 1000
 15:47:02,535: INFO tensorflow 140678508230464   display_loss_steps: 10
 15:47:02,535: INFO tensorflow 140678508230464   iterations_per_loop: 1000
 15:47:02,535: INFO tensorflow 140678508230464   max_eval_steps: 100
 15:47:02,535: INFO tensorflow 140678508230464   num_accumulation_steps: 1
 15:47:02,535: INFO tensorflow 140678508230464   allreduce_post_accumulation: False
 15:47:02,535: INFO tensorflow 140678508230464   verbose_logging: False
 15:47:02,535: INFO tensorflow 140678508230464   horovod: True
 15:47:02,536: INFO tensorflow 140678508230464   report_loss: True
 15:47:02,536: INFO tensorflow 140678508230464   manual_fp16: False
 15:47:02,536: INFO tensorflow 140678508230464   amp: False
 15:47:02,536: INFO tensorflow 140678508230464   use_xla: True
 15:47:02,536: INFO tensorflow 140678508230464   init_loss_scale: 4294967296
 15:47:02,536: INFO tensorflow 140678508230464   ?: False
 15:47:02,536: INFO tensorflow 140678508230464   help: False
 15:47:02,536: INFO tensorflow 140678508230464   helpshort: False
 15:47:02,536: INFO tensorflow 140678508230464   helpfull: False
 15:47:02,536: INFO tensorflow 140678508230464   helpxml: False
 15:47:02,536: INFO tensorflow 140678508230464 **************************

Pretrain loss: (I remove nsp_loss)
{'throughput_train': 1196.9646684552622, 'mlm_loss': 0.9837073683738708, 'nsp_loss': 0.0, 'total_loss': 0.9837073683738708, 'avg_loss_step': 1.200513333082199, 'learning_rate': '0.00038143058'}
{'throughput_train': 1230.5063662500734, 'mlm_loss': 1.3001925945281982, 'nsp_loss': 0.0, 'total_loss': 1.3001925945281982, 'avg_loss_step': 1.299936044216156, 'learning_rate': '0.00038143038'}
{'throughput_train': 1236.4348949169155, 'mlm_loss': 1.473339319229126, 'nsp_loss': 0.0, 'total_loss': 1.473339319229126, 'avg_loss_step': 1.2444063007831574, 'learning_rate': '0.00038143017'}
{'throughput_train': 1221.2668264552692, 'mlm_loss': 0.9924975633621216, 'nsp_loss': 0.0, 'total_loss': 0.9924975633621216, 'avg_loss_step': 1.1603020071983337, 'learning_rate': '0.00038142994'}

Fine-tune code:
self.train_op = tf.train.AdamOptimizer(0.00001).minimize(self.loss, global_step=self.global_step)

Fine-tune accuracy: (restore from my ckpt pretrained from https://github.com/NVIDIA/DeepLearningExamples)
epoch 1:
training step 895429, loss 4.98, acc 0.079
dev loss 4.853, acc 0.092

epoch 2:
training step 895429, loss 4.97, acc 0.080
dev loss 4.823, acc 0.092

epoch 3:
training step 895429, loss 4.96, acc 0.081
dev loss 4.849, acc 0.092

epoch 4:
training step 895429, loss 4.95, acc 0.082
dev loss 4.843, acc 0.092

Without restore the pretrained ckpt:
epoch 1:
training step 10429, loss 2.48, acc 0.606
dev loss 1.604, acc 0.8036

Restore the google's BERT-Base pretrained ckpt. Or restore from a pretrained ckpt pretrained from https://github.com/guotong1988/BERT-GPU
epoch 1:
training loss 1.89, acc 0.761
dev loss 1.351, acc 0.869

Hello. Welcome to AI SE! "fine-tune not work" is not a question, so please edit your post to explicitly ask a specific question. Moreover, note that programming questions are generally off-topic here. Here we focus on theoretical, social and philosophical aspects of AI. Your question seems to be a programming issue, so it's off-topic here.  See https://ai.stackexchange.com/help/on-topic for more info.

change
bert_output = bert_model.get_pooled_output()

to
bert_output = tf.reduce_mean(bert_model.get_sequence_output()[:,1:,:],1)


text:
Bert for Sentiment Analysis - Connecting final output back to the input
I have not found a lot of information on this, but I am wondering if there is a standard way to apply the outputs of a Bert model being used for sentiment analysis, and connect them back to the initial tokenized string of words, to gain an understanding of which words impacted the outcome of the sentiment most.
For example, the string "this coffee tastes bad" outputs a negative sentiment.  Is it possible to analyze the output of the hidden layers to then tie those results back to each token to gain an understanding of which words in the sentence had the most influence on the negative sentiment?
The below chart is a result at my attempt to explore this, however I am not sure it makes sense and I do not think I am interpreting it correctly. I am basically taking the outputs of the last hidden layer, which in this case has shape (1, 7, 768), [CLS] + 5 word tokens + [SEP], and looping through each token summing up their values (768) and computing the average. The resulting totals are outputted in the below graph.

Any thoughts around if there is any meaning to this or if i am way off on approach, would be appreciated.  Might be my misunderstanding around the actual output values themselves.
Hopefully this is enough to give someone the idea of what i am trying to do and how each word can be connected to positive or negative associations that contributed to the final classification.

The sentiment would depend on the relation between words, not just the individual words. As I see it, the best approach with the problem as you phrased it would be to break down the sentence to individual tokens, and then find the sentiment of each word.

What method would you recommend for finding the sentiment of each word? I think that is what I am after here, but unsure of how to go about obtaining that information.

Since it is based on the individual word, it would have to either be stored for each word, or something could be trained on a dataset of sentences and sentiment analysis.


text:
Can we use genetic algorithms to evolve datasets?
Genetic algorithms are used to solve many optimization tasks.
If I have a dataset, can I evolve it with a genetic algorithm to create an evolved version of the same dataset?
We could consider each feature of the initial dataset as a chromosome (or individual), which is then combined with other chromosomes (features) to find more features. Is this possible? Has this been done?
I will like to edit the details with an example so that it is easier to understand.
Example: In practice cyber-security attacks evolve over time since it finds a new way to breach a system. The main draw-back of intrusion detection model is that it needs to be trained every time attack evolves. So I was hoping if genetic algorithm can be used on the present benchmarked datasets (like NSL-KDD) to come up with a futuristic type dataset maybe after X-number of generations.
And check if a model is able to classify that generated dataset as well.

I don't really understand the problem. Let's use the Iris dataset as an example. Your features are sepal length and width and petal length and width. So you make a bunch of four-chromosome individuals based on those features. Then what? What is the fitness of one of them and what makes it better or worse than another one?

@deong Initially, the OP used the term "parent" to refer to a feature, but I edited the post to use the term "chromosome", which is more general. I don't think I changed the meaning of the post, but I added the part "which is then combined with other chromosomes (features) to find more features", given that's what the OP seemed to suggest.

@deong I have edited the actual description with an example which may hep you understand my question much better. Because Iris-dataset is not the type which I am referring to.

This question raises a lot more questions. It seems like a solution looking for a problem, instead of the other way round.

How do you measure the fitness of a feature?
What would one of the "evolved datasets" mean? What does it represent?
What would your overal purpose be? If you just wish to generate simulated datasets, there are easier ways to do this, with more control over the various aspects of the resulting datasets.

If you want to compute a new set of features to "better" describe a given dataset, there are many approaches to this, such as PCA, ISOMAP, self-organizing maps, ... If this is the kind of thing you're thinking about, I would recommend starting there.

I have edited the description with a specific example since the example is more closely related to what I am thinking. Please take a look once. I guess that will surely help to understand what I am trying to say.

The paper Evolutionary Dataset Optimisation: learning algorithm quality through evolution (2019), by Henry Wilde et al., proposes a method to generate datasets with a genetic algorithm. Their goal is to generate data for which a particular algorithm performs well, in terms of a certain metric, so that to get more insights about this algorithm and why it performs well. The individuals of the population are datasets (so not features of the dataset!), which can be combined with a crossover operator or mutated. The details are explained in section 2 (page 4) and they also provide nice diagrams that summarise their descriptions.
The authors evaluate their approach on k-means (section 3, page 12) and they use the k-means objective function as the fitness function of the genetic algorithm.
They also developed a library edo that is freely available, so you can start to play with their approach.

I only read the paper up to page 12 (this is a long paper), so I don't know about their conclusion of their approach. I suggest that you read the paper yourself to know more about it.

Thanks. I will surely take a look at it what the paper is saying.

authors: Андрей Дубинец


text:
Improving DQN with fluctuations
Hello :) I'm pretty new to this community, so let me know if I posted anything incorrectly and I'll try to change it.
I'm working on the project which aim is to create self-driving agent in CARLA. I built a neural network Xception (decaying  ε-greedy). The other parameters are:

EPISODES: 100
GAMMA: 0.3
EPSILON_DECAY: 0.9
MIN_EPSILON: 0.001
BATCH: 16

Due to the limited computer resources I chose 100 or 300 epochs to train the model, but it generates much fluctuations:

EPISODES: 100
GAMMA: 0.7
EPSILON_DECAY: 0.9
MIN_EPSILON: 0.001
BATCH: 16

Can anyone suggest how can I improve my results? Or it is only the issue of small number of epochs?

how come your gamma is 0.3? This seems quite low, and would suggest you care more about immediate returns than future returns (not something that is typically the case in RL).

Changing otherwise to gamma = 0.7 didn't help with fluctuations. I'm not sure if gamma is the issue. Maybe it is connected with the small amount of epochs?

A more common value for gamma would be 0.9 or 0.99. I would try those.

@S2673 I'm not sure if this would be ok with autonomous vehicle...

What do you mean? Do you think it would be dangerous if it was willing to crash as long as it got somewhere? I would still try it anyway to see if the agent learns and then you can change gamma or the reward system.

It is not clear form your question, how you use your replay buffer. Basically, you have to store all states/reward tuples and train your agent on the entire buffer.
Moreover, you should give the agent time to explore (all) states of the world. But if you want to speed up training, you can try to implement importance sampling

But are those fluctuations in my example a kind of unnormal thing? Or are they acceptable for someone not very experienced in field of deep learning?

in my experience, fluctuations is common


text:
NLP Bible verse division problem: Whats the best model/method?
I'm working on a project compiling various versions of the Bible into a dataset. For the most part versions separate verses discreetly. In some versions, however, verses are combined. Instead of verse 16, the marker will say 16-18. I wonder if, given I have a lot of other versions that separate them discretely, I can train an NLP model (I have about 30 versions that could act as a training set which would constitute  to separate those combined verses into discrete verses. I'm fairly new at deep learning, having done a few toy projects. I wonder how to think about this problem? What kind of problem is it? I think it might be similar to auto-punctuation problems and it seems the options there are seq2seq and classifier. This makes more sense to me as a classification problem, but maybe my inexperience is what drives me that direction. Can people suggest ways to think about this problem and resources I might use?
In answer to questions in the comment, I am dealing only with text, not images. An example might be like this:
Genesis 2, New Revised Standard Version:

5 when no plant of the field was yet in the earth and no herb of the field had yet sprung up—for the Lord God had not caused it to rain upon the earth, and there was no one to till the ground; 6 but a stream would rise from the earth, and water the whole face of the ground— 7 then the Lord God formed man from the dust of the ground, and breathed into his nostrils the breath of life; and the man became a living being.

Genesis 2, The message version:

5-7 At the time God made Earth and Heaven, before any grasses or shrubs had sprouted from the ground—God hadn’t yet sent rain on Earth, nor was there anyone around to work the ground (the whole Earth was watered by underground springs)—God formed Man out of dirt from the ground and blew into his nostrils the breath of life. The Man came alive—a living soul!

The goal then would be to divide the message version into discrete verses in the way that the NRSV is. Certainly, a part of the guide would be that a verse always ends in some kind of punctuation, though while necessary it is not sufficient to assign a distinct verse.

Hi rwreed and welcome to ai.stackexchange.com! What is NRSV? What do you mean by discrete verses?

You wonder if you can train an NLP model. You can always train an NLP model if you have text data; the question is what you want it to do (what output you want it to give) and how well you want it to perform.

@HelloGoodbye Sorry for the use of jargon (an academics' blight) NRSV=New Revised Standard Version, it is one of the many versions of the Bible, though is the preferred English translation for most academics. By discrete verses, I hope to take verses that have been merged together (like in the Message Version above) and split them so its something like the NRSV example.


text:
What is the current artificial general intelligence technology valuation?
That is, if AGI were an existing technology, how much would it be valued to?
Obviously it would depend on its efficiency, if it requires more than all the existing hardware to run it, it would be impossible to market.
This question is more about getting a general picture of the economy surrounding this technology.
Assuming a specific definition of AGI and that we implemented that AGI, what is its potential economical value?
Current investments in this research field are also useful data.

Hello. Unfortunately, the question "That is, if AGI were an existing technology, how much would it be valued to?" will only lead to pure speculations, so it's not suited for this site. I suggest that you reformulate your question so that an objective answer can be given. Take a look at https://ai.stackexchange.com/help/on-topic for more info about our site.

Apparently you have little knowledge in economics. An upcoming technology can be valued. This is one of the reasons AGI gets investors. It will not be pure speculation if people answering my question are familiar with this industry.

How can you value something that does not yet exist? It's true that I don't much knowledge of economics, but can you explain how an answer can be given about the value of something that does yet exist? We will need to assume a definition of AGI, I guess, and from that predict its potential economic value. Is that what you mean? I suggest that you clarify this, although you say "feel free to take whatever assumption is necessary to give it a try", so I guess that you're asking: "Assuming a specific definition of AGI and that we implemented that AGI, what is its potential economical value?"

Actually I complied and asked a different question, I did not expect this to reopen. But this is good, thanks. I understand this can seem strange at first glance to value something that does not exist but this happens all the time when trying to decide to invest money in research. And AGI have real world applications, like substituting skilled workforce like engineers, etc. Okay I edit the question according to your suggestion if this helps. Thank you for keeping an open mind and helping.

@nbro: the definition of this site is "for people interested in conceptual questions about life and challenges in a world where cognitive functions can be mimicked in purely digital environment". It belongs to science group of stack exchange sites. Both things means this site is more open to opinions than technology ones. The plague of questions about NN could appear this site is about technology of AI, but it is not. In conclusion, not only this question is appropriate, it is one of the few in last days that it is.

@pasabaporaqui That description needs to be changed/updated. Where did you find it? The description of the site that you should take into account is: https://ai.stackexchange.com/help/on-topic.

@nbro: https://stackexchange.com/sites#science and click on AI site. I disagree the definition must be changed.

I will try to give some sense to this question.

Artificial general intelligence (AGI) is the hypothetical[1] intelligence of a machine that has the capacity to understand or learn any intellectual task that a human being can. It is a primary goal of some artificial intelligence research and a common topic in science fiction and futures studies. AGI can also be referred to as strong AI,[2][3][4] full AI,[5] or general intelligent action.[6] Some academic sources reserve the term "strong AI" for machines that can experience consciousness.

These are the first sentences on AGI on wikipedia (link), and the softest limit there is

[learn] any intellectual task that a human being can.

Even taking only this, it would mean that any AGI has infinite economic value. As soon as there is something that can learn any human task and has the speed of current GPUs/CPUs it could potentially immediatly replace every human in every task. There are certainly enough computers with CPUs and GPUs out there.
This question is still a little flawed because you not only have to constrain the definition of AGI but also how it would actually be implemented.

Possibly not infinite, but approaching the limit of infinity.  (I suspect physics would be the limiting factor, the speed of light, available material and energy, and so forth.)  This based on the idea of recursive self-improvement resulting in an intelligence explosion.

Our CPUs and GPUs are just a mean to simulate the structure that we think may lead to the intelligence. Isn't it? So if one day we will be able to understand how the human intelligence works and what's the structure to create the intelligence, why should we still use CPUs and GPUs to model them? Can't we make something specific to the task of intelligence? something like human brains? Then it will consume a little material and maybe we will be able to attack a processing unit to it, so the artificial brain has access to logical computation and can perform superhuman tasks.

The economic value would be high indeed, as, combined with robotics, AGI would be able to replace all human workers.  So:

Whatever the economic value of the sum of human labor is, in an ideal sense

Of course, there would also be the question of the cost of computation, the cost of the hardware & software required for AGI, and whether that cost is higher or lower than the cost of human labor.  (My guess is biological machines such as humans would be cheaper, both in production and processing, until AGI leverages molecular computing via an inexpensive, ubiquitous substrate.  Also worth noting that biological systems such as humans and canines may be more fault-tolerant, and more resilient in that they can persist even where the technological base collapses.)
Currently, cost of training even narrowly superintelligent Neural Networks which exceed humans at a single function is extremely high.


text:
What is the goal of weight initialization in neural networks?
This is a simple question. I know the weights in a neural network can be initialized in many different ways like: random uniform distribution, normal distribution, and Xavier initialization. But what is the weight initialization trying to achieve?
Is it trying to allow the gradients to be large so it can quickly converge? Is it trying to make sure there is no symmetry in the gradients? Is it trying to make the outputs as random as possible to learn more from the loss function? Is it only trying to prevent exploding and vanishing gradients? Is it more about speed or finding a global maximum? What would the perfect weights (without being learned parameters) for a problem achieve? What makes them perfect? What are the properties in an initialization that makes the network learn faster?

Finding the "perfect" anything of a black box model is not feasible. You can use a good initialisation method though. As an example, the goal of Xavier initialisation is to maintain a variance at the end of each layer equal to the incoming variance. This helps dramatically with keeping gradients reasonable

@Recessive I’m using perfect very loosely and also I am not asking what the perfect weights would be, but what they would achieve in speeding up learning.

It also depends on where the line is drawn. Transfer learning is a form of weight initialisation, and in that case that is an exceptional efficient initialiser. If you only consider random initialisation, I would suggest reading this: https://www.wandb.com/articles/the-effects-of-weight-initialization-on-neural-nets. It's quite comprehensive

@Recessive I actually just read that, but thanks. I am not only considering random initialization. I am not really looking for a specific strategy, but more what a perfect one would be supposed to achieve.

This article Why Initialize a Neural Network with Random Weights? should be useful.

Is it trying to make sure there is no symmetry in the gradients?

The aim of weight initialization is to make sure that we don't converge to a trivial solution. That's why we have different kinds of initialization depending on the dataset type. So, Yes it is trying to avoid symmetry.

Is it trying to allow the gradients to be large so it can quickly converge?

The time it takes to converge, is I think a property of the optimizer and not of the weights initialization. Of course, the manner in which we initialize our weights matters but I think Optimization Algorithms contribute more towards convergence

What are the properties in an initialization that makes the network learn faster?

Glorot and Bengio believed that Xavier weight initialization would maintain the variance of activations and back-propagated gradients all the way up or down the layers of a network. Incidentally, when they trained deeper networks that used ReLUs, it was found that a 30-layer CNN using Xavier initialization stalled completely and didn’t learn at all. Thus, it depends on the particular problem at hand.

So the goal is somewhat to get every neuron involved in gradient descent? Making sure none of their gradients are very small or zero?

Yes, the aim is to have a low number of sparse weights.

The most important thing we achieve is indeed making sure the weights are not all equal. If they were, every layer would behave as if it were a single cell.
We typically want weights that are near zero (so unimportant connections will not accidentally dominate) but non-zero.
The different types of initialization all have different motivations, including those mentioned in the question.
If you're curious what the motivation for each one is, I would recommend you check the documentation and try to find the original papers where they were first introduced.

So I am getting that one motivation is that unimportant connections will not dominate. I will look into the individual initialization strategies, but what would a perfect weight initialization achieve without having contradictory goals? Is it possible to combine the individual initialization goals?


text:
How to take gradient of log policy when actions are negative?
I am currently trying to train BipedalWalker of OpenAI gym by using policy gradient approach. My action space contains 4 continuous actions, all ranging [-1.0, 1.0]. In this case, how can we calculate gradient of log policy when log(negative_action) gives nan value.

You don’t want to take log of the action. You want the log likelihood of taking an action. If your network outputs parameters of a Gaussian distribution then you would want to evaluate $\log f(a_t| \mu, \sigma)$, where $f$ is the pdf of the standard normal. Note that a tanh activation is usually wrapped around the action when the action space is symmetric in these MuJoCo environments and so the gradient will change slightly as your distribution is now $Y = \text{tanh}(X)$.

Correct me if I am wrong. Mean, SD = Policy_Model(current_state). Then, action = random.gauss(mean, SD). Then probability_density_function = exp(-0.5((x-mu)/SD)^2)/(SD√2pi). Then, log of this function then gradient. Also, I didn't get why we use tanh in the policy_models last layer.

Why are we using tanh for getting mu and sd.

You are correct that action comes from sampling the Gaussian. However, the support of the Gaussian random variable is $\mathbb{R}$, so your action can be any real number. As you want it to be constrained to $[-1, 1]$ you apply the tanh function to enforce this.

Am I correct about the probability density function. Also, should I use separate gaussian parameters for all four i.e.,  [mu1, mu2, mu3, mu4, sig1, sig2, sig3, sig4] = policy_model(current_state) Or should I use multivariate guassian distribution which has a covariance matrix.

Yes, you had the details of the Gaussian pdf correct.
Typically what is done is use a multi variate Gaussian with diagonal covariants matrix — this is equivalent to using independent gaussians.

Correct me I am wrong. I need to use tanh at the end of policy network to limit the mu, SD. Or do I need to use tanh on the sampled actions. Can we use any other non linearity in place of tanh, like clamped.

You have to use it on the sampled actions. You could use a clamped activation, but this would make the resulting pdf more complex than using tanh.

If i use tanh on limiting the parameters, the training goes normal but I've not got any useful results so far. But if I use tanh on actions the gradients become unstable, they become either 0 of Infinity. What should I do then?

Let us continue this discussion in chat.

I will add a more detailed answer if/when I get the opportunity including some tips for implementation details, hopefully within 1-2 days.

The policy gradient tells us that
$$\nabla_\theta v_\pi(s) = \mathbb{E}_\pi\left[ G_t \nabla_\theta \log \pi_\theta(a | s) \right] \; ;$$
where as usual $G_t$ are the returns, $\pi$ is policy parameterised by $\theta$ and the expectation is taken with respect to the policy $\pi$ and the state distribution induced by $\pi$.
Hopefully you can see that here we do not need to take the log of the actions, but instead the log of the probability of taking an action (or, in a continuous action case, the log of the pdf of taking an action), i.e. the log-likelihood of an action.
In your case, as with most MuJoCo environments, it is common to train a neural network to output the mean and standard deviation of a Gaussian distribution -- assuming the action has $d$ dimensions, this is done by having a neural network with two heads, i.e. a neural network that takes the input and outputs two $d$ length vectors. Note that in practice, it is common for the network to output the log-standard deviation for stability reasons. Further, it is typically assumed that the log-sd is clamped to some pre-specified range too, again to help with numerical stability. For $d > 1$ we assume that the Multi-Variate Gaussian has diagonal covariance matrix, so the neural network still need only predict a $d$ length vector for the log-sd, as opposed to a full covariance matrix.
Now, as the action range for the MuJoCo environments is typically symmetric, it is also common to see that a tanh activation is applied to the action to get the action to be in the range (-1, 1). This makes things a bit more nuanced in terms of how to apply the policy gradient, because your random variable (the action) is no longer gaussian. Luckily, as we have assumed independent gaussians, the resulting distribution of the random variable is not that tricky to calculate analytically.
If we assume that $A$ is the random variable representing the action, then $A$ is defined as $A = \tanh(X)$, where $X \sim \mbox{MVN}(\boldsymbol{\mu}_\theta, \boldsymbol{\Sigma}_\theta)$, where $\boldsymbol{\mu}, \boldsymbol{\Sigma}$ are the predicted mean and (diagonal) co-variance parameters of the gaussian, with the $\theta$ subscript added to remind us of the fact that they are predicted by the NN parameterised by $\theta$. Now, we can use the fact that the pdf of an arbitrary (one-to-one) function of a random variable defined by $Y = g(X)$ is given (for the univariate case) by
$$f_Y(y) = f_X\left(g^{-1}(y)\right) \left|\frac{dx}{dy}\right| \; ;$$
note that the multivariate case is a simple extension of this and details can be found online.
In our example, $g = \tanh$, and so the pdf of the policy is given by
$$\pi_\theta(\textbf{a}|s) = \phi_\theta(\textbf{x}|s)\left|\mbox{det}\left(\frac{d\textbf{x}}{d\textbf{a}}\right)\right| = \phi_\theta(\textbf{x}|s)\left|\mbox{det}\left(\frac{d\textbf{a}}{d\textbf{x}}\right)\right|^{-1} \; ;$$
where $\textbf{x} \sim X$ and $\phi$ is the pdf of the MVN with $\theta$ notation to imply that the mean and covariance come from the NN prediction. Note that we invert the determinant of the transformation because it is often easier to calculate the derivative this way; in the case of $tanh$ being the transformation function along with the fact that we assume the MVN has diagonal covariance matrix we get that $\frac{d\textbf{a}}{d\textbf{x}} = \mbox{diag}(1 - \tanh^2(\textbf{x}))$. Now, using this, you can evaluate $\log \pi_\theta(a|s)$ in the policy gradient term correctly by using the correct pdf.

I understood up to the 5th para. I couldn't understand what is fy = fx(g(x)|dx/dy|. Can you please elaborate more or suggest me some content to understand that?

If you go to the wiki page for a random variable and click the section ‘function of a random variable’ it should explain what this paragraph means, and also has some examples you can work through.

I have updated my question and included the code. Please help me find the reason why my agent is note converging.


text:
Estimating an $n$-Gram model using on bigrams
One of the main arguments against $n$-gram models is that, as $n$ increases, there is no way to compute $P(w_n|w_1,\cdots,w_{n-1})$ from training data (since the chance of visiting $w_n,...,w_1$ is practically zero).
Wondering why we cannot estimate $P(w_n|w_1,\cdots,w_{n-1})$ using the following:
Let $P_i(u|v)$ be the probability of having sequences where word $u$ comes exactly $i$ words after word $v$ (This is easy to compute).
Then we can esitmate $P(w_n|w_1,\cdots,w_{n-1})$ as a function of $P_i(u|v)$. I could not find any reference to such approach in the literature. The most similar approach is the smoothing/backoff methods.
Is there any reason why no-one used this approach? Or if one can share some previous work about this approach.
P.S.1. The disadvantage of this approach, comparing with standard $n$-gram model, is its running time.
P.S.2. We could use bucketing idea: Instead of computing/storing/using $P_i$, for every $i$, we can compute/store/use $PB_{i}=P_{2^i}$ . Then $P_i(u|v) \approx PB_{\log i}(u|v)$.

There is another downside to your suggested approach. It is not as representationally powerful as standard $n$-grams. For example, consider 3-grams on the alphabet ${0, 1}$ where the third value is always equal to the XOR of the first 2. With 3-grams, we can represent this, but with your suggested approach, we cannot. It's a bit more like a Naive Bayes-version in that respect. It might work well enough in practice, though.

@RobbyGoetschalckx That's right: As I said, the proposed approach is an estimator for n-gram model.


text:
Why does Q-value become negative during training of DQN, while the agent learns to play?
I have implemented a simple version of the DQN algorithm for CartPole-v0. The algorithm works fine, in the sense that achieves the highest possible scores. The below diagram shows the cumulative reward versus training episode.

The scary part is when I tried to plot the q values during training. For this purpose, 1000 random states were generated and stored. After each training episode, I fed these states to the Q-network and computed the Q-value for all actions. Then for each state, the max Q-value was computed. Finally, these max Q-values were averaged to yield a single number. The following plot shows this quantity during training.

As you can see, there is a slight increase in average-max-q in the beginning and it is followed by a sharp decrease. My question is, how can this value be negative? Since I know all rewards received by the agent are positive, and expected cumulative reward of a state can not be negative.
Edit
I added the code for clarity:
from matplotlib import pyplot as plt

import torch as th
import torch.nn as nn
import torch.nn.functional as F

import gym
import random
import numpy as np
from collections import deque

from test_avg_q import TestEnv

class ReplayBuffer():
    def __init__(self, maxlen):
        self.buffer = deque(maxlen=maxlen)

    def add(self, experience):
        self.buffer.append(experience)

    def sample(self, batch_size):
        sample_size = min(len(self.buffer), batch_size)
        samples = random.choices(self.buffer, k=sample_size)
        return map(list, zip(*samples))

class QNetwork():
    def __init__(self, state_dim, action_size):
        self.action_size = action_size
        self.q_net = nn.Sequential(nn.Linear(state_dim, 100),
                                   nn.ReLU(),
                                   nn.Linear(100, action_size))
        self.optimizer = th.optim.Adam(self.q_net.parameters(), lr=0.001)

    def update_model(self, state, action, q_target):
        action = th.Tensor(action).to(th.int64)
        action_one_hot = F.one_hot(action, num_classes=self.action_size)
        q = self.q_net(th.Tensor(state))
        q_a = th.sum(q * action_one_hot, dim=1)
        loss = nn.MSELoss()(q_a, q_target)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

class DQNAgent():
    def __init__(self, env):
        self.state_dim = env.observation_space.shape[0]
        self.action_size = env.action_space.n
        self.q_network = QNetwork(self.state_dim, self.action_size)
        self.replay_buffer = ReplayBuffer(maxlen=1_000_000)
        self.gamma = 0.97
        self.eps = 1.0

    def get_action(self, state):
        with th.no_grad():
            q_state = self.q_network.q_net(th.Tensor(state).unsqueeze(0))
            action_greedy = th.argmax(q_state).item()
        action_random = np.random.randint(self.action_size)
        action = action_random if random.random() < self.eps else action_greedy
        return action

    def train(self, state, action, next_state, reward, done):
        self.replay_buffer.add((state, action, next_state, reward, done))
        states, actions, next_states, rewards, dones = self.replay_buffer.sample(50)
        with th.no_grad():
            q_next_states = self.q_network.q_net(th.Tensor(next_states))
            q_next_states[dones] = th.zeros(self.action_size)
            q_targets = th.Tensor(rewards) + self.gamma * th.max(q_next_states, dim=1)[0]
        self.q_network.update_model(states, actions, q_targets)

        if done: self.eps = max(0.1, 0.99 * self.eps)

env_name = "CartPole-v0"
env = gym.make(env_name)

agent = DQNAgent(env)
num_episodes = 400
testEnv = TestEnv()
avg_qs = []
rewards = []
render = False

for ep in range(num_episodes):
    state = env.reset()
    total_reward = 0
    done = False
    while not done:
        action = agent.get_action(state)
        next_state, reward, done, info = env.step(action)
        agent.train(state, action, next_state, reward, done)
        total_reward += reward
        state = next_state
        if render and ep > 80:
            env.render()

    avg_qs.append(testEnv.run(agent.q_network.q_net))
    rewards.append(total_reward)
    print("Episode: {}, total_reward: {:.2f}".format(ep, total_reward))

plt.plot(avg_qs)
plt.show()
plt.figure()
plt.plot(rewards)
plt.show()

here is the code for class TestEnv:
import torch as th
from torch.utils.data import Dataset, DataLoader

class TestEnv:
    def __init__(self):
        self.dataloader = DataLoader(TestDataset(), batch_size=100, num_workers=10)

    def run(self, model):
        out_list = []
        model.eval()
        with th.no_grad():
            for batch in self.dataloader:
                out = model(batch)
                out_list.append(out)
            qs = th.cat(out_list, dim=0)
            maxq = th.max(qs, dim=1)[0]
            result = th.mean(maxq)
        return result.item()

class TestDataset(Dataset):
    def __init__(self):
        self.db = th.load('test_states.pt')

    def __len__(self):
        return len(self.db)

    def __getitem__(self, idx):
        return self.db[idx]

Maybe link your implementation. I would suspect a bug in your code somewhere, as you point out accurate Q values for real states should at least be positive. I don't think it is possible to predict where the bug is from the graphs.

for me, it would be possible that your 1000 random states are not a good representation of the gigantic state space of the problem, most states in the problem yield very bad outcome, only a part of them can be considered of interest (e.g. the ones where the pole is more of less straight).
if your 1000 all lead to certain fail, the max expected reward for those states can only go down, because you never want to reach them, nor their child.

I'm having a similar problem (mean q-value increases and then decreases). Did you figure this out?

Look up the deadly triad. There is a lot of literature devoted to the overestimation of Q values.


text:
How to design an observation(state) space for a simple `Rock-Paper-Scissor` game?
For weeks I've been working with this toy game of Rock-Paper-Scissor. I want to use a PPO agent learn to beat a computer opponent whose logic is defined as the code bellow.
For short, this computer opponent, named abbey, uses a strategy that tracks all two consecutive plays of the agent, and gives the opposite play of the most likely guess of the agent's next play, according to the agent's last play.
I design the agent(using gym env) to have an internal state, keeping track of all counts of its two consecutive plays, in a 3x3 matrix. And then I normalized each row of the matrix to be an observation of the agent, representing the probabilities of the second play given the previous one. So the agent will get the same knowledge as what abbey knows.
Then I copied an PPO network algorithm from some RL book, which works well with CartPole. Then I did some minor changes which are commented in the code bellow.
But the algorithm does not converge even a little, and abbey always wins the agent about 60% of the time from first run to the last.
I doubt the state and observation space I designed is the reason why it does not converge. All I get is that the agent maybe should find something from the histories of its own successes and fails, and find its way out.
Can you give me some advice for the designing of a state space?
Thank you very much.
### define a Rock-Paper-Scissor opponent

abbey_state = []
play_order=[{
              "RR": 0,
              "RP": 0,
              "RS": 0,
              "PR": 0,
              "PP": 0,
              "PS": 0,
              "SR": 0,
              "SP": 0,
              "SS": 0,
          }]
def abbey(prev_opponent_play,
          re_init=False):
    if not prev_opponent_play:
        prev_opponent_play = 'R'
    global abbey_state, play_order
    if re_init:
        abbey_state = []
        play_order=[{
              "RR": 0,
              "RP": 0,
              "RS": 0,
              "PR": 0,
              "PP": 0,
              "PS": 0,
              "SR": 0,
              "SP": 0,
              "SS": 0,
          }]
    abbey_state.append(prev_opponent_play)
    last_two = "".join(abbey_state[-2:])
    if len(last_two) == 2:
        play_order[0][last_two] += 1
    potential_plays = [
        prev_opponent_play + "R",
        prev_opponent_play + "P",
        prev_opponent_play + "S",
    ]
    sub_order = {
        k: play_order[0][k]
        for k in potential_plays if k in play_order[0]
    }
    prediction = max(sub_order, key=sub_order.get)[-1:]
    ideal_response = {'P': 'S', 'R': 'P', 'S': 'R'}
    return ideal_response[prediction]

### define the gym env
import gym
from gym import spaces
from collections import defaultdict
import numpy as np

ACTIONS = ["R", "P", "S"]
games = 1000

class RockPaperScissorsEnv(gym.Env):
  metadata = {'render.modes': ['human']}

  def __init__(self):
    super(RockPaperScissorsEnv, self).__init__()
    self.action_space = spaces.Discrete(3)
    self.observation_space = spaces.Box(low=0.0, high=1.0,
                                        shape=(3,3), dtype=float)
    self.reset()

  def step(self, actions):
    assert actions == 0 or actions == 1 or actions == 2
    opponent_play = self.opponent_play()

    self.prev_plays[self.prev_actions * 3 + actions] += 1
    reward = self.calc_reward(actions, opponent_play)
    terminal = False

    self.calc_state(self.timestep, opponent_play, actions)
    self.prev_actions = actions
    self.prev_opponent_play = opponent_play
    self.timestep += 1
    return self.get_ob(), reward, terminal, None

  def reset(self):
    self.opponent = abbey
    self.timestep = 0
    self.prev_opponent_play = 0
    self.prev_actions = 0
    self.prev_plays = defaultdict(int)
    self.init_state = np.zeros((3,3), dtype=int)
    # the internal state
    self.state = np.copy(self.init_state)
    self.results = {"win": 0, "lose": 0, "tie": 0}
    return self.get_ob()

  def render(self, mode='human'):
    pass

  def close (self):
    pass

  def calc_reward(self, actions, play):
    if self.timestep % games == games - 1:
      pass
    if actions == play:
      self.results['tie'] += 1
      return 0
    elif actions == 0 and play == 1:
      self.results['lose'] += 1
      return -0.3
    elif actions == 1 and play == 2:
      self.results['lose'] += 1
      return -0.3
    elif actions == 2 and play == 0:
      self.results['lose'] += 1
      return -0.3
    elif (actions == 1 and play == 0) or (actions == 2 and play == 1) or (actions == 0 and play == 2):
      self.results['win'] += 1
      return 0.3
    else:
      raise NotImplementedError('calc_reward something get wrong')

  def opponent_play(self):
    re_init = (self.timestep == 0)
    opp_play = self.opponent(ACTIONS[self.prev_actions], re_init=re_init)
    return ACTIONS.index(opp_play)

  def calc_state(self, timestep, opponent_play, actions):
    self.state[self.prev_actions][actions] += 1

  def get_ob(self):
    '''return observations'''
    state0 = self.state[0]
    sum0 = state0.sum()
    state1 = self.state[1]
    sum1 = state1.sum()
    state2 = self.state[2]
    sum2 = state2.sum()
    init = np.ones(3, dtype=float) / 3.0
    ob = np.array([
      state0 / sum0 if sum0 else init,
      state1 / sum1 if sum1 else init,
      state2 / sum2 if sum2 else init,
    ])
    # print(ob)
    return ob

### Learning Algo copied from some book

import  matplotlib
from    matplotlib import pyplot as plt
matplotlib.rcParams['font.size'] = 18
matplotlib.rcParams['figure.titlesize'] = 18
matplotlib.rcParams['figure.figsize'] = [9, 7]
matplotlib.rcParams['axes.unicode_minus']=False

plt.figure()

import  gym,os
import  numpy as np
import  tensorflow as tf
from    tensorflow import keras
from    tensorflow.keras import layers,optimizers,losses
from    collections import namedtuple
env = RockPaperScissorsEnv()
env.seed(2222)
tf.random.set_seed(2222)
np.random.seed(2222)
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
assert tf.__version__.startswith('2.')

gamma = 0.98
epsilon = 0.2
batch_size = 32

Transition = namedtuple('Transition', ['state', 'action', 'a_log_prob', 'reward', 'next_state'])

class Actor(keras.Model):
    def __init__(self):
        super(Actor, self).__init__()
        self.fc1 = layers.Dense(18, kernel_initializer='he_normal') # I changed 100 to 18
        self.fc2 = layers.Dense(3, kernel_initializer='he_normal') # I changed 4 to 3

    def call(self, inputs):
        x = tf.nn.relu(self.fc1(inputs))
        x = self.fc2(x)
        x = tf.nn.softmax(x, axis=1)
        return x

class Critic(keras.Model):
    def __init__(self):
        super(Critic, self).__init__()
        self.fc1 = layers.Dense(18, kernel_initializer='he_normal') # I changed 100 to 18
        self.fc2 = layers.Dense(1, kernel_initializer='he_normal')

    def call(self, inputs):
        x = tf.nn.relu(self.fc1(inputs))
        x = self.fc2(x)
        return x

class PPO():
    def __init__(self):
        super(PPO, self).__init__()
        self.actor = Actor()
        self.critic = Critic()
        self.buffer = []
        self.actor_optimizer = optimizers.Adam(1e-3)
        self.critic_optimizer = optimizers.Adam(3e-3)

    def select_action(self, s):
        s = tf.constant(s, dtype=tf.float32)
        # s = tf.expand_dims(s, 0)   # I removed this line, otherwise we will get a (1,3,3) tensor and later we will get an error
        prob = self.actor(s)
        a = tf.random.categorical(tf.math.log(prob), 1)[0]
        a = int(a)
        return a, float(prob[0][a])

    def get_value(self, s):
        s = tf.constant(s, dtype=tf.float32)
        s = tf.expand_dims(s, axis=0)
        v = self.critic(s)[0]
        return float(v)

    def store_transition(self, transition):
        self.buffer.append(transition)

    def optimize(self):
        state = tf.constant([t.state for t in self.buffer], dtype=tf.float32)
        action = tf.constant([t.action for t in self.buffer], dtype=tf.int32)
        action = tf.reshape(action,[-1,1])
        reward = [t.reward for t in self.buffer]
        old_action_log_prob = tf.constant([t.a_log_prob for t in self.buffer], dtype=tf.float32)
        old_action_log_prob = tf.reshape(old_action_log_prob, [-1,1])

        R = 0
        Rs = []
        for r in reward[::-1]:
            R = r + gamma * R
            Rs.insert(0, R)
        Rs = tf.constant(Rs, dtype=tf.float32)

        for _ in range(round(10*len(self.buffer)/batch_size)):

            index = np.random.choice(np.arange(len(self.buffer)), batch_size, replace=False)

            with tf.GradientTape() as tape1, tf.GradientTape() as tape2:

                v_target = tf.expand_dims(tf.gather(Rs, index, axis=0), axis=1)

                v = self.critic(tf.gather(state, index, axis=0))
                delta = v_target - v
                advantage = tf.stop_gradient(delta)
                a = tf.gather(action, index, axis=0)
                pi = self.actor(tf.gather(state, index, axis=0))
                indices = tf.expand_dims(tf.range(a.shape[0]), axis=1)
                indices = tf.concat([indices, a], axis=1)
                pi_a = tf.gather_nd(pi, indices)
                pi_a = tf.expand_dims(pi_a, axis=1)
                # Importance Sampling
                ratio = (pi_a / tf.gather(old_action_log_prob, index, axis=0))
                surr1 = ratio * advantage
                surr2 = tf.clip_by_value(ratio, 1 - epsilon, 1 + epsilon) * advantage
                policy_loss = -tf.reduce_mean(tf.minimum(surr1, surr2))
                value_loss = losses.MSE(v_target, v)
            grads = tape1.gradient(policy_loss, self.actor.trainable_variables)
            self.actor_optimizer.apply_gradients(zip(grads, self.actor.trainable_variables))
            grads = tape2.gradient(value_loss, self.critic.trainable_variables)
            self.critic_optimizer.apply_gradients(zip(grads, self.critic.trainable_variables))

        self.buffer = []

def main():
    agent = PPO()
    returns = []
    total = 0
    for i_epoch in range(500):
        state = env.reset()
        for t in range(games):
            action, action_prob = agent.select_action(state)
            if t == 999:
              print(action, action_prob)
            next_state, reward, done, _ = env.step(action)
            # print(next_state, reward, done, action)
            trans = Transition(state, action, action_prob, reward, next_state)
            agent.store_transition(trans)
            state = next_state
            total += reward
            if done:
                if len(agent.buffer) >= batch_size:
                    agent.optimize()
                break
        print(env.results)

        if i_epoch % 20 == 0:
            returns.append(total/20)
            total = 0
            print(i_epoch, returns[-1])

    print(np.array(returns))
    plt.figure()
    plt.plot(np.arange(len(returns))*20, np.array(returns))
    plt.plot(np.arange(len(returns))*20, np.array(returns), 's')
    plt.xlabel('epochs')
    plt.ylabel('total return')
    plt.savefig('ppo-tf.svg')

if __name__ == '__main__':
    main()
    print("end")

```


text:
How parameter adjustment works in Gradient Descent?
I am trying to comprehend how the Gradient Descent works.
I understand we have a cost function which is defined in terms of the following parameters,
$J(_{1},_{2},.... , w_{n}, b)$
the derivative would tell us which direction to adjust the parameters.
i.e. $\dfrac{dJ(_{1},_{2},.... , w_{n}, b)}{d(_{1}}$ is the rate of change of the cost w.r.t $$
The lecture kept on saying this very valuable as we are asking the question how should I change $$ to improve the cost?
But then the Lecturer presented $w_{1}$, $w_{2}$, ... as scaler value, How can we differentiate a scalar value.
I am fundamentally missing what is happening.
Can anyone please guide me to any blog post, a book that I should read to understand better?

Imagine we have the curve $f(x) = x^2$, and we want to find the minimum of this function. The derivate of $f$ with respect to $x$ is $2x$. Now, gradient descent works by updating our current estimate of the minimum, say $c_t$, by the following iterative process
$$c_{t+1} = c_t - \alpha \times \nabla_xf(x=c_t),$$
where $\alpha$ is some constant to control how much we shift towards the gradient.
Intuitively, this should make sense. Imagine our current estimate of the minimum is $c_t = -1$. The update would then give us $c_{t+1} = -1 - \alpha \times -2 = -1 + 2\alpha > -1$. As you can see, the update has shifted our estimate in the direction of the minimum. If our estimate were $+1$ then you can probably see that the update would again have shifted us in the direction of the minimum again.
Now, what happens in machine learning is we have a loss function $L$ that we typically want to find the minimum of in terms of the parameters of our model. By applying gradient descent on the loss function as we did above with $f(x)$ we iteratively apply the update rule which will eventually lead us to the minimum of our loss function with respect to the weights. The process is exactly the same as above except it is likely to happen in higher dimensions, where the derivative of $f$ becomes a vector of partial derivates. Note that whilst $w_i$'s are scalar values, we are not differentiating these values, rather we are differentiating the loss function with respect to these scalars.
I would recommend you to try this out with a simple linear model of maybe 2 or 3 parameters. After a quick Google I found this article that may be useful.


text:
What is the best GNN for a NMT task?
I am doing a machine translation task using a Graph2Seq graph neutral network.
There are many different variants of GNN:

GCN
GAT
GraphSage
GGNN

Which one would be the most effective for a machine translation task?
I have around 400 nodes in the graph, and on average, a node is connected to 3.5 neighbours.

The choice of GNN depends highly on the nature of the graphs you are dealing with, so the information you are giving is not enough to argue about which architecture might be best: How do you construct the graphs? How large are the graphs and what is the size of the node neighborhoods are some questions to consider. Even then you can only assume which GNN might work, you will still have to do empirical evaluations on different GNNs to be sure

@Chillston added information about the graph

I will start by saying that I do not have any experience with Graph2Seq networks or GGNN, but I have some knowledge about GNN in general and the other three architectures.
Firstly, it is essential to define your definition of effectiveness because each of these architectures excels at different things. For instance, GraphSage is most of the time the least expressive of these networks but will scale best to larger graphs, something that GGNN and GAT can not do as easily on most computers.
In most cases, it is best to look at previous literature to see which architecture they used for their implementation. Lastly, it is good to experiment with each architecture. For example, you could start with GCN and see how the performance is. If the GCN is accurate enough, stay with it or try GraphSage, which takes less computation and is thus, in most cases, faster. If GCN is not accurate enough, I recommend trying either GAT or GGNN, which should be more expressive.
Explanation Expressivity
In the context of GNN, expressivity means how much relevant information and patterns the GNN architecture can capture from the input graph. GraphSage has the worst expressivity because it does not use all the node neighbours when aggregating information, but only samples some of the node neighbours. Sampling is used because the belief is that most nodes in a neighbourhood share similar node features. However, this does not have to be the case and could mean that GraphSage will miss important information.
GAT is the opposite of this. GAT uses the information of all its neighbours in the same way as a GCN does and weighs the importance of those features with an attention mechanism. These two aspects make it more expressive because it will use all the available information and weigh the importance of neighbouring node features. Important information will be captured more easily by GAT for this reason.
Based on your graph
Your graph is quite small and should work fine with either GAT, GGNN or GCN. GraphSage is designed for significantly larger graphs and does not work with edge attributes. Again, I advise you to look at previous literature about Graph2Seq, to see what kind of architecture they use. Also, add more information about the graph like if it is directed or undirected, and information about if it only has node features or also edge weighs and attributes.

would you mind you explain the expressiveness of GCN, GAT and GGNN in a bit more detail. Would really appreciate it.

would you mind to explain why some of the GNN variants are not expressive enough?

I edited my initial answer to include more information about expressiveness.

I have been struggling firstly where to ask this question. Secondly finding materials about this topic. Appreciate it.

I am using a directed graph.

Can you pls give me a paper that rates GNN in terms of expressivity. Is expressivity a common term?

also what about GGNN. Is GGNN more expressive than GAT?


text:
What is the difference between prompt tuning and prefix tuning?
I read prompt tuning and prefix tuning are two effective mechanisms to leverage frozen language models to perform downstream tasks. What is the difference between the two and how they work really?
Prompt Tuning: https://aclanthology.org/2021.emnlp-main.243/
Prefix-Tuning: https://arxiv.org/abs/2101.00190


text:
What is the difference between fine tuning and variants of few shot learning?
I am trying to understand the concept of fine-tuning and few-shot learning.
I understand the need for fine-tuning. It is essentially tuning a pre-trained model to a specific downstream task.
However, recently I have seen a plethora of blog posts stating zero-shot learning, one-shot learning and few-shot learning.
How are they different from fine-tuning? It appears to me that few-shot learning is a specialization of fine-tuning. What am I missing here?
Can anyone please help me?

@nbro no. They don't discuss about few shot learning and their difference.

Ok, then edit your post to explain that you're just interested in few-shot learning and why that other post doesn't answer your question.

I believe the standard meanings are as follows, but not everyone uses words in the same way, so you might see examples that differ.
Fine tuning refers to slightly changing the weights of a pre-trained model (i.e. using a low learning rate) to adapt to a specific task.
Few shot learning refers to using a very small dataset to adapt to a specific task.
Someone might do both at the same time (fine-tuning with a small dataset), just fine-tuning (with a big dataset), or just few shot learning (e.g. giving GPT a few examples to prompt a particular kind of answer).


text:
Estimating $\sigma_i$ according to maximum likelihood method
Let be a Bayesian multivariate normal distribution classifier with distinct covariance matrices for each class and isotropic, i.e. with equal values over the entire diagonal and zero otherwise, $\mathbf{\Sigma}_i=\sigma_i^2\mathbf{I},~\forall i$.
How can I compute the equation for estimating the parameter $\sigma_{i}$ by the maximum likelihood method? Here $\sigma_{i,j}$ is is the covariance between $x_i$ and $x_j$. So $\sigma_i$ is just the variance of $x_i$.
Attempt:
Suppose $\mathcal{X}_i = \{x^t_i\}^N_{t=1}$ i.i.d, $x_i^t$ is in the class $C_i$ and $x_i^t \sim \mathcal{N}(\mu, \sigma^2)$.
Do I have to find the log-likelihood under $p(x) = \frac{1}{\sqrt{2 \pi} \sigma} \exp \left[ -\frac{(x-\mu)^2}{2 \sigma^2}\right]$, find the derivative and put it equal to $0$ to find the maximum?
EDIT
Suppose my data points are $m$-dimensional, and I have $K$ classes.

Comments are not for extended discussion; this conversation has been moved to chat.

How can your data points be $m$ dimensional if they follow a univariate normal distribution?


text:
Why does my "entropy generation" RNN do so badly?
I'm new to relatively RNNs, and I'm trying to train generative and guessing neural networks to produce sequences of real numbers that look random. My architecture looks like this (each "circle" in the output is the adverserial network's guess for the generated circle vertically below it -- having seen only the terms before it):

Note that the adverserial network is rewarded for predicting outputs close to the true values, i.e. the loss function looks like tf.math.reduce_max((sequence - predictions) ** 2) (I have also tried reduce_mean).
I don't know if there's something obviously wrong with my architecture, but when I try to train this network (and I've added a reasonable number of layers), it doesn't really work very well.

Colab Notebook

If you look at the result of the last code block, you'll see that my generative neural network produces things like

[0.9907787, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827]

But it could easily improve itself by simply training to jump around more, since you'll observe that the adverserial network also predicts numbers very close to the given number (even when the sequence it is given to predict is one that jumps around a lot!).
What am I doing wrong?

If you are trying for purely random numbers, how do you determine what a "true" value would be to measure loss?

I think there are two problems with your network.
The first one, always having very similar outputs, is the rather simple one. As it seems, your network suffers from the so-called, very common Mode Collapse problem. The attached link provides both an explanation and some potential remedy to that problem.
The second problem is more fundamental. You say that you want your network to produce random numbers. Or numbers that at least appear as such. However, once training is finished, your model is going to be a static function which will not change any further. Given the same input x, it will always produce the same output y. Consequently, unless the inputs to your network contain some randomness already or are, at least, always slightly dissimilar, you will not end up having a random generator. So, whether that is going to be useful will depend on your usecase. But if you make sure that the true random variable (like date&time) serves as input to the RNN and the RNN just translates this into some different format, that might work again. Just keep in mind that randomness can never arise out of a trained model.

Thanks, I'll have a look at your link. Regarding randomness -- I agree, but the point of my demonstration is to partially illustrate the fact that trained models, like human brains, are poor sources of entropy. Nonetheless, I would expect that they can be as good as human brains are, and at least produce sequences that aren't completely trivial to predict.

A RNN could theoretically emulate a simple PRNG, and I think that would serve OP's need. PRNGs are also deterministic and work with a starting seed value. The OP wants to generate sequences that look random, they have not asked for "true random", so I think a PRNG would be OK. However, I don't know how to create an RNN that could behave like a PRNG - what architecture and training would be likely to work.

The discriminator is going to have all the same data the generator does, so it should be able to predict all the numbers if it just copies the generator. But one hope is that perhaps it won't be able to converge towards the same weights just by seeing the output.

I suspect a fundamental problem here is how you determine what the correct outputs should be so that you can measure loss.  It's a very subjective problem.  Could the next number be equal to the last?  Certainly.  Could it be much higher?  Sure.  Much lower? Absolutely.  It sounds like OP is trying to capture the impression someone would have looking at the data.


text:
What is the difference between distant supervision and self-supervision?
Weak supervision is supervised learning, with uncertainty in the labeling, e.g. due to automatic labeling or because non-experts labelled the data [1].
Distant supervision [2, 3] is a type of weak supervision that uses an auxiliary automatic mechanism to produce weak labels / reference output (in contrast to non-expert human labelers).
According to this answer

Self-supervised learning (or self-supervision) is a supervised learning technique where the training data is automatically labelled.

In the examples for self-supervised learning, I have seen so far, the labels were extracted from the input data.
What is the difference between distant supervision and self-supervision?

Is it that for self-supervision, the labels must come from the input data and for distant supervision it can come from anywhere (which would make self-supervision a type of distant supervision)?
Or must the labels from distant supervision come from somewhere else than the input data?
If "In robotics, this can be done by finding and exploiting the relations or correlations between inputs coming from different sensor modalities." then for self-supervised learning, the labels do not even have to originate from the input data. (Or did I misinterpret the quote?)

(Setup mentioned in discussion:

Comments are not for extended discussion; this conversation has been moved to chat.

I have updated my answer. Anyway, after having read something about distant supervision, it seems like it refers to learning where the data comes from a remote database rather than a local dataset (and so the term "distant supervision" makes sense). In any case, this term is definitely less common than SSL.

@nbro: Which articles have you read regarding distant supervision? Maybe I can also have a look.

The main difference between distant supervision (as described in the link you provided) and self-supervision lies on the task the network is trained on.
Distant supervision focuses on generating weak labels for the very same task that would be tackled with supervised labels, and the final result could be directly used for that matter.
Self-supervision is a means for learning a data representation. It does so by learning a surrogate task, which is defined by inputs and labels derived exclusively from the original input data.
I can imagine cases in which an implementation of self-supervision could be considered distant supervision (if the task casually matches the target task). On the other hand, if external data sources would be employed for training on a surrogate task, that would be a case of representation learning (that could incorporate self-supervision too).

Ok, taking https://ai.stackexchange.com/questions/10623/what-is-self-supervised-learning-in-machine-learning/13749?noredirect=1#comment34138_13749 into account, the surrogate task is not the requirement, but learning from the input data is. While distant learning seems to be about learning from external data. I got that much. Now, the example of https://ai.stackexchange.com/a/10624/38174 would then not be self-supervised... unless you say that the proximity sensor is part of the "input", but then anything could be "part of the input".

I would say the input must be the input of the target class, or not? But then, with a surrogate task, the "input" where the labels are extracted from, is not the input of the target class. E.g. the documents of word2vec would be the input where the labels are extracted from, but the input of a target task is the (one-hot) word-vector. So "input of target task" has its origin of the dataset from which labels are extracted, but it is not the same. Or am I missing something? In variational autoencoder they aren't the same either. In "normal" autoencoders however they are the same.

Also, a lot of the publications mentioned in https://ai.stackexchange.com/questions/22184/does-self-supervised-learning-require-auxiliary-tasks claim that the auxiliary task the deciding factor. (The more I look into the naming in the machine learning literature, the more frustrating it gets.)

You got it right in the first comment. The robotics example is a bit farfetched, but it's correct as far as the sensor data is part of the input, but only trustworthy in some cases.
The input is usually modified in order to generate the auxiliary task e.g. masking a word in order to predict it afterwards.
The auxiliary task is key due to its implications in the representation learned e.g. predicting the mean color of an image is probably a very poor choice if you aim to perform facial recognition.

When we say that "self-supervision requires that data labels are extracted from the input data", we need to define what the input data is. Look at the new graphic in the question: Is it the input to the target task (3), the input to the aux. task (2) or a data that lies even further "outside" (1). For word2vec it is (1). For variational autoencoders used for image denoising and my similar task we are actually extracting the (3) from the (1) and the off-the-shelf-data is the out target data. But: if (1) is the input data for the definition then anything could be part of that input data.

That is supporting my suspection that (1) is the input data. In that case the robotic case would also be self-supervised (if having an auxiliary task is no requirement), because we just say any data is input data in the (1)-sense. 3) The strong connection between representation learned and having an aux. task is obvious. However, if we want to define self-supervision and it is not supposed to be a requirement, we cannot use it in a (mathemathical) definition of self-supervision. And yes, I am searching for a definition that is good enough that it could be made into a math. def.

@Make42 I really think you're overthinking it. Self-supervised learning is relatively new, and there's probably not a consensus on the definition of SSL. There's also the difference between SSL in robotics and in other contexts, as I emphasize in my answer. In the last edit of my answer, I attempt to give a more general definition of SSL: a machine learning technique that automatically generates a supervisory signal (and I am emphasizing the word automatically).


text:
How to tune hypeparametes in A2C-ppo?
Im currently working with A2C. The model was able to learn open ai pong, i ran this as a sanity check that i havent made any bugs. Now im trying to make the model play breakout, but still after 10m steps the model has not made any significant progress. Im using baseline hyperparameters which can be found here https://github.com/openai/baselines/blob/master/baselines/a2c/a2c.py, except my buffersize have been from 512 to 4096.  Ive noticed that entropy decreases extremely slowly

given the buffersize from the interval which i just gave. So my questions are how to make entropy decrease and how to increase rewards per buffer?  Ive tried to decrease the entropy coefficient to almost zero, but still it acts very weirdly.
Update: Even when i set the entropy coef to zero entropy wont decrease, i guess i might have a bug?

Human performance on Breakout is ~30, if you refer to the original DQN paper (Table 1). In the original A3C paper, it takes around 5 epochs to reach that score, so 20 millions frames (Figure 3).
Is the total loss going down in your original experiment? If you set the entropy coefficient to zero, then the contribution of the entropy loss to the overall loss is zero'ed out. That's why it's actually expected that the entropy loss won't decrease.
You're taking the gradients of
$$L_{\text{total}}(\theta_{\text{policy}}, \theta_{\text{value}}, \theta_{\text{entropy}}) = L_{\text{policy gradient}}(\theta_{\text{policy}}) + c * L_{\text{value}}(\theta_{\text{value}}) - 0 * L_{\text{entropy}}(\theta_{\text{entropy}}) = L_{\text{policy gradient}}(\theta_{\text{policy}}) + c * L_{\text{value}}(\theta_{\text{value}})$$ where $\theta_{\text{policy}}$ are the parameters of the policy network, etc.
Therefore, $L_{\text{entropy}}(\theta_{\text{entropy}})$ can be anything, it's unconstrained by the optimization of $L_{\text{total}}$
To test that you don't have a bug, you should in fact increase the entropy coefficient and check that the entropy loss is decreasing.
As a reminder, the entropy loss is actually here to maximize the entropy of the policy so that it doesn't collapse to a deterministic one.


text:
When to use convolutional layers as opposed to fully connected layers?
I am still new to CNNs, but I would like to check my understanding between when to use convolutional layers versus fully connected layers.
From what I have read, we can use convolutional layers with filters, rather than fully connected layers, with images, text, and audio. However, with regular data, for example, the iris dataset, a convolutional layer would not perform well because of the structure. As in the columns can be swapped, yet the record or sample itself does not change. For example we can swap the order of the Petal Length column with Petal Width and the record does not change. Where as in an image or audio, changing the column items would result in a different image or audio file.
These convolutional layers are "better" for images and audio because not all the features need to connect to the next layer. For example, we do not need the background of a car image to know it is a car, thus we do not need all the connections and we save computational costs.
Is this the right way to think about when to use convolutional layers versus fully connected layers?

I am not sure I understand your logic. First, you say that convolutional layers would not perform well on the Iris dataset, then you say "As in the columns can be manipulated, yet the record or sample itself does not change", but I don't understand what you mean by this sentence. Can you reformulate that sentence and clarify how it is related to the previous sentence? Then you say "These convolutional layers are "better" because not all the features need to connect to the next layer", but before that you say that convolutional layers would not perform well on the Iris dataset.

Please, edit your post to clarify it.

Apologies on the confusion. I have updated it to clarify as requested.


text:
Are FSA and FSTs used in NLP nowadays?
Finite state automata and transducers are computational models that were widely used decades before in natural language processing for morphological parsing and other nlp tasks. I wonder if these computational models are still used in NLP nowadays for significant purposes. If these models are in use, can you give me some examples ?

Both are used, for example,  in the GATE framework, which is still widely used. I suspect that this also applies to many other applications.
I would think that many recent academic publications are now on other approaches, as FSAs and FSTs are fairly established and mature technologies, but I've been out of academia for a while now.

GATE is a good example for discussion. The framework provides a processing layer including named entity recognition, co-reference resolution, etc. Surely, machine learning models could be more efficient than FSA and FSTs for these tasks, but the GATE would still use the latter computational models. One reason for doing this is that the framework was started in the mid-1990s and many human hours have spent building and maintaining it. It is difficult to change methods and technologies from a mature framework. But, I wonder if there is a reason for using these computational models nowadays.

@DimitrisDimitriadis If something works well, why replace it? Just because there are other frameworks available, that seem more 'trendy'. And why would ML necessarily be more efficient?

About your first question, I think that a reason for the replacement is the lack of expertise from the new generations. If the current trend becomes an established technology then people will prefer to work with this technology rather than to work with (e.g. FSA) only for the GATE framework. About your second question, that's why I posted this question :) I search for a reason why to prefer these obsolete methods (my opinion) rather than ML methods, which have a lot of benefits.

@DimitrisDimitriadis This is not the place for an extended discussion, but I disagree: they are not obsolete. ML is mostly a black box, ie you cannot explain its outcomes or fine-tune it, unlike other methods, which are not obsolete at all.


text:
Is it common that an applied deep learning research paper does not disclose any raw data and source code?
I think it is important for a research paper to include raw data and code for scientific replicability, verifiability, and falsifiability. However, recently, most of the research paper I read does not give raw data or source codes.
Those papers are from: European Journal of Operation Research; IEEE; International Conference on Cloud Computing and Big Data; MDPI
Some of them are highly reputable outlets.
Is this common that an applied CNN and DRL research does not disclose the code and data they used? Are there any journal with a data/code policy on that publishes applied research?

You're asking here 2 distinct questions, both of them are barely on-topic here. I would remove the second one from this post "Are there any journal with a data/code policy on that publishes applied research?", as it's not answered by the answer below that you accepted, and it's more off-topic, and, nevertheless, could be asked on another post. Having said that, your question is a (partial) duplicate of this.

@nbro Hi nbro, thanks for pointing out the related questions! I will write another post.

Many papers provide links to code, typically on GitHub.com. I have often found that even if the authors don't give a link, I can search the authors' names on GitHub and find a repository for the paper. My wild guess is that about 50% of papers provide reproducible code.
Papers With Code is a good website for finding papers with code.

When you trust Papers With Code's section on publishing trends among frameworks code availability only 25-30% come with code of all ML related topics. I guess that it might by roughly the same for only deep learning (sadly you can't filter that chart). Accounting the fact that they include community implementations biases the number slightly higher than they should be.

Wow! wonderful link! Truly helpful. Love this answer so much

I also think that saying that 50% of the papers come with "reproducible code" is probably wrong. There may be some implementations, not by any of the authors, as you mention, on Github or other code hosting services, but those do not necessarily reproduce the experiments or even the exact same models. In terms of really reproducible code, that percentage may be 5%.


text:
Are there fundamental learning theories for developing an AI that imitates human behavior?
Most, if not all, AI systems do not imitate humans. Some of them out-perform humans. Examples include using AI to play a game, classification problems, auto-driving, and goal-oriented chatbots. Those tasks usually come with an easily and clearly defined value function, which is the objective function for the AI to optimize.
My question is: how is deep reinforcement learning, or related techniques, to be applied to an AI system that is designed to just imitate humans but not outperform humans?
Note this is different from a human-like system. Our objective here is to let the AI become a human rather than a superintelligence. For example, if a human consistently makes a mistake in image identification, then the AI system must also make the same mistake. Another example is the classic chatbot to pass the Turing test.
Is deep reinforcement learning useful in these kinds of tasks?
I find it is really hard to start with because the value function cannot be easily calculated.
What is some theory behind this?

There is an interesting discussion the progress achieved in this field by far in the paper of Francois Chollet - https://arxiv.org/abs/1911.01547. At the present time, many architectures are able to outperform the human in particular tasks, because they have a strong priors coded into them and ability to process a huge amount of data.
However, when it comes to generalization, or even more difficult task of doing the things, that the coder has not put into the model, the present algorithms do no perform well. There is a rather sophisticated and developed mathematical definition of what is required from the system to be called intelligent. And in a nutshell, it is ability to develop a sensible behaviour and enough accuracy for new tasks without putting strong priors and large amount of experience.
In the end of the paper, there is proposed a benchmark to measure the intelligence of AI system.


text:
96.91% accuracy on MNIST after 2 hours of training using custom made neural net library. Ways to improve?
I wanted to understand back-propagation so I made a basic neural network library. I used momentum, with learning rate = $0.1$, beta = $0.99$, epochs = $200$, batch size = $10$, loss function is cross entropy and model structure is $784$, $64$, $64$, $10$ and all layers use sigmoid. It performed terribly at first, so I initialized all the weights and biases in the range $[10^{-9}, 10^{-8}]$ and it worked. I am quite new to deep learning and I find TensorFlow doesn't seem as friendly to beginners who want to play around with hyper-parameters. How do you find the right hyper-parameters? I trained it on 100 digits (which took 10 minutes), tweaked hyper-parameters, chose the best set and trained the model using that set on the entire data set of $60,000$ images. I also found that halving the epochs and doubling the training set size gave better results. Are there fool proof heuristics to find good hyper-parameters? What is the best set of hyper-parameters (without regularization, dropout, etc) for MNIST digits? Here is the code for those who want to take a look.

Also, how is TensorFlow super fast? How does training in TensorFlow result in higher accuracy while training a custom network using same parameters take longer? I am not utilizing my GPU or anything so what gives?

There is no singular best set of hyperparameters. Even more, there is no real search algorithm for hyperparameters. You can do a grid search, but this obviously will take some time. Most people either do that or will try to handpick their parameters.
A few other things to note: Initializing your weights at [10^9,10^8] doesn't seem right to me. They should be centered and close to zero initially. You should also take a look again how to set up train/test/val splits.

I meant the range $[10^{-9}, 10^{-8}]$. It was a typo. But almost all MNIST tutorials using TensorFlow seem to use the 6:1 train-test split ratio so I went for it.

i was confused by the word set chose the best set and trained the model using that set on the entire data set but you are right, it is a good ratio. At the end of the day, good hyperparameters are most of the time default ones with actions taken to minimize overfitting.


text:
Is there any network/paper used to analyse music scores?
As I am curious on music theory I would like to know that If is there any such network that analyse like labeling chords, or doing a roman numeral analysis.
Like an example below:

Source
It does not seem to be a difficult task.
Some other examples are given here[external link]
Also I am curious that If it is a possible task for AI to accomplish.

I think that you should try RBM's if you intend to make the machine make the music.

I have heard them make poems, but not something as you said.

Sorry if I understood and guessed it wrong.

Actually I do not want to generate music scores, I want to label them as like as YOLO does. Thank you for your comment

I have not come across music labeling algorithms but upon a google scholar search, I found a couple of papers that aim to do quite the same task.
In general, if you have a labeled dataset then you can take an approach of a general speech recognition model. It should work fine for music labeling too, but you might need to tweak certain parameters.


text:
Is it possible to classify resistors using ResNet50?
I want to train ResNet50 model using resistor images like below:

I tried it by collecting data from google images and there were quite few. So accuracy was very low (around %10) but I wonder If it is due lack of images or is it really possible to classify these images? Because as it is seen the object to be classified is very small and its value as color coded. I thought maybe this is not a good idea. Searched it on google but could not find anybody tried to do it before. I have also tried data augmentation and changing to other models but still its accuracy was quite low.
P.S: I have also tried changin epoch numbers, optimizers and all other parameters. So I want to make sure If it is due low data or is it just very hard task to complete for a computer vision model.
And Is it rational to crop the image by using a mask before classifying it to make sure all color codes are bigger and easily valuable by model?

Yes it should be possible. You may have a bug in your code, or the wrong hyperparameters. Training ResNet-50 will take a long time. Try training on other sets of images and see what accuracy you get to check if your approach is correct. Or, try loading a pretrained model, and training from that.

Thank you a lot for your answer. I was really worried about If it is due lack of images but now I will put effort on it.

How many images do you have?

I do not remember the exact numbers but as I remember It was around 3 per class and I have augmented it to around 100 per class by data augmentation. I know that this data is quite low to train a model. That is why I asked for If it is due low data (Because I thought maybe those color coded lines are just too small to be categorized because they are all in same shape -resistors-) If so I will start to collect data myself to train it because there are very few data online for resistors

You don't need ResNet-50 to learn resistors. They always look the same, a small convolutional network will suffice. You probably don't even need deep learning you could use do some kind of color thresholding and see how the colors are ordered or something like that


text:
Search recall optimization - what appropriate loss function to use?
I am studying machine learning and wanted to work on a project of my own so that I have better chances after graduating college. I'm studying the application of ML to improve searches using a toy dataset. Here's a couple of scenarios:

Document retrieval (search) system: We have a (source) document with us and we're trying to find a matching document in a database. The source document has text and image attributes - for simplicity let's say a title and a single image. Each search result will also be a document - with a title and at most one image.
A search engine: We have a query comprised of both text and an image (like google image search allows text to be added to the query as well). Each search result will be a website with text and image attributes (for simplicity, webpage title and at most one image)

More generally, I have a search system - whatever we're trying to search for has text and an image associated with it. Each search result will also have its own attributes. Let's denote each source document (or in case of search engine, a query) with $D$ and each search result with $R$.
The metric that makes the most natural sense to me is top $k$ search recall (not sure what the more widely used name for this is): suppose we have a sample of source documents or queries $D_1,\ldots,D_n$, and let the search results for $D_i$ be $\{R_{i1},R_{i2},\ldots,R_{im_i}\}$. If any of $R_{i1},\ldots,R_{ik}$ is a match for $D_i$, then we can say the search worked well for $D_i$. So the top $k$ search recall is the proportion of source docs/queries for which at least one match was found in the top $k$ search results.
Labels
Just to clarify, I'm looking for a search result scoring/ranking model that uses source doc/query attributes and attributes of search results.
Now to tweak my search system, let's say I also have a dataset with source document - search result pairs with labels - whether the search result is a match for the source document. i.e. something like
D1   R11   No Match
D1   R12   No Match
D1   R13   Match
.
.
D1   R1100 No Match
D2   R21   No Match
D2   R22   Match
D2   R23   Match
.
.
D2   R2100 No Match

(as you can imagine it would be a highly imbalanced dataset)
My current approach to tweak the system is like this:
Features and modelling
I can form features linking a source doc/query to a search result. e.g. some sort of embedding similarity between titles, or jaccard similarity of titles, or embedding similarity of their images, etc. There will be many null values too since some results may not have images, or some source docs/queries may not have images, or a result may not have a clear title, etc.
I'm using an XGB classifier - training, validating, testing it using the labeled dataset - the usual routine. I'm using the predict_proba method of the classifier to output the probability of a doc-result pair being a match. I use that probability as a score. Then for each $D_i$, the results with the top $k$ scores are shortlisted. And then the top $k$ search recall is measured for the $D_1,\ldots,D_n$ sample.
Problems
I'm not completely satisfied with the approach since I don't think I'm using the optimal loss function for the classifier. Right now it's just the standard XGBClassifier loss function. The classes are highly imbalanced as I said before. Artificially balancing the classes leads to a model that can better classify a matching or non-matching result, BUT it actually reduces top $k$ search recall compared to a model trained on unbalanced dataset for some value of $k$.
What else have I tried: So I'm using jaccard text similarity and image similarity as two of the features.

One modification I tried was using all features as separate "scores", and then calculated ranking for each of those "scores". So now I have different feature ranks - I used reciprocal rank fusion to combine them into a single rank. RRF top $k$ does not do as well as the classifier top $k$
Another modification: I calculated classifier top $k-5$. Then I clubbed those results together with jaccard text similarity top $3$ and image similarity top $2$. This method does better than the classifier top $k$

The last point clearly indicates that my way of combining various features into a model is suboptimal. Otherwise the model itself should have learned to accommodate this top $3$ and top $2$ into its top $k$.
My question is, what is the recommended loss function for simultaneously maximizing the top $k$ search recall and minimizing $k$? What is the best way to combine all this feature info into the optimal top $k$?

You should look into retrieval/ranking patterns used by modern recommendation systems. For the loss function, you typically use embedding similarity for retrieval and binary classification for ranking. Most ranking metrics are nondifferentiable and not used as loss functions