text:
model to generate suggestions for improving the cosine similarity of two documents?
I am working on a system that compares a source document to a target document and then generate alternative variations of the source document. The goal is to reach a higher cosine similarity between the two documents, say +80%. There's already plenty of articles on how to find the similarity difference, but not many on how to generate suggestions or improvements.
I've never heard of a model that can do this, so how do I build a model that can suggest altered variations of a source document with the ultimate goal of increasing its similarity to a given target document?
Articles on document similarity

https://medium.com/@adriensieg/text-similarities-da019229c894
https://www.sciencedirect.com/topics/computer-science/cosine-similarity

Linking the few articles that you found would be nice, and maybe also adding some info about what you're planning to try or tried already. As it is phrased right now the question sounds like asking other people to do your job

I don't really understand the problem. If you have the target document, is your ideal source document the target document itself (Since it has 1 cosine similarity)?


text:
How to learn a neural network with equivalent constraints on the weights
Let $f(x)$ be an output of a neural network with input $x$.
My data is a pair $(x,y)$ and my loss function is a function of $f(x)$ and $f(y)$, i.e., $g(f(x),f(y))$.
What kind of architecture enables this learning?
I could make two copies of the same neural network, but can the weights be coordinated in the back propagation?

Can you please put your specific question in the title? "Duplicating a neural network" is not a question and it's not specific.

A similar question has been answered at https://ai.stackexchange.com/a/34320/32722 and https://stackoverflow.com/q/49875127/3731823. You don't make a copy of the network, there is just one but it is used for separate inputs.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Head Pose estimation using Car Interiors depth infromation
I am trying to determine head pose of a driver sitting in a car with the depth interiors of the car known to me. Is there any research work which exploits that information in determining the head pose of the driver ?
One paper along the lines of using depth as a privileged information are,

https://arxiv.org/abs/1811.07376
which talks about using depth as a privileged information while training and not while inferring.

Also there are papers where depth is used as an auxiliary depth information in facial recognition.
https://openaccess.thecvf.com/content_cvpr_2018/html/Liu_Learning_Deep_Models_CVPR_2018_paper.html
But I am not able to find any reference where the depth of the background is used as an additional information in head pose/hand pose/human pose estimation.
Any help would be great!

What are your findings so far? Tell us what you have found so far.

Something along the lines of depth privileged information, https://arxiv.org/abs/1811.07376 for hand. I am finding quite a few papers where the depth map is available for the object in interest but not the background.

Sadly, I am not finding anything which has contextual depth information for doing head pose/hand pose/human pose estimation.

Edit your post to include this information there.

@nbro : Have edited the post.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Order of multiple Convolutional and Pooling layers in generated CNNs
I am reading this article: https://www.sciencedirect.com/science/article/pii/S2210650221000249
There, a multi layered particle swarm optimization of CNN parameters is presented. First step (layer) is to determine the architecture of the CNN and the second is to determine hyperparameters of each layer. On this image, all parameters (and their ranges) which are being determined are presented:

As you can see, there are three parameters regarding the architecture of the CNN, which are :

Number of convolutional layers
Number of pooling layers
Number of fully connected layers

What I am confused about is (and it is not written in the article) when these parameters are determined, how to build an actual CNN out of this. I understand that all fully connected layers are at the end of the CNN. I also understand that Convolutional layer is usually followed by a Pooling layer. But what I don't understand is how to determine the actual architecture if we have a different amount of Convolutional and Pooling layers. What is the order of those layers if that is the case?


text:
Hierarchical Navigable Small World Graphs : Expected Number of Steps in a Layer
Paper: Efficient and robust approximate nearest
neighbor search using Hierarchical Navigable
Small World graphs
In the Search Complexity section, the author estimates that the expected number of steps in a layer is bounded by $1/(1 - exp(-m_{L})$.
It says:

There is a fixed probability $p=exp(-m_{L}
)$ that
the next node belongs to the upper layer. However, the
search on the layer always terminates before it reaches the
element which belongs to the higher layer (otherwise the
search on the upper layer would have stopped on a different element), so the probability of not reaching the target on s-th step is bounded by $exp(-s· m_{L}
)$.

Shouldn't the probability of reaching the target on s-th step be $(1-exp(-m_{L}))^{s} *exp(-m_{L})$ since this would mean we are reaching a node which belongs to higher layer on the (s+1)th hop ? If yes, doesn't it defy the bound introduced in the above paragraph ?

Hello. You can use latex/mathjax on this site. I recommend that you edit your post to format your symbols and equations with it to improve the readability. You can also prepend the quotes with > to make it clear it's a quote.

Thanks @nbro, edited.


text:
Example of games in reinforcement learning where no model is available?
I'm reading the Sutton & Barto's book "Reinforcement Learning: An Introduction" (2nd Edition), as the classes I took were a long time ago, and I'm struggling to understand this part (p. 12):

To do this, it had to have a model of the game that allowed it to foresee how its environment would change in response to moves that it might never make. Many problems are like this, but in others even a short-term model of the effects of actions is lacking. Reinforcement learning can be applied in either case.

I don't understand how can games not have a model. If there is randomness in the environment's reaction to the agent's event, we still have a model don't we? If we have a game that does not have a model with states and values, how can RL be incorporated?
So my question would be:
Do you know examples where "even a short-term model of the effects of actions is lacking", i.e. where an MDP cannot be drawn or estimated?
I hope this is not too broad of a question. I've mainly done RL on Markov decision processes, and I'm struggling to see how a game would not have the possibility to be modelled by one (even if very big!).

Can you please put your specific question in the title? "Reinforcement learning no model of the game" is not a question and it's also not specific. Thanks.

I've tried to clarify the title and the post. Sorry.

But do you understand what "model" refers to in RL? If yes, then your question is not a duplicate, as I thought. Edit your post to clarify this.

Well I think I do but I'm not 100% sure otherwise I think I'd understand the statement I've quoted. I've tried editing my post, hopefully it'll be more clear.

So, it seems to me that you're asking: "Is there a game where there is no way to estimate a model of the game?". Is this correct? By the way, the quote "To do this, it had to have a model of the game that allowed it to foresee how its environment would change in response to moves that it might never make.... " was taken from where?

Yes, not necessarily a concrete example but I would like to understand how that would be even possible. To me everything can be summarized by states, actions and rewards.
It was taken from Reinforcement Learning An Introduction, 2nd Edition

Any idea? @nbro
Sorry to ping you but I'm struggling to find an example or a way to understand how that would be possible.

So, are you asking if there are cases where we wouldn't be able to estimate a model of the environment? And why wouldn't we able able to do so?

I voted to reopen this post, as I don't think it's a duplicate anymore. Anyway, I think that the answer to your question is: they are not saying that it's not possible to estimate the model, they are saying that the model is lacking. So, I cannot think of a problem where it would be impossible to estimate a model. There may be cases where it may be difficult to estimate a good model because maybe the state and action spaces are very big and/or it takes a long time to explore the environment. If the post gets reopened, I may convert this comment into a formal answer.

Oh ok so basically they're talking about problems where the state and action spaces are massive and not where there is no state and action space. That's what I was confused about.


text:
How to group multi-dimensional audio, video, and numerical data based on relatedness?
I have a data set that includes image arrays, point clouds, audio waveforms, and plain numerical data. I want to use unsupervised learning to group the data based on relatedness.  So, if the audio and video are changing simultaneously, then the algorithm should group them together. If I am not mistaken, this is called heterogenous data clustering.
My data looks like this:
Audio
[[first frame's audio samples] [second set] [third set] ...

Video
4D array of shape (1654, 500, 128, 3)
              # of imgs   l    w   channels

Gyro
[[roll, pitch, yaw], [roll, pitch, yaw], [roll, pitch, yaw], [roll, pitch, yaw].....

And a bunch of 1D numerical data

Is there a way to do this?
I am new to this so if there is some critical information missing, let me know.

What are your findings so far? Why can't you use k-means? What's the problem you're having? Provide more details about why you're stuck.

From my research, k-means is used for numerical data and not high-dimensional data that changes shapes across categories.

See this paper Head-to-head comparison of clustering methods for heterogeneous data: a simulation-driven benchmark or PyMix - The Python mixture package - a tool for clustering of heterogeneous biological data. They could be useful. Feel free to provide an answer to your own question, once you find it.

Thanks! I will check it out.

Is there a python implementation of this? This seems theoretical.

I see some example code but I don't know how to make it work with my data.

I don't know if there's any Python implementation. Check out on Github or Gitlab.


text:
How can I find the factor contributing to a prediction?
I've developed a neural network which predicts if someone is diagnosed with cardiovascular disease. Is it possible to find out which feature is the main cause of the prediction? For example, when a person is predicted to have CVD, it should show that his serum sodium level is too high.

You seem to be looking for Principal Component Analysis (PCA)

PCA is just a dimensionality reduction technique, it gives you no hints whatsoever about feature importance for a specific prediction of a trained model. At best you can use it to preprocess the features before training.

what you're looking for are a bunch of techniques falling into the macro category of explainable AI, you can check a fair summary of these techniques in this  answer

In the context of statistics, what you want to do is often called inference, which is different from prediction. Inference is just another term for what many people in AI now call explainable AI.


text:
Ready to use (already trained) ANN for small (64x64) RGB image compression
In my research, there is a current task of small (64x64 pixels) image compression. The images are photos, represented as RGB. While it is not a core interest, I wanted to compare some compression methods with ML compression methods. I am NOT a machine learning specialist (I never trained any network) so I need something off the shelf, which I can just download and compress/decompress images. I am a decent user of Python so this part is not a problem.
Can someone advise on (in some sense) a respected and already trained NN framework that I can use for this purpose? The "respected" part is important because if I want to include these results in my scientific paper, it cannot be just a random guy on GitHub :)

Why don't you have a look at paperswithcode they have a list of image compression benchmarks and papers that have open-source implementations. I didn't go through any of the papers or implementations, but it looks to me that the majority are ML-based methods.

Thanks for that! I was not really aware of this, will check it out

Yauhen Yakimenka, did you find what you were looking for?

To be honest, it takes longer than I expected to figure out which of those fit my needs...


text:
How to reduce loss of Bi-LSTM handwriting recognition model?
I am currently training an bi-LSTM model which predicts the handwriting of an individual. I am hitting a current min loss of 1.2 and I think it is not a problem with the model because I copied a model used by this study, which uses the IAM dataset as training data which is what I am currently using.
The way I am looking at this problem is that maybe my image preparation was not the same as said in the research. I prepared the image by changing the size of the words preserving the aspect ratio copied into a blank black canvas on the center. which makes my images normalized to a single size.
As said in the study

The built network takes in a variable width image as an input, where the length of the image is sixty pixels

So I think I am doing something wrong here or just misunderstanding somethings.
Is it possible maybe that this model accepts images with variable widths, or is it just doing what I did?
Also, the study had a loss of under 1 in just 5 epochs. Which I can hardly hit with mine.
And does the image's scale a factor here? If so, can I improve it by introducing data augmentation in my dataset?
A sample of the dataset:

Below, is the summary of my model made in keras
Model: "handwriting_recognizer"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to
==================================================================================================
 image (InputLayer)             [(None, 360, 60, 1)  0           []
                                ]

 Conv1 (Conv2D)                 (None, 360, 60, 64)  1664        ['image[0][0]']

 batchnorm1 (BatchNormalization  (None, 360, 60, 64)  256        ['Conv1[0][0]']
 )

 pool1 (MaxPooling2D)           (None, 180, 30, 64)  0           ['batchnorm1[0][0]']

 Conv2 (Conv2D)                 (None, 180, 30, 64)  36928       ['pool1[0][0]']

 batchnorm2 (BatchNormalization  (None, 180, 30, 64)  256        ['Conv2[0][0]']
 )

 pool2 (MaxPooling2D)           (None, 90, 15, 64)   0           ['batchnorm2[0][0]']

 Conv3 (Conv2D)                 (None, 90, 15, 256)  147712      ['pool2[0][0]']

 Conv4 (Conv2D)                 (None, 90, 15, 256)  590080      ['Conv3[0][0]']

 batchnorm3 (BatchNormalization  (None, 90, 15, 256)  1024       ['Conv4[0][0]']
 )

 pool3 (MaxPooling2D)           (None, 45, 7, 256)   0           ['batchnorm3[0][0]']

 Conv5 (Conv2D)                 (None, 45, 7, 512)   1180160     ['pool3[0][0]']

 Conv6 (Conv2D)                 (None, 45, 7, 512)   2359808     ['Conv5[0][0]']

 batchnorm4 (BatchNormalization  (None, 45, 7, 512)  2048        ['Conv6[0][0]']
 )

 pool4 (MaxPooling2D)           (None, 22, 3, 512)   0           ['batchnorm4[0][0]']

 reshape (Reshape)              (None, 22, 1536)     0           ['pool4[0][0]']

 dense1 (Dense)                 (None, 22, 64)       98368       ['reshape[0][0]']

 dropout (Dropout)              (None, 22, 64)       0           ['dense1[0][0]']

 bidirectional (Bidirectional)  (None, 22, 256)      197632      ['dropout[0][0]']

 bidirectional_1 (Bidirectional  (None, 22, 256)     394240      ['bidirectional[0][0]']
 )

 bidirectional_2 (Bidirectional  (None, 22, 128)     164352      ['bidirectional_1[0][0]']
 )

 dropout_1 (Dropout)            (None, 22, 128)      0           ['bidirectional_2[0][0]']

 label (InputLayer)             [(None, None)]       0           []

 dense2 (Dense)                 (None, 22, 85)       10965       ['dropout_1[0][0]']

 ctc_loss (CTCLayer)            (None, 22, 85)       0           ['label[0][0]',
                                                                  'dense2[0][0]']

==================================================================================================
Total params: 5,185,493
Trainable params: 5,183,701
Non-trainable params: 1,792
__________________________________________________________________________________________________

I've added a link to the paper that I think you were referring to. Make sure that's the correct link.

It is  the correct study. Thanks for editing my question.


text:
Is there a shared minimalist taxonomy of the fields of AI?
Is there a reasonably accepted/shared view on how to minimally classify the various fields of AI? There are hundreds of techniques and I have not been able to find a shared exhaustive classification in reasonably disjointed 'fields', or approaches, or 'schools'.
One possible classification (please, feel free to correct my imprecise terminology) is given by Wikipedia in the Artificial Intelligence page
• Symbolic.
• Deep Learning.
• Bayesian Networks.
• Evolutionary.
But it seems to be somewhat lacking (for example, not all NN are deep learning, and where do SVN methods fit here? Bayesian networks seem too specific to accomodate them).
On the "Outline of Artificial Inteligence" page, wikipedians are giving a more detailed view, but it does not look minimalist to me (moreover, 13 Search a field of its own? Is it not 'dispersed' in all other fields? see Note)
A slightly more minimalistic, yet apparently complete, classification can be found in the laymen book "Artificial Intelligence for Dummies". Here the authors mention five "tribes" (I might have placed them in a different order):
• Symbolists: The origin of this tribe is in logic and philosophy. This group relies on inverse deduction to solve problems.
• Bayesians: This tribe's origin is in statistics and relies on probabilistic inference to solve problems.
• Analogizers: The origin of this tribe is in psychology. The group relies on kernel machines to solve problems.
• Connectionists: Thia tribe's origin is in neuroscience and the group relies on backpropagation to solve problems.
• Evolutionaries: The evolutionaries tribe originates in evolutionary biology, relying on genes e programming to solve problems.
Is this the most comprehensive yet mininalist classification? Is there an even more comprehensive one, along these lines? It also seems to me that Bayesian and Analogizer can both fall into the realm of "Statistical Learning".
What I am looking for, in order to orient myself, is some sort of big Venn diagram or better yet partition with all the approaches/tribes (the more general, the better) and examples of techniques (for example, Naive Bayes under Bayesian: Expert Systems under Symbolists: Perceptrons, Neural Networks under Connectivists...)
To be clear, I am not trying to accomodate transversal approaches, such as Classical vs Hachine Learning, or Supervised va Unsupervised vs Reinforcement Learning, or Strong vs Weak AI..
I understand there are hybrid approaches, of course. What I am interested in is the possibly most disjointed, 'pure approaches on the line of the five tribes described above. Am I perhaps looking for a magical unicorn?
Note: As stated above, I am also a bit confused about how to fit searching algorithms, because I tend to see them as either transversal to the various tribes' or even orthogoal to them imaybe part of Optimization goals, along with Predicit ion, Classification and... what else? Planning?), but this might be the subject of another question

Apology for the formatting, but since the last charne of the SE site I can no longer post from laptop and desktop. I had to take a screenshot of my laptop screen, pass it through an AI to convert into text and try to fix formatting with big hands on a small phone.

Perhaps consider distinguishing between theory and practice, or philosophy and implementation. Speaking of "tribes" places focus on people; "origin" on history. Another possible focus is on goal or aim.


text:
How can the agent be defined in a reinforcement learning problem with a tabular dataset as the environment?
Let's assume we need to train an RL model that drops duplicates in a tabular dataset? The actions should probably defined as drop or do nothing.
But what should be the agent itself then? To me, it doesn't make sense to see it just as a navigator looping over the states (dataset indices from the first to the last) and decide on which to drop.

To me the approach of using RL for such a problem sounds a bit strange. RL is commonly applied to episodic control tasks. So each state $s_{t}$ usually has something to do with the state $s_{t-1}$ before that.
This of course can be a property of tabular data as well. For identifying duplicates that are 100% similar you probably wouldn't apply ML, right? And for finding duplicates based on some similarity you could train an encoder model and compute some distance between any two samples in the latent space to find duplicates.

Duplicate dropping is just an assumption to give an example how an RL model can be used with tabular data, because it's the simplest case I could think of. It's not the actual matter if you read the question carefully, even pandas can do it by one line though.


text:
Do we need backpropagation if there is only one class?
A am interested in physiologic neural network. Altough there are some opposite views, most probably there seems to be no plausible way to explain a physiologic backpropagation in the brain.
So I am trying to code a neural network without backpropagation yet my mathematical  understanding is inadaquate, so I want to ask folowing simple question:
“If we do have only one node at the right side of the network, accepting that the inputs are on the left, can we train the network without backpropoagation and using the mean of weights instead? As we would know the y for all x it should be possible to calculate the mean w?”
The idea is that the system should work continiously, and train continiously for one class, until the trained network decides that the input is different from the known previously trained classes. And if it is different, it should create and train for that new class of inputs. I think that should be the working system of the brain, and as the cortex has similar cells, mathematically it also must be that easy?
But where is my flaw (with simplified math please :))

If there is only one class, why do you need a neural network at all? Just do function getClass(input) {return "class 0";}

Thanks for the answer.

Thanks for the answer. True. But would a training result in a mean of weights? If yes, then it would be able to differentiate that class 0 from others, that are significantly different from “other” classes.Than it would begin to train for that new “ class”…So we would reach a new class producing system or general intelligence without backpropagation?

If there's only one class there is nothing to differentiate. If you have "class" vs "not class" that's 2 classes

I am really sorry for my childish question, but I do want to understand: If we know the range of  x that would be a cluster of information, and classify that information as class 0, wouldnt that be still one class to train?

I don't understand what the purpose of a network with only one class is. You ask it what class a cat is, and it says class 0... you ask it what class a dog is, and it says class 0... you ask it what the sky is, and it says class 0... because that's the only possible thing it can output... so why bother having a neural network at all?

Lets say there is a network we trained for a class(class0) of information(input (x,y) , say, that is a cluster of dots in a range of  2<x<4 and 3<y<5, we would be able to say that a point (2.5,4.5) is in this class ( class 0) and not in another class. Lets think that the information ,coming continiously,is  seperable, that is lets say there is another class( class 1) which as at least 10 units distant from the class 0, as an example,a cluster of dots in the range of 12<x<14 and  13<y<15, for that the network should begin for another class to train.

Oh so you are not just looking for a one-class neural network, but for one that can learn more and more classes, starting with one. Do you at least know that some data isn't in the class?

Yes, exactly, a one by one learning neural network.No, I was thinking, that can be calculated from naive statistics, whether it is “significantly” according to the weights to the same class or not..

The point is, that we need backpropagation if more than 1 classes, but there should be no need for bp, if there is only one class, which can be trained in realtime…That would be more near to physiologic brain. As you did understand what I was meaning, could you please give some examples near that idea?

If there is only one class then your neural network is completely useless and doesn't do anything

@Eren no meaning if you classify to 1 class, but a little ok if you regress to zero value

I was trying to ask One Class SVM… It solved my question


text:
Is there a variant of Thompson Sampling that works with variable bandits?
Does there exist a variant of TS, such that, while computing the returns of multi-armed bandits, we have the possibility of introducing an extra bandit?
For instance, while we are applying TS to 3 slot machines, we come to know about the existence of a fourth slot machine. Therefore, we'd like to take that machine into account within our algorithm.

So, are you asking about applying TS to bandit problems with a variable/dynamic number of actions?

Yes, that’s what I am asking.


text:
Given the high resolution signal and the low pass filter (kaiser filter), is there a way to reconstruct the low resolution signal?
When we upsampling a discrete 1d signal by 2x, we first interleave the signal by 0, then pass through a low pass filter.
low resolution signal [x1, x2, x3, x4] -> interleave 0 -> [x1, 0, x2, 0, x3, 0, x4] -> low pass filter (kaiser filter, convolution) -> high resolution signal
Given the high resolution signal and the low pass filter (kaiser filter), is there a way to reconstruct the low resolution signal? If there is not a way to perform precise reconstruction, is there a way to perform roughly reconstruction?

I Think you'll have more luck if you move the question to the signal processing communiy

Thank for a lot. I will try it.


text:
What could be causing the poor performance (MSE) of a dense neural network on a real time-series dataset?
I am trying to understand time series analysis and actually I am following the course "Sequences, Time Series and Prediction" in Coursera. The course is based on a synthetic dataset, arguably complex, that is studied with different network configurations. The difference among the various networks architectures exist but within certain range. Roughly, all the architectures succeed to grasp the pattern, and they are more or less precise in describing the spikes of the data proposed.
The last exercise is with real data (sunspots) and propose a complex model including convolutional, recurrent and dense layers.
I added a trial applying only dense networks (very successful on the synthetic data), but it did not work at all. The performance is assesed by the mean squared error, and it is three times bigger with the feed forward network than with the other model including recurrent and convolution layers.
As there are many details, it is difficult to present all them in text format, and to show them all the notebook with the data and the code comparing both proposals (and also using or not a crafted callbacks) is here.
I wonder if I had done any mistake when coding, or if there is another reason for obtaining these results.
Any clarification will be well received.

Can you be more specific about "it did not work at all"? What does that mean? How are you assessing the performance of this fully connected neural network? You probably should also describe your real data and the hyper-parameters of the model and training procedure. Which algorithm are you using to train the model? How are you training the model? How big is the dataset? How are you testing the dataset?

I edited the question in order to clarify, as suggested by @nbro. The algortithm, and all the details about the data, the training and the evaluation are in the notebook.

Thanks for providing these clarifications. I've also changed the title to be what I think is your actual question. Make sure that's the case.

This title is ok


text:
What is the next-character perplexity of the PaLM model?
In the 2022 paper "PaLM: Scaling Language Modeling with Pathways", what is the bits-per-character perplexity of the resultant pre-trained model for next-word prediction?

Can you clarify why you're asking this question? What is "bits-per-character perplexity"? Why are you interested in it? I suppose it's not reported in the paper and you want to know if anyone knows its value for that model, i.e. "the resultant pre-trained PaLM model for next-word prediction"?

I'm asking what the log likelihood of a test character from the test set of PaLM is, averaged across sampled contextual windows the model can handle.


text:
how to define or calculate the similarity betweeen two curves as the loss funtion to optimize in the generative model?
I want to train a neural network as the curve productor that can generate the specific type of curves (e.g. exponential decay curves). I take the encoder-decoder structure, the curves in a dataset is used as input, the encoder encodes them as the hidden variables, and then decoder decodes those hidden variables to regenerate the same curves. The loss function is the difference between the input curve and the generated. But the tough problem is how to measure similarity or consistent degree between the two curves as depicted in the pasted figure. In fact, the curves are permited to shift left or right, which make it the difficult thing to deal with.
Note that, Don't need to consider the points that is lower than 0.05.

Try taking a look at Dynamic Time Warping, it returns a similarity score between two curves and it's shift invariant.

Ok, thank you!  It's helpful.


text:
How do I perform automatic evaluation of my NLP model?
I have a model which converts sets of keywords to sentence, but I've to quantify it's quality. In computer vision, we would calculate the model's accuracy, I'm kind of lost and how do I go about using baselines as well, am thinking of using BLEU and ROUGE on pairs of data not used during training, is this the correct way to go? Can someone please advise?
And should I average my BLEU/ROUGE scores I got for each iteration? Is there a better metric to evaluate NLP models?
My dataset is like
keyword, keyword, keyword -> sentence

Can you provide a concrete example of what would be a good and bad conversion and explain why that is the case?

Do you mean an example of how the model works? It's like input is [David,eat,pizza], output = 'David ate a delicious pizza yesterday'. I guess bad conversion would be if it doesn't include those keywords or if the generated sentence is far from what is expected from the dataset?


text:
What is the correct formula for the loss function?
I have used  the Delayed sin echo prediction with Tensorflow that predicts the sin wave. However, I'm not sure of the correct formula for the loss function. The problem is that I feed the training mini-batch as batch_x and batch_y, containing a combination of historical and current information. However, I provide the model with only the current information $o_{t}$ in the prediction. In my current setting, $historical \subseteq(o_{t-k+1},o_{t-k+2},\ldots,o_{t})$ where the $k$ is the size of the historical information. So, my question is how to select $y_i$ if both batch_x and batch_y use the historical and the current information.
\begin{equation*}
 \mathrm {MSE=}\frac {1}{n}\sum _{i=1}^{n} {({y}_{i}-\hat{y}_{i})}^{2}
\end{equation*}

I would guess from context that this is a regression problem with some short length of sequence as input, and a single-valued output in which case yes probably the loss function is MSE. The question needs some work though. Apparently you have used some third-party code snippet in a personal experiment that you do not explain, which is the answer to some other Stack Overflow question, and you want to know the mathematical definition of loss that they are using in their code?

It is unclear to me why you have put the MSE formula in the question.  Are you asking if that is correct, or if it is the correct loss to use?

I have updated the question.


text:
Resolving Derivation Discrepancies for Differentiating through Optimization Paths
I'm reading the paper "Optimizing Millions of Hyperparameters by Implicit Differentiation". The key contribution of the paper is to show that you can replace optimizing through the optimization process/path by using implicit gradients to effectively optimize hyper-parameters.
Problem:
I don't quite understand how the optimization path fits into the derivation. For example, per their derivation, let's assume that we want to optimize some hyper-paramaeters $\lambda$, where $w^{*}(\lambda)$ is the locally optimal base-parameters using some value of $\lambda$. Furthermore, let us assume we are using some vanilla gradient descent style optimization process for both the inner and outer optimization.
\begin{align}
\lambda_{new}
&= \lambda - \frac{\partial}{\partial \lambda}\mathcal{L}(w^{*}(\lambda)) \\
&= \lambda - \frac{\partial}{\partial w^{*}(\lambda)} \mathcal{L}(w^{*}(\lambda)) \cdot \frac{\partial}{\partial \lambda} w^{*}(\lambda)
\end{align}
Where $\frac{\partial}{\partial\lambda}w^{*}(\lambda)$ can be replaced with some closed-form solution as follows using the implicit gradients theorm:
\begin{align}
\frac{\partial}{\partial\lambda}\bigg[\frac{\partial}{\partial w} \mathcal{L}(w(\lambda), \lambda) \bigg] & = 0 \\
\frac{\partial}{\partial w}\bigg[\frac{\partial}{\partial\lambda} \mathcal{L}(w(\lambda), \lambda) \bigg] & = 0 \\
\frac{\partial}{\partial w}\bigg[\frac{\partial\mathcal{L}}{\partial w} \cdot \frac{\partial w}{\partial\lambda} + \frac{\partial\mathcal{L}}{\partial\lambda} \bigg] & = 0 \\
\frac{\partial^{2}\mathcal{L}}{\partial w \partial w^{T}} \cdot \frac{\partial^{2} w}{\partial\lambda\partial w^{T}} + \frac{\partial^{2}\mathcal{L}}{\partial\lambda\partial w^{T}} & = 0 \\
\frac{\partial^{2}\mathcal{L}}{\partial w \partial w^{T}} \cdot \frac{\partial w}{\partial\lambda} + \frac{\partial^{2}\mathcal{L}}{\partial\lambda\partial w^{T}} & = 0 \\
\frac{\partial^{2}\mathcal{L}}{\partial w \partial w^{T}} \cdot \frac{\partial w}{\partial\lambda} & = - \frac{\partial^{2}\mathcal{L}}{\partial\lambda\partial w^{T}} \\
\frac{\partial w}{\partial\lambda} & = - \bigg[\frac{\partial^{2}\mathcal{L}}{\partial w \partial w^{T}}\bigg]^{-1} \cdot \frac{\partial^{2}\mathcal{L}}{\partial\lambda\partial w^{T}}
\end{align}
However, I don't understand how $w^{*}(\lambda)$ gets transformed into the following:
\begin{align}
\frac{\partial}{\partial \lambda} w^{*}(\lambda)
&= \cdots \\
&= \cdots \\
&= \frac{\partial}{\partial\lambda}\bigg[\frac{\partial}{\partial w} \mathcal{L}(w(\lambda), \lambda) \bigg]
\end{align}
Intuitively, $w^{*}(\lambda)$ is the result of the optimization path taking $k$ gradient steps:
\begin{align}
w_{0} &= \cdots\\
w_{1} &= w_{0} - \frac{\partial}{\partial w_{0}} \mathcal{L}(w_{0}(\lambda)) \\
w_{2} &= w_{1} - \frac{\partial}{\partial w_{1}} \mathcal{L}(w_{1}(\lambda)) \\
\vdots \\
w^{*} &= w_{k-1} - \frac{\partial}{\partial w_{k-1}} \mathcal{L}(w_{k-1}(\lambda)) \\
\end{align}
Which we can substitute back into our equation as follows
$$\frac{\partial}{\partial \lambda} w^{*}(\lambda) = \frac{\partial}{\partial \lambda} \bigg[w_0 - \sum^{k}_{i=1}\frac{\partial}{\partial w_{k-1}} \mathcal{L}(w_{k-1}(\lambda))\bigg]$$
However, this isn't the quantity we want since
$$\frac{\partial}{\partial\lambda}\bigg[\frac{\partial}{\partial w} \mathcal{L}(w(\lambda), \lambda) \bigg] \neq \frac{\partial}{\partial \lambda} \bigg[w_0 - \sum^{k}_{i=1}\frac{\partial}{\partial w_{k-1}} \mathcal{L}(w_{k-1}(\lambda))\bigg]$$
Can anyone help resolve this issue for me?

Could you please put your specific question in the title? "Differentiating through Optimization Paths" is quite general. It seems that your question is related to a discrepancy between their derivation and your derivation. So, your title should probably ask about that.

Thanks for the suggestion, I have updated the title to better reflect the contents of the question.


text:
How to use oxford5k for training?
Generally, we have training data with landmark IDs, their GTs (positive samples), and then separate query images and corresponding positive samples for evaluation.
In the Oxford5k or ROxford5k, one finds the landmark images and other images for that landmark. For example, everything starting with all_souls correspond to the all_souls building. Nevertheless, when I see other images containing this tag, it contains people and indoor images, which are possibly junk (But they are not, I guess). In the GT files, I see files like all_souls_1_query.txt, all_souls_1_good.txt, all_souls_1_ok.txt, all_souls_1_junk.txt, and so on.
I want to make sure I get the standard practice of how to train on these datasets and evaluate it properly.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

Can you provide more details about your task? What task are you trying to solve? It seems that your problem is related to "data preprocessing", i.e. you want to know how to pre-process the data in these mentioned datasets. I recommend that you use the tag [tag:data-preprocessing].

@nbro, I am just starting to train a model on this dataset but I found structure of it to weird. but it is widely-used dataset and I just want to know how experts use this dataset.


text:
Is there an unsupervised learning method for determine the most common questions within a dataset?
I have a dataset consisting of questions from customers. I am curious of the n most frequent asked questions, regardless of the variation the questions might appear in.
Is there NLP methods for finding n most frequent asked questions?

How are you measuring "closeness" for questions? If you are looking for a general answer, you probably can use a pre-trained NLP model like bert - https://towardsdatascience.com/bert-for-measuring-text-similarity-eec91c6bf9e1. If you want something specific, can you provide more details?

How do you define variation of the same question and different question?

How I measure closeness, variation or distance is part of the question.  I know its common to use cosine to measure similarities in semantic spaces. But I'm curious if there were some methods I've overlooked for this problem.


text:
Which existing model could be used for wind speed and direction prediction?
I am trying to predict the wind speed and wind direction in a graph network for a geographical area. The dataset includes the start and end nodes, the distance between them, and wind speed and direction. I want to predict the wind speed and direction for a certain number of line segments (edges) in a graph network.
Which existing prediction model could be used for this problem? For example, my dependent and independent variables are wind speed and wind direction. How these can be used to predict their future values?
Note: The network here represents a graphical network constructed using NetworkX Python library. Nodes represent the cities and edges represent the distance between the cities. Furthermore, I am currently having a dummy dataset for wind speed and wind direction for each edge. Using this data, I want to predict the wind speed and wind direction for a certain number of edges that are between a given source and destination.

When you say "graph network", do you mean a "graph neural network" or "graphical network" (like a Bayesian network)? I suppose you're talking about the first. If yes, you can edit your post to use the tag [tag:graph-neural-networks].

@nbro I meant a graph-based network built using the NetworkX python library.

So, you don't mean a "neural network", but just a graph, correct?

Can you please explain what the nodes and edges in your graph represent then? You say "I want to predict the wind speed and direction for a certain number of line segments (edges) in a graph network", but what does it even mean? This depends on what a edge represents.

Yes, it is just a graph. Nodes represent the cities and edges represent the distance between the cities. Furthermore, I am currently having a dummy dataset for wind speed and wind direction for each edge. Using this data, I want to predict the wind speed and wind direction for a certain number of edges that are between a given source and destination.

I recommend that you edit your post to include this info directly there because comments are meant to be temporary and can later be deleted.


text:
Could I cluster the audio clips in order to improve the speed of their classification?
I have a neural network which is very resource intensive and is used to classify audio clips. The classification is done in batches, where I record for a set period of time and then go through and classify the audio.
However, the time for classification is far too long.
So I was thinking what if I could somehow embed the audio and then cluster them into similar audio files. This way I'd only need to classify a couple audio clips in the cluster and assume they're all the same.
Is this possible?
All suggestion, improvements or help is very much appreciated!

Can you be more specific? Why is the classification slow? Is it because there are a lot of audio clips?

@DKDK The time of detection of clip > the audio length of the recording

It sounds like something that is possible, yes.

The clustering will need to do similar processing as the classifier, if it is to work any good at all. I think you are much better off optimizing your neural network

What neural network are you using? Number of parameters?

Are you using batching? Multi-processing?


text:
Ongoing competitions on reinforcement learning?
Are there any ongoing competitions except the following?
Battlesnake, SC2 AI Arena,
Bomberland, ConnectX

It would be cool if there was somewhere that aggregated these kind of competitions. The question is not great for the AI Stack Exchange site though, because it will quickly go out of date as competitions change. It's not really possible to make an answer which is valid for more than a few months. I think it could be made better, by specifying the year. The question would then naturally go from being "current" to being a historical record.

There is still the issue that this is basically a request for a list of external resources. My suggestion would be to not exclude the ones you have found so far (but do list them in the question, to show you have done some research), make the question "Where can I find public reinforcement learning competitions for 2022?" or similar, and we can have a Community Wiki answer to try and build the list.


text:
Are there assumptions made about Self-Play that don't hold up in regular MA competition?
I read about this paper Efficient Competitive Self-Play Policy Optimization which proposes an algorithm for training a population of agents with self-play using a perturbation based matchmaking approach.
I was wondering if this algorithm can also be used with regular MultiAgent Competition, e.g. an asymmetric game like robo-soccer with one goalie and one striker.
Are there specific properties that self-play fulfils, that doesn't hold up in regular MA Competition? Can approaches for self-play be used in scenarios where not the same policy is used for both agents? If not, what are some properties one has to keep in mind when trying to adapt those approaches?

Can you clarify how the "regular multi-agent competition" (like robo-soccer) is different from the task that the authors of the paper addressed in their paper? What did they use the population of agents for?

@nbro As the authors specifically use their algorithm for self-play, I was wondering what this might imply for its usage. With ''regular MA-competition'' I refer to the circumstance where we use different kind of policies for the agents. Considering the robo-soccer example, I meant that e.g. one might use a different policy for goalie and striker. I see that the authors use an agent population, which in the end uses multiple policies too, but I was wondering why then being overly specific and coin it only to self-play, rather than more general Multi-Agent Competition.


text:
What are the benefits of using multiple convolutions, as opposed to one, before the pooling layer in a U-Net?
I have seen U-Nets that use a single convolution before the pooling operator and some that use two or more.
My question is, what is better? Or what are the benefits of using more or less convolutions?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

Can you please provide links to these U-net versions that you're mentioning?


text:
Is there a way to improve the low-quality data?
I'm on a robotics team and we've been tasked to write a program to differentiate between a live and dead fish. We've been given ~15 minutes of training footage and it's absolutely terrible. It's low quality, hard to label (even for humans) and it's like 20 frames a second.
I have tried everything I can think of. YOLO, 3D convolutions (to take movement over time into account), residual networks with anywhere from 1-10 layers and more. I have narrowed it down to the data is just terrible.
Is there anything I can do to fix this? I know of data augmentation and have used it, but that doesn't increase the usefulness of the data, it just creates more terrible data. I feel like using machine learning to clean the data wouldn't be helpful (because of studies I can't remember the name of showing that adding one white pixel to an image can completely confuse an object classifier I just assume that using a machine learning model to alter an image would also just confuse a network), is this an accurate assumption?
Either way: is there anyway to improve the data I've been given? Or another way to approach this problem?

The study mentioned in your question (halfway through answer): https://ai.stackexchange.com/a/14264/17742

I suggest using a model of super resolution to enhance the quality of your dataset and then annotate it properly because by that time it will be readable at least from a human being perspective.
You can check out this blog post for more information.

These models fill in data that doesn't exist. Won't this hurt the accuracy of any network I train? Even if there isn't a noticable difference to the human eye, wouldn't this be useless to a computer?

You can use super resolution for ease of annotation purposes only. and then you can reduce the qualityof your dataset and the ground truth as well to return back to the original case you started with. So you can train a model that takes an input with bad quality.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
How to model the probability of click-through in bandit problem?
Consider a bandit problem in which we want to maximize the probability of click-through based on bid values ($b$ is the value of bid and $\Pr(b)$ shows the probability that a customer clicks on a link given bid $b$). I am wondering what the possible modeling approaches are? If we consider a discrete set of bids ($b\in \{b_1,b_2,\cdots,b_k \}$), it is possible to model the probability as a Bernoulli distribution for each bid $\Pr(b_k)=p_k$ (with Beta or Dirichlet prior, e.g., $p_k \sim Beta(\alpha_k,\beta_k)$ ). Another case is model as a Logistic bandit in which it is possible to extend it to continuous values for bids, $\Pr(b_k)=\frac{1}{1+\exp\left(a_0 + a_1 b_k \right)}$.
I am wondering what other approaches are possible? Can we model it in form $\Pr(b_k)=\beta_0 + \beta_1 b_k$ with Normal prior on $\beta_0$ and $\beta_1$? If yes, what should we do if $\beta$ is estimated such that the probability exceeds 1?

Can you define more precisely (i.e. mathematically) what you mean by "click-through"? I know that the click-through rate may depend on the bid that you pay, but why do you think it only depends on the bid? Moreover, can you explain why you want to model is as a bandit problem (at least according to the title), but then you're wondering if you can model is as a logistic model? What is your main question exactly? Please, highlight that, because otherwise people may not know what you're really asking, given that you seem to be asking many questions, actually.

If we consider the values of bid to be discrete values, then we consider each value as an arm. Because the reward is in form of probability, I have considered logistic function as the output to make sure that it is between 0 and 1. There are some Logistic Bandit papers and I have used their idea.

I am used to bandit problems where the task is to learn quickly, or with minimal regret, through trial and error attempts on action choice, with a goal of obtaining the maximum reward per action. It is not clear whether this is your task, or what the action and reward is here. If the action is to be bid choice, and the reward a constant on obtaining a click-through, then most of your models imply maximising the bid directly and no exploration would be required. However that feels wrong, and I think you are missing some important details of your setting. Please link a description


text:
Is 3D pose prediction directly from point clouds commonplace?
I've spent a couple of days experimenting and trying to find papers on learning 3D pose directly from unordered point clouds with no color information. This paper from 2020 claims to be the first that does it. The results are not encouraging for my use case and I'm not sure I agree with their methods (I tried it for my problem and got poor results).
Overall I'm surprised that 3D pose regression with point clouds would not be commonplace. Have a missed a whole section of the literature? Is there some fundamental reason why this hasn't been figured out yet?

What do you mean by "thing" here?

The most common informal meaning of the phrase. Image classification with CNNs  is "a thing" or it's "commonplace". Importantly, it's commonplace because it works well and is justified. Image classification with GNNs is not commonplace, and I guess there are reasons for that.

Do you mean 'task'? If yes, please update the title and body accordingly.

No I don't mean task.

Oh, I got................


text:
How to make neural networks more robust(intuitive explanation)?
Low spectral Norm ensures tight Rademacher complexity and ensures low generalization error for neural nets. Can anyone explain me this in a intuitive manner along with Rademacher view point.
Rademacher Complexity

Could you please provide links to your source material (assume I have never heard of Rademacher complexity for example - there are dozens of data science metrics, and once you have seen a few it is reasonably quick to understand a new one)? Also, it would help someone answering if you gave more details about what you do already understand about low spectral norm and how it may improve generalisation errors (most regularisation does this, so it's not unexpected from a high level view). Otherwise someone answering has to make a lot of effort based on guessing what level to pitch at.

I have updated, also trying to post my view on these. Thanks for your response


text:
Why do smaller weights converge faster for RNNs?
I am writing a Recurrent Neural Network using only the NumPy library for a binary classification problem. When I initialize the weights with np.random.randn, after 1000 epochs it gets ~60% accuracy, whereas when I divide the weights by 1000 first, it reaches 100% accuracy after the same amount of epochs.
Why is this? Do RNNs work better with smaller weights or does the number 1000 mean something?
Any and all help is welcome, thanks.

There is no magic value that work for every network but in general:

too large initial weights lead to exploding gradients (i.e. no convergence)
too small initial weights lead to vanishing gradients (i.e. small loss without real convergence)
the best initialization strategies use uniformly of normally distributed weights, and set an order of magnitude that depends on the amount of parameters in the previous layer. (see xavier initialization and kaiming initialization)

Check this blog post for pretty good animations of both problems.


text:
Determining a policy to play a game of chance
I'm trying to optimize the expected return from a game of chance, but have quickly realized the problem outclasses the introductory AI course I took in college years ago.  I would appreciate any guidance into what algorithms might be relevant to this type of game.
Game Description
The player has 31 tiles that they can flip over and reveal a hidden color. There are 6 blue tiles, 4 red tiles, and possibly one gold tile.  (The rest are blank.)  The player is only granted 11 attempts to turn over tiles.  Turning over all tiles of a single color results in a reward (a different reward for each color).  Uncovering no complete sets of tiles results in 0 reward.  Due to illustrations on the sets of tiles, revealing a single red or blue tile will also inform the player of the locations of the other tiles of the same color. (i.e. if I flip over one red tile, I know which other three tiles would be red if I were to flip them over too)
The distribution of colors across these 31 tiles is not uniform (but can be assumed as known).  Further, there are only ~100 possible tile color configurations, and these configurations have been discovered by the players of this game.
(Concretely, the game is Faux Hollows from the video game FFXIV; the above description merely removes the flavortext.)
My object is maximize the expected reward over multiple plays of the game; I want to have a program that, given a board state, tells me the optimal tile to flip to get that reward.
What I've considered
My initial thought was to treat this as a Markov Decision Process, since we are changing state, receiving some value/reward based on state change, and I want a policy to maximize that expected reward.  However, the only way I could think to model the state space (i.e. state = which tiles have been flipped and what color they are) results in literal billions of states ($\sum_{1\leq k\leq11}\binom{31}{k}$), which is (AFIAK) intractable for classical solvers of MPDs.
I thought maybe Reinforcement Learning could help, but I'm not already deeply familiar with it.  My cursory glance at q-learning suggested that that high-cardinality state spaces are handled by approximating the value function, but --- because the game has very few states that give rewards --- I worry about how well approximating the value function will work.  (Also, I'm not sure how I would incorporate the 11 tile-flip limit.)
I also considered using a decision tree to determine the board configuration that currently is selected, but this doesn't optimize for reward at all; it merely finds the fewest moves that would indicate which color configuration I have.
Could I have some recommendations of algorithms that I should learn/consider for use in finding an optimal strategy to this game?

Do you have access to the data of the ~100 configurations? This makes a huge difference to the likely reward and to how you would approach a solution.

On a quick look, one other important fact is that coloured tiles appear together in a block on a 2 dimensional grid (so that together they reveal a picture with the colour as a background). This fact can be used by the game player. It can be used more effectively if the illustration is recognised following a tile flip - if that is possible, then it will immediately show where the rest of the tiles of the same colour must be.

@NeilSlater I do have access to that data and it can be assumed correct (i.e. not given to us in game files, but crowdsourced from players and the discovered configs have been stable for months).  And, you are correct --- revealing a single tile of a color will show part of the illustration and the player can derive positions of all other tiles of that color.  However, the player must then choose if hunting for rarer, higher-value tiles (e.g. the 1x1 gold tile) is worth not turning over the rest of the tiles that have known positions (due to the turn limit)

I am not clear from the way that you talk about it, what a "tile colour configuration" is. I was assuming initially that it is the precise location of all tiles - i.e. there are literally only 100 randomised states of the whole board? However, your note about "hunting over the rest of the tiles" implies that is not the case (or at least not the whole story)? Otherwise, if you found e.g. one red tile on move 6, and this narrowed the possible states down to only 4 matching variations, then you could just visit the 4 possible locations of the gold tile.

@NeilSlater There are 100 randomized states of the whole board. A picture might help: https://i.stack.imgur.com/2XxJe.png (dark grey cells are "blocked"/not a tile, white are blank tiles, then there's the blue/red/gold tiles)  Finding one red tile on move 6 does mean that you know where the other 3 red tiles are and you can derive some information about the rest of the board, but not necessarily total information. (1/2)

For example, the first 6 boards in the first row of the image I linked only differ in the location/presence of the gold tile (and one doesn't have a gold tile at all), so until you check all 6 of those spots, you can't actually be sure which config you have. (2/2)

This gives rise to the optimization issue --- on each move, is it better to "explore" and gain information about the board configuration, or is it better to flip over the tiles that you know about already to lock in points.  (e.g. if you have 3 turns left and have only turned over one red tile but no other colored tiles, do you turn over the last 3 red tiles and get a small amount of points, or do you hunt for a possible gold tile and maybe get lots of points or maybe get nothing?)

Please, rewrite the title of this post in the form of a specific question. "Determining a policy to play a game of chance" is not a question and it's also not specific. Thanks.


text:
What object detection algorithm is the best for my particular problem?
I am trying to write a program to put a bounding box around dead fish, and not the live ones, in a video. I have minimal data (~5k frames and ~7k objects in total ) and it is VERY low quality (poor resolution, very exposed, lots of partially obscured fish, etc).
I understand there may not be a "Best Object Detection Algorithm" but for my specific problem, can anyone recommend one? Accuracy is key, speed is secondary.
I have tried YOLOv5 with less than adequate results and I am currently looking into using EfficientDet.

you should describe more in detail what you tried so far, and provide a couple of examples of images if possible. As you put it now we have to trust you you have bad images (and maybe bad annotations) and that you did everything right when trying yolo.

The YOLOv5 predictions couldn't tell the difference between the dead fish and the time stamp in the corner of the video, so in short, it was unusable. I already made it clear that the images are horrible, and I can tell you with confidence that the annotations are perfect. I do believe I did everything right when using YOLO, I just don't think it was the right algorithm for the job.


text:
What reinforcement learning architecture is recommended for multiple outputs in continuous resource management?
I would like to develop an agent to provide resources to multiple machines simultaneously. The overall resources are limited. The agent should distribute the resources in such a way that the machines finish their task as early as possible and switch to standby mode, i.e. the agent returns with each prediction which machines get how many resources. Additionally, the total resources per time interval should be used up as much as possible. Furthermore, a machine has a consumption interval. Resources below the consumption interval are ignored and resources that exceed the consumption interval are reduced to the maximum possible absorption.
Can anyone guide me on how to design such an agent and what topics I can research further?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

Can you please clarify why you want to use RL for this?


text:
What is the intuition about the fact that a set of images will live on a three dimensional manifold?
The following question refers to a dissertation in Bishop's book (see the attachment)
Can someone give me an intuition about the fact that a set of images will live on a three dimensional manifold? I've understood the general context and that each point in the space corresponds to an image. But how does the author come up with the notion of three dimensional manifold?

Could you please put your specific question in the title? "Curse of Dimentionality - Manifold" is not a question and it's also not specific. Thanks.

Maybe it is like a fiber bundle with image valued model fibers. With projections onto a 3d base space (x,y,angle)? Plus some kind of mapping from high dimensional fiber bundle to low dimensional fiber bundle.

Perhaps the mapping/embedding from high dimensions to low dimensions discards some model parameter, like using bounding boxes instead of a mask or something.

I am guessing the total space is maybe [0,255]^(WxHxC). Don't really understand how the projection to (x,y,angle) would look. Perhaps some kind of rotated crop from each possible image. And not sure what the fibers would be, perhaps some kind of feature.

On second thought, a fiber might be all the image crops stacked on top of eachother, or something similar. And (x,y,angle) would be some kind of tag identifying the region.

Perhaps you could alternatively think of it as a volumetric grid that can have different intensities in it, that way strange paths like "moving towards a red circle" might make more sense.


text:
How to quickly train your agent to get a sense of the problem?
I have a custom implementation of DQN. My robot/agent is running in Gazebo with ROS. Though I am trying a very simple task of pushing a cube, the DQN agent is taking too much time. DQN of 300 episode with step limit of 10 took 7.2 hours to train with worst performance. I suspect that the extremely slow training is mostly due to gazebo(+ joint trajectory controller, resetting controllers etc) and ROS. I want to nail down if I am facing one of the following problems:

Problem with my implementation of DQN
Problem is not well defined - For instance, state space is not defined well
Not enough training

DQN had two hidden layers with 64 and 32 neurons. Observation space is 9D, action space is 2D.
Exploration/Exploitation is handled using a decay epsilon greedy policy
max_exploration_rate = 1.0
min_exploration_rate = 0.001
exploration_decay_rate = 0.005
exploration_rate = min_exploration_rate + (max_exploration_rate - min_exploration_rate) * np.exp(-exploration_decay_rate * episode_number)
if np.random.uniform(0, 1) < exploration_rate:
    explore
else:
    exploit

In an ideal case, the exploration/exploitation curve would look like this:

Hi @goldfinch! Could you please explain how your agent is handling the exploration vs. exploitation dilemma? For instance, are you using epsilon-greedy exploration?

@DeepQZero Yes, it is using epsilon-greedy exploration

How are you annealing epsilon? Does it start at 100% and decrease to a small value such as 5% or 2% over time? If so, how many steps does the annealing process take?

Hi @DeepQZero To answer that, I have edited the question, so that I can add few lines of code and a plot.

Have you tried different annealing rates, and other values for remaining hyperparameters? With slow rate of experience, you will want to be more aggressive about using the experience you have. Faster epsilon decay, faster switch of frozen target network, larger batch sizes

Also, at the really slow rate of an action every few seconds, it's worth saving all your data. Keep experience replay records from each run and combine them. Even if your goal is to make an agent that can learn from scratch in this kind of environment, that data has cost you a lot to collect and can be reused


text:
What is the most accurate way of building a Perceptron using only NumPy?
For context, I am trying to write a bunch of neural network programs using no other packages besides NumPy for educational purposes. I am trying to make them as simple as possible, i.e. removing the features that might help training or testing, but which technically aren't supposed to be there.
I am currently writing a Perceptron program, and it seems everywhere I look, it is implemented differently. The program I have written works, I am more concerned about whether or not it is accurate to what a Perceptron is (again, this is for learning purposes). For example: some people seem to use an activation function and others don't- does the Perceptron use an activation function? I know it will function either way, but which is correct?
Here is the code I have right now. Please correct any and all mistakes I might have.
##### Imports #####
import numpy as np

##### Perceptron Class #####
class Perceptron:
    def __init__(self, input_size, num_epochs, learning_rate):
        # Hyperparameters
        self.learning_rate = learning_rate
        self.num_epochs = num_epochs

        # Network
        self.weights = np.zeros(input_size)
        self.bias = 0

    # Forward Propogation
    def forward(self, input):
        layer_output = np.dot(input, self.weights) + self.bias
        return layer_output

    # Back Propogation
    def backward(self, error, input):
        self.weights += self.learning_rate * error * np.array(input)
        self.bias += self.learning_rate * error

    # Train
    def train(self, inputs, labels):
        # Iterate Epochs
        for _ in range(self.num_epochs):
            # Iterate Pairs of Inputs and Labels
            for input, label in zip(inputs, labels):
                # Predict
                prediction = self.forward(input)

                self.backward(label - prediction, input)

    # Test
    def test(self, inputs, labels):
        # Iterate Pairs or Inputs and Labels
        for input, label in zip(inputs, labels):
            # Predict
            prediction = self.forward(input)

            # Print
            print(f'Input: {input}, Prediction: {prediction}, Label: {label}')

# Initialize Perceptron
perceptron = Perceptron(input_size = 2, num_epochs = 1_000, learning_rate = 0.01)

##### Training #####
training_inputs = [[1, 1], [1, 0], [0, 1], [0, 0]]
training_labels = [1, 0, 0, 0]
perceptron.train(training_inputs, training_labels)

##### Testing #####
testing_inputs = [[1, 1], [0, 1]]
testing_labels = [1, 0]
perceptron.test(testing_inputs, testing_labels)```

If you wish to have more than one layer (MLP) then you must use an activation function or multiple layers have no value since it simplifies to a single linear transformation matrix.

I'm looking at your update in the backpropagation...  Are you sure you want to implement the update in that way?  Just thinking about it, it feels like it will have a tough time converging.

In response to your first comment: This is supposed to be a regular Perceptron, not a Multi-Layer Perceptron.

In response to your second comment: I'm not sure- is this correct? It does converge, albeit not well, but I assumed that was because of how basic a model this is.

Please, edit your post to put your specific question in the title so that it's clearer what your main specific question really is. Thanks. Right now, "Perceptron w/ NumPy" is not a question and it's also not specific. Also, note that each post should contain only one question. If you have more, ask each in its separate post.


text:
As someone starting out in RL, could you help me understand the differences between actor-only, critic-only, and actor-critic methods?
I have been reading some medium articles and these three methods pop up a lot. I am wondering what the differences between these are, what are the advantages of one over the other, etc. Also from my understanding, actor-only method is synonymous to policy gradient methods, critic-only method is synonymous to value-based methods, and actor-critic methods use policy gradient methods for the actor part and value-based methods for the critic part. Am I correct? Any help would be greatly appreciated. Finally, what should one keep in mind when considering one or the other for their projects?

You're asking many questions here. I think they can barely be in the same post, provided that you edit your post to emphasize what your main question is and the other questions are just sub-questions that would also help to answer the main one.

Hi, thanks for the reply. My main question would be what the difference is between these three: actor-only, critic-only, and actor-critic methods.


text:
How to establish baselines, with different training loops
My objective is to test out a new algorithm that I designed. However, I am confused whether my methodology to train the networks is correct.
I am just concerned about the training loops:
In the first algorithm (DIAYN, SAC Based Algorithm), the pseudocode follows a high-level pseudocode of:
Run for N-STEPS:
   1. Run for around 5000 steps
   2. Add in Replay Buffer
   3. After each step, after 5K, choose action using policy
   4. Step in env, collect reward, next_obs ..
   5. Update networks by sampling from replay buffer, batch size of 1025

In the new algorithm, I update the same networks, but in a new manner (which is required for the algorithm to do some other stuff.
Run for n epochs:
   1. Run and collect the 1000 samples of next_obs, reward .. by choosing the action from the policy.
   2. Then, run some algorithm (this the new addition) to the replay buffer.
   3. Run a training loop, which runs 1000 times, which updates the networks of a batch size of 128.

As you can see, in the first algo_1, the batch size is 1024, and we collect new samples after each update. Whereas, in algo_2 we update the network 1000 times with replay buffer samples, then we collect new samples.
However, in algo_2, we collect 1000 new samples again. In algo_1, only one sample is added to the replay_buffer after each update. So one new data point is generated from a new updated policy. In algo_1, 1000 samples are generated using the new policy updated 1k times from old replay_buffer.
My question is this, if I wanted to establish a baseline using algo_1, and say that my algo_2 is better as it does X better. Can I do so, if I make sure that the N-STEPS in algo_1 are equal to epochs*1k_training_loop in algo_2?
I apologise for not making this post succinct.

Can you please put your specific question in the title? "Training Loop for Soft Actor-Critic Algorithms" is not a question and it's also not very specific.

Thank you for your input. I hope the details are sufficient now.


text:
Is this the correct way to backpropagate a Neural Network?
I am writing a Neural Network frorm scratch. Below is what I have right now, based off of the math that I think I understand.
##### Imports #####
from matplotlib import pyplot as plt
import numpy as np

###### Activation Function #####
def sigmoid(input, derivative = False):
    if derivative:
        return sigmoid(input) * (1 - sigmoid(input))
    return 1 / (1 + np.exp(-input))

##### Feed Forward Neural Netowkr Class #####
class FFNN:
    def __init__(self, learning_rate, num_epochs):
        # Network
        self.w1 = np.random.randn(30, 5)
        self.w2 = np.random.randn(5, 3)

        # Hyperparameters
        self.learning_rate = learning_rate
        self.num_epochs = num_epochs

    # Forward Propagation
    def forward(self, input):
        self.z1 = np.dot(input, self.w1)
        self.a1 = sigmoid(self.z1)

        self.z2 = np.dot(self.a1, self.w2)
        self.a2 = sigmoid(self.z2)

    # Backward Propagation
    def backward(self, input, error):
        error2 = error * sigmoid(self.z2, derivative = True)
        d2 = np.dot(self.a1.T, error2)

        error1 = np.dot(self.w2, error2.T).T * sigmoid(self.z1, derivative = True)
        d1 = np.dot(input.T, error1)

        self.w1 -= d1 * self.learning_rate
        self.w2 -= d2 * self.learning_rate

    # Train
    def train(self, inputs, labels):
        for _ in range(self.num_epochs):
            for input, label in zip(inputs, labels):
                self.forward(input)
                self.backward(input, self.a2 - label)

    # Test
    def test(self, inputs):
        for input in inputs:
            self.forward(input)

            print('Image is a', 'ABC'[np.argmax(self.a2)])
            plt.imshow(input.reshape(5, 6))
            plt.show()

# Initialize Neural Network
feed_forward_neural_network = FFNN(learning_rate = 0.1, num_epochs = 100)

##### Training #####
a = [0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1]
b = [0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0]
c = [0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0]

y = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]])
x = [np.array(a).reshape(1, 30), np.array(b).reshape(1, 30), np.array(c).reshape(1, 30)]

feed_forward_neural_network.train(x, y)

##### Testing #####
feed_forward_neural_network.test(x)

However, after looking at someone else's code, they have the same thing except the backward function does this instead:
    # Backward Propagation
    def backward(self, input, error):
        error2 = error
        d2 = np.dot(self.a1.T, error2)

        error1 = np.dot(self.w2, error2.T).T * sigmoid(self.z1, derivative = True)
        d1 = np.dot(input.T, error1)

        self.w1 -= d1 * self.learning_rate
        self.w2 -= d2 * self.learning_rate

Notice the missing sigmoid(self.z2, derivative = True) multiplication by the layer 2 error.
Both of these functions converge just fine, but obviously one of them is wrong. Which one, and why?

Your $d_2$ is the gradient used to update $w_2$, which is of course $\frac{dL}{dw_2}$. To compute this gradient, using your notation:
$$ \frac{dL}{dw_2} = \frac{dL}{da_2}\frac{da_2}{dz_2}\frac{dz_2}{dw_2} = err \cdot \sigma'(z_2)\cdot a_1$$
So your version seems to be correct.
One possibility is that the forward is also different, and there is no sigmoid after the second layer in your colleague's network (which is often the case for the last layer). In which case their version would also be correct.

I check their forward, as that was what I thought originally, but they're the same. Good to hear I haven't misunderstood though. Thanks.

Please don't forget to accept and/or upvote answers. Thanks.

The latter one seems to correct implementation assuming your loss function is binary cross-entropy.
The partial derivative for cross-entropy w.r.t z2 is self.a2 - label in your example. You can check mathematics in more detail here.
So, there is no need to multiply it again with derivative of sigmoid.

I don't think I'm using Binary Cross Entropy Loss? The error I backpropagated was just the output of the network - the real label. Cross Entropy involves logarithms does it not?

May I know, what is the loss function that you are using here and if you are using least squares then your version seems to be fine. Though people generally use binary-crossentropy for binary classification (and sigmoid activation for last layer). Finally, both implementation be right based on the loss function that you are using.


text:
Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of $p$ (3.4) and $r$ (3.5)
I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:

Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of the three argument function $p$  (3.4) and the two-argument function $r$  (3.5).

Here are the 2 functions:
(3.4) :
$$p(s'|s,a) = P(S_t=s'|S_{t-1}=s,A_{t-1}=a)=\sum_{r\in\mathcal{R}}p(s',r|s,a)$$
3.5:
$$r(s,a) = E[R_t|S_{t-1}=s,A_{t-1}=a]=\sum_{r\in\mathcal{R}}r\sum_{s'\in\mathcal{S}}p(s',r|s,a)$$
Here's my solution:
$$v_{\pi}(s) = \\ E_{\pi}[G_t|S_t=s]= \\ E_{\pi}[R_{t+1}+\gamma G_{t+1}|S_t=s]= \\ E[R_{t+1}|S_t=s]+\gamma E_{\pi}[G_{t+1}|S_t=s]= \\ \sum_{s', r,a}p(s',r|s,a)r\pi(a|s)+\gamma \sum_{s',a}p(s'|s,a)v_{\pi}(s')\pi(a|s) = \\  \sum_a\pi(a|s)r(s,a) + \sum_a\pi(a|s)\sum_{s'}v_{\pi}(s')p(s'|s,a)= \\ \sum_a \pi(a|s)[r(s,a)+\gamma \sum_{s'}p(s'|s,a)v_{\pi}(s')]$$
$$v_*(s) = \\ \text{max}_a q_*(s,a) = \text{max
 }_aE_{\pi}[G_{t}|S_t=s,A_t=a]= \\ \text{max}_aE_{\pi}[R_{t+1}+\gamma G_{t+1}|S_t=s,A_t=a]= \\ \max_a \sum_{s',a}p(s',r|s,a)r+\gamma\sum_{s'}p(s'|s,a)v_*(s')= \\ \text{max}_a r(s,a) + \gamma \sum_{s'}p(s'|s,a)v_*(s')$$
$$q_\pi(s,a)  = \\ E_{\pi}[G_{t+1}|S_t=s,A_t=a]= E_{\pi}[R_{t+1} + \gamma G_{t+1} |S_t=s,A_t=a]= \\ E_[R_{t+1}|S_t=s,A_t=a]+\gamma E_{\pi}[G_{t+1}|S_t=s,A_t=a] = \\ \sum_{s',r}p(s',r|s,a)r+\gamma \sum_{s'}p(s'|s,a)v_{\pi}(s') = \\ r(s,a) + \gamma \sum_{s'} p(s'|s,a) \sum_{a'}q(a'|s')\pi(a'|s')$$
$$q_*(s,a)=r(s,a) + \sum_{s'}p(s'|s,a)\text{max}_{a'}q(a'|s')$$
Is this correct?

Can you please edit this post to focus on one bellman equation at a time and write the other bellman equations in other posts? Moreover, please, don't just ask "is this correct?". Maybe try to argue why or not.

@nbro as you said in a previous comment, I asked separate questions for each equation. The one that you referenced is only 1 of them. Shall I delete this question or?

Yes, exactly, you should delete it. I voted to close this as a duplicate of only 1 because I cannot do it for multiples ones. Thank you.


text:
Can I create and train a feedforward layer with only K active connections?
I had an idea for a layer, but I'm not sure if it exists already and/or if it's implementable in tensorflow.
I would like to have a layer that is similar to a Dense layer, in the way that it's connected to every nodes of the previous layer, but only K connections can be "active" (weights >= 0), and all the other ones will have their weights equals to 0
Which ones of those K connections will be learned by the network, not chosen randomly when building it.
So for example, in a Dense layer that goes from 1000 to 2 features, I would have 10002 + 2 = 2002 parameters that can be different from 0.
In my layer, if I chose K = 10, I will have only 210+2 = 22 parameters that can be different from 0.
Any way I can do that and train a network with it?

perhaps Dropout is what you're referring to

Dropout is a regularization method used during the training to prevent overfitting. It randomly choses which connection to "drop", but should not be used at inference time.
I am looking to have my trained network sparse :)


text:
What is the logic behind using a trained classifier's gradients to synthesize controllable image?
In the controllable image synthesis, we are manipulating a noise vector z such that our generator ( in our GAN model ) creates images that the desired feature exists. For instance, take the feature of "smiling". In order to learn how to change the z (noise) vector , the technique is described as following :

Train a classifier ( besides the GAN model, just a single discriminator neural networks as a classifier) that classifies an face image according whether there is a smiling person on the image or not.

Give the image generated by the GAN model with the initial noise ( say z1) to the classifier.

Compute gradients of the score with respect to z1. And finally Perform gradient ascent on z1 vector in the direction of improving the score.

I cannot understand how to gradients are used to change the z1 vector in  a way that it now leads the GAN model to create "smiling" faces, could you please help ?

It's unclear whether you're only interested in the GAN or some other model. Can you please edit your post to clarify this? I would recommend that you focus on a specific model.

What do you mean by "GAN or some other model", could you please explain a little bit I did not understand ? There is both a GAN model and another Discriminator model ( not the GAN's discriminator instead just a single discriminator neural network ) in the question.


text:
How to find AI research papers where authors provide source codes that will reproduce the experiments described in the paper?
It is important for a research paper to include raw data and code for scientific replicability, verifiability, and falsifiability.
However, as noted by nbro, roughly 5% percent of papers have code that precisely reproduce the experimental results. About 20% of papers might have implementations coded by other researchers, but they usually differ from the original code and implementation of the author.
Is it possible to find AI research papers where the author provide source codes aimed to reproduce the experiments described in the paper and produce exactly the same plots?

Wait, note that I didn't say for sure that only 5% of papers include source code. What I said or at least meant is that, from my experience, I think that only roughly 5% of the papers have code that you can use to really reproduce the experiments in the paper (but this estimate could be wrong - it's just a very very rough estimate). Maybe I should have been more precise in that comment. Please, edit your post to fix that.

@nbro Does it sound better now?


text:
How to represent the state and action space when modeling card game Love Letter using Markov Decision Process
Now my task is to use model checking to find a best strategy for card game Love Letter, and I need to model the game using Markov Decison Process first.
I have done a lot of resaerch and I decided to use "what cards are in the deck and what cards are in the other player's hand" to be my state.
But I am not sure which will be the best way to represent the states, because I think the states should include all the information I need and I can make action base on these states that I have got.

I don't know the game "Love Letter"; can you add a reference to it, please?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

I have added a reference about card game Love Letter, thank you :)

I have read the description on the link. You should note the game as it is usually played requires secrecy (not knowing what is in other players' hands), and that secrecy is a major part of the game. This means that it is not possible to construct an MDP for the game as it is usually played. Are you proposing any changes to game in order to make it into an MDP? It may help to understand your underlying goal here - e.g. are you trying to learn about MDPs and picked this game because you thought it might be a challenge? Or are you trying to write a learning agent specifically for Love Letter?


text:
Basic question about gradient for nominal regression
Say that we want to binary-classify images using a sigmoid function with the entropy-loss function. This algorithm is quite slow.
The sigmoid function is:

I find that this could be traced to the $L(y,y') = -log(y')$ for $y=1$ and $L(y,y') = log(1-y')$,  for $y=0$.
EDIT But actually that is a great, desired property! The problem is the
second term in the total derivative respect to the Loss ($L$):
$$dL = dL/dy * dy/dz ...$$
because it is $dy/dz$ is very small for large or small values of $z$:

Linear regression wouldn't have this problem as that term is 1 (or non existent). But there could be an "explosion" of the gradient because of the $Loss = (y-y')^2$.
Question:
So I'd suggest to use a cutoff value for y' in, say, 10 if your labels are (-1 for not, 1 for yes).
Has this been tried before?  If so what were the results?
Is this a reasonable approach or would it just not work?

why are you saying that the loss goes to 0 when we have a good prediction is a problem? also for MSE this holds

@AlbertoSinigaglia Can you check now? I expanded to my best.

Take the BCE and substitute x with the sigmoid function, and you will see that you have a linear gradient (BCE undoes the sigmoid)


text:
Object detection on UAV images
So, the idea is that I have a custom set of UAV images. I zoomed them and I used LabelImg in order to draw the rectangles. I trained the model and when I run the code for object detection it does detect the object it was trained for. My concern is that I used zoomed UAV images in order to train and unzoomed (original size images) images for object detection. Can this be the problem? What could be wrong? I used this code for train: https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html and this code  https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/auto_examples/plot_object_detection_saved_model.html#sphx-glr-auto-examples-plot-object-detection-saved-model-py for object detection.

Did you mean that the model does NOT work as expected after training? Or what problem are you referring to? In case yes, the spatial resolution, i.e. meters per pixel, should be the same in training and inference. Not only that, if you train on images gathered from one place and test on images gathered on a different place that might also inficiate a lot the model performances.

I don't see any rectangles in object detection/inference stage, on the outputted image...

The yes, if you're sure about the code you got your culprit. Zoom also your images in inference phase, or even better, skip unnecessary preprocessing steps (resizing, no metter what type, always introduce artefacts), and train and infer on the original spatial resolution.

How many image in your dataset? Do you use only drone image or other image (how many different label in your dataset)?


text:
ML model giving rank errors on 3D layers on converting 2D images to 3D models
i am currently working on a model to convert 2d images to 3d models through a ml model. For this i have taken into reference a research paper which had this diagrammatical flow of layers & i have tried to put that into work but in vain. Would really like to know where i am going wrong, looking forward for help and suggestions. Thanks!
reference image for model-
my code-
image = Input(shape=(None, None, 3))

# Encoder
l1 = Conv2D(64, (3,3), strides = (2), padding='same', activation='leaky_relu')(image)
l2 = MaxPooling2D()(l1)
l3 = Conv2D(32, (5,5), strides = (2), padding='same', activation='leaky_relu')(l2)
l4 = MaxPooling2D(padding='same')(l3)
l5 = Conv2D(16, (7,7), strides = (2), padding='same', activation='leaky_relu')(l4)
l6 = MaxPooling2D(padding='same')(l5)
l7 = Conv2D(8, (5, 5), strides = (2), padding = 'same', activation = 'leaky_relu')(l6)
l8 = MaxPooling2D(padding='same')(l7)
l9 = Conv2D(4, (3, 3), strides = (2), padding = 'same', activation = 'leaky_relu')(l8)
l10 = MaxPooling2D(padding='same')(l9)
l11 = Conv2D(2, (4, 4), strides = (2), padding = 'same', activation = 'leaky_relu')(l10)
l12 = MaxPooling2D(padding='same')(l11)
l13 = Conv2D(1, (2, 2), strides = (2), padding = 'same', activation = 'leaky_relu')(l12)

# latent variable z
l14 = Reshape((60,512))(l13)
l15 = Dense((512),activation = 'leaky_relu')(l14)
l16 = Dense((128*4*4*4), activation = 'leaky_relu')(l15)
l17 = Reshape((60,4,4,4,128))(l16)

#Decoder
l18 = UpSampling3D(size = (3,3,3))(l17) #-->throws error->IndexError: list index out of range
l19 = Conv3DTranspose(60, (8, 8, 8), strides = (64), padding='same', activation = 'leaky_relu') (l17)
l20 = UpSampling3D((3,3,3))(l19)
l21 = Conv3DTranspose(60, (16,16,16), strides =(32), padding='same', activation = 'leaky_relu')(l20)
l22 = UpSampling3D((3,3,3))(l21)
l23 = Conv3DTranspose(60, (32, 32, 32), strides = (32), padding='same', activation = 'lealy_relu')(l22)
l24 = UpSampling3D((3,3,3))(l23)
l25 = Conv3DTranspose(60, (64, 64, 64), strides = (24), padding='same', activation = 'leaky_relu')(l24)
l26 = UpSampling3D((3,3,3))(l25)
l27 = Conv3DTranspose(60, (64, 64, 64), strides = (1), padding='same', activation = 'leaky_relu')(l26)

model3D = Model(image, l27)

error-
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_17/3800219535.py in <module>
     34 #Decoder
     35 l18 = UpSampling3D(size = (3,3,3))(l17) #-->throws error->IndexError: list index out of range
---> 36 l19 = Conv3DTranspose(60, (8, 8, 8), strides = (64), padding='same', activation = 'leaky_relu') (l17)
     37 l20 = UpSampling3D((3,3,3))(l19)
     38 l21 = Conv3DTranspose(60, (16,16,16), strides =(32), padding='same', activation = 'leaky_relu')(l20)

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    975     if _in_functional_construction_mode(self, inputs, args, kwargs, input_list):
    976       return self._functional_construction_call(inputs, args, kwargs,
--> 977                                                 input_list)
    978
    979     # Maintains info about the `Layer.call` stack.

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _functional_construction_call(self, inputs, args, kwargs, input_list)
   1113       # Check input assumptions set after layer building, e.g. input shape.
   1114       outputs = self._keras_tensor_symbolic_call(
-> 1115           inputs, input_masks, args, kwargs)
   1116
   1117       if outputs is None:

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _keras_tensor_symbolic_call(self, inputs, input_masks, args, kwargs)
    846       return tf.nest.map_structure(keras_tensor.KerasTensor, output_signature)
    847     else:
--> 848       return self._infer_output_signature(inputs, args, kwargs, input_masks)
    849
    850   def _infer_output_signature(self, inputs, args, kwargs, input_masks):

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _infer_output_signature(self, inputs, args, kwargs, input_masks)
    884           # overridden).
    885           # TODO(kaftan): do we maybe_build here, or have we already done it?
--> 886           self._maybe_build(inputs)
    887           inputs = self._maybe_cast_inputs(inputs)
    888           outputs = call_fn(inputs, *args, **kwargs)

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _maybe_build(self, inputs)
   2657         # operations.
   2658         with tf_utils.maybe_init_scope(self):
-> 2659           self.build(input_shapes)  # pylint:disable=not-callable
   2660       # We must set also ensure that the layer is marked as built, and the build
   2661       # shape is stored since user defined build functions may not be calling

/opt/conda/lib/python3.7/site-packages/keras/layers/convolutional.py in build(self, input_shape)
   1546     if len(input_shape) != 5:
   1547       raise ValueError('Inputs should have rank 5, received input shape:',
-> 1548                        str(input_shape))
   1549     channel_axis = self._get_channel_axis()
   1550     if input_shape.dims[channel_axis].value is None:

ValueError: ('Inputs should have rank 5, received input shape:', '(None, 60, 4, 4, 4, 128)')```

This looks like a duplicate question of your previous inquiry about this paper.

yes...and still unable to solve >_<

Please read these answers: You shouldn't post the exact same question on multiple sites: https://meta.stackexchange.com/a/64073/282094 and How do I get an answer for my unanswered question: https://meta.stackexchange.com/a/7054/282094 - Thanks.


text:
What is the difference between planning and model-based machine learning?
I am familiar with planning. Given a description of the possible initial states of the world, a description of the desired goals, and a description of a set of possible actions, the planning problem is to synthesize a plan that is guaranteed to generate a state which contains the desired goals. The problem can be formalized as a MDP, the known transition and reward functions allow us to determine the next state and the next reward, according to the current state and the action to execute.
But, I do not really know model-based machine learning. And, when I read articles about this subject, I do not understand the difference with planning.
Thank you.

In RL (which seems to be what you’re asking about) you would use a model to do the planning, ie rollout environment steps without actually interacting with it.

Thank you for your response. Yes, but in planning, we also use a model. So I don't understand the difference

I think this question is a bit off. If you do know planning, you should clearly be able to understand the difference to model-based machine learning -- provided you know what the latter is. So it seems you are actually simply asking "What is model-based machine learning? Please explain!". This seems to be an overly abstract question and I do not expect that anybody is going to answer it for you unless you make some effort in explaining what you think it is and why it's still unclear to you where the differences lie. (I would not be surprised about downvotes right now!)


text:
The result of back propagation for a neural network
I have created a neural network that feeds an image into a convolutional neural net, then feeds the flattened output of this network into an artificial neural network.
I have a feeling that my backward propagation is not working properly.
Mind you, I wrote all of it from scratch in vanilla javascript.
The final point of control for me as the programmer is when I input the desired output into the backwards propagation algorithm to calculate the deltas and update the weights. After running backwards propagation I run forward propagation with identical input data to the data that was put into the network before backward propagation to see the change in the output layer based on the new weight values.
For each individual output of my neural network's final layer if it is between the output before backward propagation and the desired output then the network successfully propagated backwards for that output. In my network I had 4 outputs. I took the average of how many outputs were correct for each time back propagation was run and on average I got 0.773206751. About 77% of the time the gradient moved in the correct direction. So 23% of the time the network returned an output in the incorrect direction or returned an output that surpassed the desired output(went too far in the right direction).
In a neural network after running backwards propagation should the output after back propagation be closer to the desired output 100% of the time? What could cause these types of inaccuracies(or are there too many potential causes)?
It is unfortunately a problem that I cannot debug(because there are too many small numbers to keep track of). Unless there is a way to debug this problem, but I'm not sure.
architecture
input to CNN[102, 102, 4]
conv: kernel[9*9] stride:3 output:[32, 32, 16] act: ReLU
conv: kernel[5*5] stride:3 output:[10, 10, 32] act: ReLU
max-pool: pool_dim[2*2] stride:2 output:[5, 5, 32]
flatten: from [5, 5, 32] to 800 neurons
inp layer: 800 neurons
hidden layer: [800, 402] act: ReLU
output layer: [402, 4] act: for first 3 in output softmax for the fourth in output linear.
it is an a2c network.

I'm having a bit of a trouble understanding your question. Maybe some context would help. Are you trying to design a new custom backpropagation method or loss function, or are you still using the standard cross-entropy loss?

@PeaBrane It uses a2c. let's say that I get output of [0.9, 0.1] from my neural network. I run backward propagation and set desired output to [0.91, 0.09]. Ideally if I were to rerun the program with the same input after backwards propagation and get [0.905, 0.095] the propagate backward clearly seems to be working. lets say instead I get [0.895, 0.105] instead. Does this mean that my neural network is not working? In my case, I would have original output: [0.34, 0.28, 0.38, -0.07], desired output: [0.36, 0.27, 0.37, 0.98], and end up getting [0.33, 0.275, 0.375, -0.02]. 75% ofThemMvInRtDirec

of them move in the right direction. Do you see my problem now. It is not really gradient descent but more of the function of a neural network. A neural network that is probably not implemented properly. bc i implemented it.


text:
Define possible?
In Reinforcement Learning, policies are defined in terms of possible actions (see for instance page 58 of the book by Sutton et al.). So, is any action that an agent has in its repertoire always "possible", even when it has a policy-value of zero? Or are possible actions dependent on the state?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

Suppose that the action space is defined as $\mathcal{A} = {1, 2, 3}$ and the state space is $\mathcal{S} = {a, b, c}$. If we can only take actions $1$ and $2$ in state $a$ then what this means is that any policy must have $p(3 | a) = 0$ by definition. Here I guess I am trying to emphasise that whilst action $3$ is still part of the action space for state $a$, it is not a possible action, and so any policy must assign 0 probability to selecting it.

Clear. Thank you.