text:
How do you pass from the 192 depth in the first tensor to the 256 in the second tensor?
I thought 112x112x192 depth tensor convoluted with 3x3x192 would give 56x56x(192x192)
But this is different. How do you pass from the 192 depth in the first tensor to the 256 in the second tensor?

I suppose that 3x3x192 in Conv. Layer refers to "Kernel" size - you may think of it as a "scanner" that scans through an input tensor. This kernel (scanner) has a length and width of 3, and a depth of 192. However, the result of one kernel application, one scan is just one, scalar number. So one kernel in convolution results in a flat, 1 dimension depth result tensor. In your case: 56x56x1
So what about this 256 depth in output dimensions? I think, that this architecture may use 256 different filters, 256 different kernels/scanners. It is not stated anywhere in your question, but I would suppose that's the case, as a often-used technique.
That basically means that the convolution process is performed on your left-side tensor 256 times with 256 different kernels and results are stacked to one output tensor of depth 256.
This is used for example, when you want to train your model to recognize many (in this case 256) different features in the input tensor.
You may find the picture below helpful, note that the depth of result (H) is not related to input size, and it's just number of filters/features we want to detect.

Source of image: here


text:
How does "noises" in computing convolution affect the model precision and the training speed?
Consider the discrete convolution written in a matrix form, if a small amount $s$ of the zero entries (represented as white blocks) are deviated from zero, can the model precision or the training speed be increased? A "small" amount means, for example, $s \leq n$, where $n$ is the dimension of the matrix.

I guess this idea is similar to an inverse procedure of dropout/dilution, where a small amount of matrix entries are replaced by zero. Here, a small amount of zeros are replaced by some nonzero random values.

As a newbie to AI, I am not familiar with many historic papers on the development of AI. Probably some researchers have already analyzed this several years ago.
Any related papers or conference presentations are highly appreciated.

After even one training step you'll find that none of them are exactly zero. Maybe instead you should talk about adding some large noise to some random elements regardless of whether they are zero?

1D convolutions are also fairly unusual. Are you really thinking of a 1D convolution, are you thinking of convolutional neural networks which are usually 2D, or are you thinking of a layer of a traditional dense neural network with the weights being a 2D matrix which is not a convolution?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What computer vision techniques can help me select the best image for object detection inference?
I have a lot of duplicate images. I need to make a selection to reduce the amount of images the Mask RCNN model will perform inference on.
In every collection of duplicates, the images slightly differ. One is brighter, darker, different shadows, different reflection, etc. But all duplicate images are essentially capturing the same object(s) from the same angle with the same background, etc.
What computer vision methods can help me to know before inference what image would produce the most results after inference?
I have tried:

edge detection (cv2.Canny followed by sum()): Selecting the image with the most edges. In my use case there does not seem to be a strong predictive relation between amount of edges and amount of objects detected.
...

clarify what you mean by "produce the most results after inference?"

to get an idea of the scale, how many images are there? is it a public or private dataset? did you try any of techniques mentioned here https://medium.com/mlearning-ai/a-scalable-solution-to-detect-duplicate-images-97d431c2726d#60bc

What I mean by "produce the most results" is that the model would detect the highest amount of objects. The model does not always detect the same amount of objects for every one of the duplicates. It seems that certain properties of the image increases the amount of detections that the model makes.
The dataset would in total contain more than 100 000 images and it is private.
I haven't tried them. Thank you for the link. Will look into that!

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why can't traditional neural networks learn to perform the same tasks that attention layers do?
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:

Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on predicting $t_{n+1}$ (with all the embedding / layernorm / skip connection stuff that transformer models have)
Attention - take $t_n$ plus information from $t_1,...,t_{n-1}$ relevant to $t_n$ computed in the attention layer (plus positional encoding) and feed that into a neural network to predict $t_{n+1}$

In one case you're putting everything into the NN, equally weighted. In another, you're taking certain information from each token weighted by relevance to $t_n$.
My question is: why can't the straight NN learn this? What was wrong with the straight NN model so that we needed into introduce attention? If it was helpful to upweight certain things by a relevance metric, why wouldn't the NN learn that?
My only thought so far is long-range dependency: that a NN might forget information far back in the sequence (as in general it will be less useful). For example, 'he walked into the kitchen, took the chocolate from the drawer, opened wide and put it into' – to predict the next word you really need to look all the way back to the start of the sentence.
Thanks for your help!

May I suggest asking your question in the title?  'Why can't traditional neural networks learn to perform the same tasks that attention layers do?`

Edited. Thanks!

I believe it boils down to asking why a neuron (output=relu(A*x+B*y)) can't learn to multiply its inputs (output=x*y) which is what an attention head does a lot of


text:
Will the AI be able to formulate scientific principles or theories missed by humans in the future?
I do not know at all how AI works.
After checking out the first open AI system available to the public, ChatGPT, I am curious whether systems like this could contribute to scientific theory in the future.

This already happened:
What directly comes to my mind is AlphaTensor that developed a novel method for matrix multiplication. Matrix multiplication is a computational expensive and often used operation. So it is safe to assume that over the past decades many humans have put effort into finding faster algorithms for this.
There are also scientific papers written by an AI. But without a close look at these papers, I would hesitate to claim that they contain novel scientific principles or theories.

Yes, from first principles.
Discovering new scientific theories is a mixture of following a set of mathematical logic principles and using creativity to come up with something new and useful.
Regarding the first criteria, logic and math, it is clear that computers are superior to humans in carrying out long chains of logic inferences, thanks to the precision of the hardware on which they are built.
Creativity is often mentioned as something differentiating humans from machines, but actually machines have access to the highest level of creativity: random number generators. Nothing is more creative than just randomly picking an option!
While current systems are still primitive in manipulating results and having the big picture necessary to put them to use, the state of the art is rapidly advancing. For instance, given the recent advances in NLP AIs will soon be able to chain together different concepts according to common patterns, such as the one used by scientists. Chain-of-Thought techniques are a good example. Now imagine gluing this NLP system to a source of truth, such as a tool (say, an equation solver or a microscope) able to test the proposed theory with an experiment. By brute forcing over many hypotheses you can imagine how these systems may eventually come up with relevant theories.
Assuming the above, that is LLMs or chatGPT-like systems are in principle able to formulate valid theories, are these systems in practice able to tame the combinatorial explosion in logic formulas arising when probing non trivial theories? I would lean on saying no, models which are limited to predicting the next word would be inefficient in performing the smart reasonings required to avoid combinatorial explosions, even though you may imagine that clever tricks could be devised to remind the language models the laws and mental models needed to trim the tree of possibilities at each step of the reasoning process.
What could do it then? I believe a substantial leap in capabilities would come from a fully multimodal model, able to also Chain-of-Thought reason with images, 3d representations, embodiments and physics simulations. In a sense these modalities are summarising many words at once, cutting down a continuum of possible inference chains. This is often called Visual Thinking.


text:
How important is to fix this common Tensorflow warning?
I have seen some people with the same problem, this is the warning:
W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia

W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia

W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

I have tried installing a few things without luck.
Currently I am testing it in Colab.
What are these libraries and are they worth installing?

If it does not mess up with your code, don't worry about it

@Minh-LongLuu , Disagree

what kind of instance is your Colab notebook running on?

@Conic standard using GPU, and as far as I know the code uses pytorch only, this is the repo https://github.com/Megvii-BaseDetection/YOLOX, and the google colab https://colab.research.google.com/drive/1tNWiufDYxB4lNShoIzCmGsY1xLhF-w9s#scrollTo=vHVgSOFRzTI7 thank you for replying


text:
predict a one-hot vector which sum is 2 (so not really a one-hot vector)
So I basically have a $n$ classes. I have an input. My data is organised in the following way: each input has a label, this label is 2 classes. It can be twice the same class, or two different classes.
So I want to output a vector with only 0s, 1s and 2s which sum is 2. Similarly to what we would do if the label is only one class in a regular classification model. I'm just a bit confused about how we would handle the standard the prediction is the argmax because here we would have to make arbitrary rules about when to predict 2 classes and when to predict two distinct classes (like in this case [0, 0.5, 0.4, 1.1, 0, 0], it's a bit hard to determine if class 4 is predicted twice or if it's one time class 2 and one time class 4, etc ...)
Is this a thing, is there a scientific term about it? Am I not supposed to do that because it breaks something fundamental about classifiers? I'm planning on training Neural Networks, XGBoost, Random Forest, Extremely Randomized Trees and comparing the different methods (are there other methods where this would be possible? basically predicting a vector whose sum is 2).

What is the value of $n$? If it is not too large, you can create all combinations of it, and use regular one-hot vector. For example, if you have 4 classes, consider using $2^4$ = 16 output nodes

Very good idea although $n$ is very large in my case, around 30, making this solution probably too computationnally expensive.

If computation allows, I suggest setting the baseline as the One-vs-All problem first. In your case, train $n$ models, each for a specific class.

@Minh-LongLuu So training $n$ models that output a vector of size 3? How do I compile the outputs, it doesn't seem as straightforward as doing an argmax .. or maybe it is easy and i'm missing it?

Or maybe $3n$ models, 3 because for each category i have three models (binary classification): class 0 for this or something else, class 1 for this or something else and class 2 for this or something else. But I would have the same problem of compiling the outputs ..

I imagine it would be 30 models with 3 outputs: one is "Is this class class i?", two is "is the other class also this class?", and third is "no, it is another class, different from i".

30? What do you mean?

And how do you aggregate the results if for example, one model predicts this is class 2 and another one predicts this is class 1? Or I guess we would take the two maximum predictions that are coherent? But what if the maximum is class 1 and the second maximum is class 2, it's not possible because sum is 2, so we look for another one that says maximum is class 1?


text:
What are the main contributors to memory overload in NNs
In this article they talk about computational considerations for convnets.
I used to think that the main bottleneck were the weights. They name:

activations and gradients

input data (like images)

Are activations meant as both the weights and the (single) function that operates?

Activations are the outputs from each layer. I think the reason they are significant is because of batching. You don't run 1 image through at a time, but 100, 1000, however many you can fit in memory at once. Whereas there is only one copy of the weights.

@user253751 interesting, yet you only need the L-1 output for the L layer, but probably all the outputs have to be held in memory before backpropagation..


text:
CNN without actuators
After training CNNs without actuators, I have an idea to compare their weights with each other using image mirroring. I am looking for ideas about reality perception of CNNs in this way.
What might CNN models need to act in next situation? How I can understanding that this is not problemomatic?
*The question should be accepted philologic. This is important for finding short way algorithms.

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

CNN's do not have anything to do with actuation. They transform input in a specific way. Most CNN's do not have actuators. They might be used in systems that have actuators. Or used as a subcomponent of a RL algorithm that actually is tasked with making decisions. But lacking actuation does not affect a CNN or change them in a fundamental way. I think you need to re-phrase this question for additional clarity or revisit your understanding of CNNs. What do you mean by actuator?

Tethering that provide the output.  Connection points are required for inputs and outputs as a result.  I do not equate the operation of a network model without output tethering with sending output weights to the GC.


text:
Inference time of VGG16 when initialised with different weights
I’m trying to understand the differences in inference time and training time between two models:
VGG16 with weights initialised from a Glorot uniform distribution and the same network with the only difference being that weights are initialised to the imagenet values.
Obviously the performance of the latter is higher and training time on my dataset is lower.
However, the training time per epoch is also lower. I’m not sure why this is, I also measured the inference time which was lower and roughly explains the difference in training time. But why would the inference time be lower given that it’s the exact same model with the same number of parameters?
Note that all tests were performed on the same device. VGG16 used was from the Keras library

Can you share the time and time differences and the benchmarking code you used

Are the weights the same data type ?


text:
Request for assistance with converting legal contracts to environment for DQN
I want to convert the Extractive QA task as a Reinforcement Learning Problem Statement. So I want to integrate NLP problem into Reinforcement Learning and see if my results were achieving better when compared to only NLP model and see why it is performing better or why it is performing worse
I have a set of legal contracts and I want to use an extractive QA task to extract relevant information such as payment terms and insurance. I have tokenized the data and have a Hugging Face transformer model for this. My ultimate goal is to use a DQN where the transformer model serves as the base model.
To achieve this, I need to convert the tokenized data into an RL environment and train the combined model on a group of contract environments. However, I am currently facing difficulties in creating the environment for DQN.
I would greatly appreciate it if you could offer any advice or guidance on how I can proceed with this task. I am eager to learn and am open to any suggestions you may have.
Thank you very much for your time and consideration.

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

The problem is, like how RL agent is been used to play video games using images, in a similar manner I want to test, if RL agent is able to perform better than Extractive QA transformers, when I am using DQN and Transformer as base model for DQN. I want to see how RL DQN+Transformer is performing when compared to only Transformer. So here I want to convert legal documents into environment and action space to get start and end logits or Answer token to extract from document based on question I ask to the model. So I have fields like payment terms, insurance, Indemnity, etc in my legal contract

For Video games using RL agent to play the game, trained on images of game play, similarly I want to use RL agents to perform Question Answering NLP Task for my legal documents, where I want to see is it performing better or worse and why is it performing better or worse..


text:
Creating or using a corpus for a chatbot interaction modelling for an existing intelligent tutor system
I'm currently researching and building a chatbot to link to an existing intelligent tutoring system for the domain of the topic of relational databases. The current intelligent tutoring system is rudimentary (forms with checkboxes and the sort, a quiz type) and anything could serve the purpose of modeling the dialogue that would be better than just reading a textbook or having a passive attitude in the classroom. The main problem is that I don't have any corpora to utilize, besides the hardcoded questions, and I don't have any time at all to create a better corpus from scratch.
My thesis adviser gave me the starting point for a dialogue model: the chatbot would ask a question and the student would try to answer. Depending on the feedback, the student could ask "how", "why", or "explain".
There is research showing that using theoretical frameworks to guide the learning goals of a chatbot could help students learn more effectively by moving from passive to active learning. It is really necessary to understand learning objectives before starting to develop. A theoretical framework can guide the dialogue modeling.
This happens, for instance, with CodingTutor which is an agent to help students learn how to program. It uses the revised Bloom taxonomy to combine different types of knowledge to help the person learn. This is still complex because there are many types of questions, answers, and other types of interaction like solving a programming exercise. The CodingTutor approach could be an option if I didn't lack time.
On the other hand, other ITS with a chatbot module was implemented by using the user's own questions to a previous iteration of the ITS chatbot and Google queries. The learning goal was only memory retention. The students asked "Why", "Advantages/Disadvantages", "Application of", "How" and "Who". This is a more reasonable approach than CodingTutor but with a caveat: I have no corpus.
There are other ITSs, like AutoTutor, that use expectation-and-misconception tutoring (EMT). Still, the complexity is greater and I would have to hardcode every single misconception to a single expectation.
My question is if there are some corpora that I could use for the simple learning objective of memory retention using "why", "how", and "explain" for the topic of relational databases.

EDIT: given that it's thesis research, according to my adviser, the topic of relational databases of the intelligent tutor isn't fundamental, but it has to be useful for instance K-12 Geography. The chatbot is a complementary module to an intelligent tutor, and the intelligent tutor is still bare bones that any corpus - with some learning objectives in mind - would suffice. Also, the end result of that said intelligent tutor far into the future would be to be cross-domain (not open-domain), for instance, to be able to be adapted for different domains in a standalone fashion (different training sets, different corpora, the works).
However, given that there isn't any corpus for that intelligent tutor, ANY corpus would be better. So, a corpus for relational databases Q&A created with ChatGPT would be enough (with all the inherited problems doing so, like bias, incorrect answers, etc.). Now the "how to" is more important than the actual accuracy. In other words, the corpus had to be accurate enough.

Try to use GPT-4, it has the knowledge you want to teach, and the prompting would be fairly easy.

@VolkerSiegel actually to make a corpus for "why" "how" and "explain" it would be easy to use ChatGPT. I would gather several questions about relational databases and create a corpus like that. According to my adviser, it would be better than it actually is, because there is no corpus.

I was thinking of using GPT-4 as the chatbot directly. I am quite sure that it already has the knowledge, you do not need data to teach it, only prompts to ask it. But I am sure GPT-4 can just create a corpus. How large should it be? ChatGPT (GPT-3.5) can do that too, but has a shorter context length (4000 vs 8000) The GPT-4 with 32,000 is not yet available. But you can work around that limit by splitting the task.


text:
Problem of extremely varied reward value in DDQN
I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here.
state shape = 5

action space = 0,1,2, ..., 100

episode length = 1669

episodes = 300

Neural-Net shape = (5, 25, 50, 101)

target_model_update=1e-2,

gamma= 0.99

Batch size = 128

learning rate = 1e-3

warmup steps = 1500

Linear Annealed Policy with number of steps=1669*290, minimum epsilon = 0.001, maximum epsilon = 1

Replay Buffer limit = 1669*30

target_model_update = 1e-2

The total reward of each episode wrt the episode number while training can be shown in the following picture
If I change the learning rate from 1e-3 to 1e-4 then I get
Reducing the learning has increased the maximum reward value while training. However, both learning rates still result in high fluctuation in the total reward. It seems like the reward is very sensitive to the actions performed by the DDQN agent. The fluctuation in reward is much bigger than the start of the training.
This is counter-intuitive to me since the exploration rate decreases as the step number increases. Lower exploration should result in a lower variance in the reward value. Even though the average reward does flatten out, in both cases the reward variance is much higher at the end than at the beginning of the training.
The Four values out of the five values of state is not dependent on the previous action made by the agent. So it makes the convergence of the algorithm very hard. I have tried tuning hyper parameters, but could not achieve better than the above two cases.
Kindly help me to understand why there is such high variance in reward as the training steps increase and recommend possible solutions.

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

Hi @Subhajit Saha, and welcome to AI Stack Exchange! Interesting question! If possible, could you please state the specifics of your environment? Thanks, and we look forward to more of your questions in the future!

Without further algo details information, maybe you're just entering the deadly triad zone.


text:
Why ChatGPT output one token at a time?
My understanding of the language model is that the output of the model is a tensor. So the whole output should be computed all together. But why ChatGPT like models can output one token at a time like a stream? And the time difference between the output of first and last token is significant.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

The output of a GPT model is a probability for each possible next token.

See https://ai.stackexchange.com/questions/38923/why-does-chatgpt-not-give-the-answer-text-all-at-once/38929#38929


text:
What degree of randomness is required for a stochastic policy learning task?
I am working on an reinforcement learning training task:
My policy network is LSTM+Gumbel-Softmax, so the sampled policy (i.e. a list of discrete categorical intergers) can be differentiable.
The obtained policy will interract with the environment to generate a final overall accuracy (The goal here is to improve some classification performance).
However, due to the randomness of Gumbel, the training/validation curves look super noisy and it always fluctuates. However, if I deliberately fixed random seed to 0 for each input sample, the curves are stable and achieved a decent validation accuracy.
My objective is for enigneering, instead of publishing or claiming a better method than others. But I understand the ideal scenario is that, I don't set a seed at all, and learning just gave me a nice model which works for any seeds.
My experiment is like this:  During training, I reset random seed to 0 before each input sample enters the network.  During testing, I also reset random seed to 0 before each test sample enters the network. And the final accuracy is high for test samples (However, if I change to a different seed, the accuracy can be really low).
Could someone provide some comments and feedback whether my experiment is correct? Can I use the above trained model with seed (0) for real world prediction tasks?  My feeling is that, if I zoom out to view this as a normal deep-learning task (training data and testing data are separated and test accuracy looks decent), I don't see any problems. But it is kind of scary that my model has to work with random seed fixed to 0.


text:
How to perform graph embedding on a DAG graph with point weights and restore the resulting embedding vector to the DAG with point weights?
How to perform graph embedding on a directed acyclic graph with point weights and restore the resulting embedding vector to the directed acyclic graph with point weights?


text:
Is there an advantage of using GPT model compared to BERT if you have a specific domain?
ChatGPT really shines when the task/question is very broad and is general. Using it and then limiting it's knowledge to a specific domain (via prompts as far as I know) would be kind of a waste of GPT.
Is there any advantage of choosing GPT instead? maybe the learning process can be easier and faster than other NLP like BERT?

ChatGPT is a GPT model, so it's unclear to me if by GPT you're referring to the original GPT model or not.


text:
Has anyone tried decision transformers with PPO?
I am trying to implement a PPO agent to try and solve (or at least get a good solution) for eternity 2 a tile matching game where each tile has 4 colored size you have to minimize the number of conflict between adjacent edges.
I thought that using a decision transformer would be a good way to go, since transformers should be good at 'understanding' the relationship between the pieces. I am trying to get the Decision Transformer to predict what next tile should be played, but i am getting nowhere and my policy does not seem to learn anything.
The value loss converges but the policy loss stays around a constant mean. Are decision transformers not very compatible with PPO or am i missing something?
If you want to take a look here is the code.
https://github.com/mt-clemente/Projet_INF6201/tree/transformer/Network


text:
Is there a deep learning model when train data and test data (or real data) have some common and uncommon labels?
I am building DL model that predict test data (or real) that has some different labels compared to training dataset.
My semi-supervised model is kind of terrible to predict test data (77~85% ish), and I want to improve the test score.
Is there is any DL paper about this kind of work? The most similar area I can think is Domain-adaptation area. If it's just domain adaptation genre, then what papers can you recommend to me?
And
I'm trying to make train dataset with multiple datasets that have common labels but also have uncommon labels. (these datasets from same domain by the way) these datasets have some common features but also have other features too. So the integration of these data is kind of unusual thing to me, is there good tips or guide for this kind of data integration?


text:
What can convolution layer without batch normalization and activation function contribute to models?
I recently saw some paper about stereo matching :
End-to-End Learning of Geometry and Context for Deep Stereo Regression [1]
https://openaccess.thecvf.com/content_ICCV_2017/papers/Kendall_End-To-End_Learning_of_ICCV_2017_paper.pdf
End-to-End Learning for Omnidirectional
Stereo Matching with Uncertainty Prior [2](which isn't free to see this paper. But the DL model is almost as same as Gc-net paper, except the model is for wide angle multi camera )
These papers are about stereo matching in computer vision area, The models in the papers are composed of Unary feature extraction + cost volume (As simple you can think it as concatenating 2 or more data's feature) + 3D convolution.
What I don't understand fully is in the Unary feature extraction part, the models have some 2D convolution layers with residual connection, and then after this there's a 2D convolution layer with no activation and batch normalization before cost volume part.
The gc-net [1]'s unary feature extraction works like
(1) 5x5 conv, 32 features, stride 2-> (2) 3x3 conv, 32 features -> (3) 3x3 conv, 32 features
(4) (1)-(3) residual connection
(5) (2), (3) repeat a few times with residual connection
(6) 3x3 conv, 32 features (no ReLu or BN)

What does this (6) convolution layer without activation and batch normalization contribute to the model?
In the 'End-to-End Learning for Omnidirectional Stereo Matching with Uncertainty Prior'[2] paper says that

the last layer without the ReLU in the feature extraction part allows
the network to distinguish between negative features and invisible
areas, where the network is set to zero by the ReLU and warping
operations, respectively.

But I can't understand how the layer without activation function and normalization contribute above things.
And in 3D convolution part of Gc-net [1] also has no-ReLu, or BN too. The paper says

The network can also pre-scale the matching costs to control the
peakiness (sometimes called temperature) of the normalized
post-softmax probabilities (Figure 2). We explicitly omit
batch normalization from the final convolution layer in the
unary tower to allow the network to learn this from the data.

Which means 3D conv layer without BN helps 'softargmin' (or softargmax) function not to have multi-modal distribution. This thing I don't understand how the conv layer without activation and and normalization has such ability.


text:
Final Model Training Problem - Overfitting
I am working on a CNN project for multiclass classification. I implemented hyperparameter optimization to find the most suitable model, during which I got a best accuracy of 97.38%. I then took this model and applied Early Stopping to it, but the results were much worse. To start, the accuracy dropped greatly. Second, the model seems to be overfitting the data, as shown in the learning curve below. Test loss stops dropping after the shown point, and the training loss goes down to almost zero. I have tried implementing some techniques like Dropout, weight decay, and using different batch sizes, but the issue still persists: the train loss goes down rapidly and converges quickly while the test loss saturates at a specific point and does not yield good prediction results. What are some other steps which I can implement to solve this issue. Can I fix the model during Early Stopping straight away, or do I have to rerun the hyperparameter optimization from the beginning?


text:
Image Alignment/Homography of Subject in Bounding Box
I've been trying to design an algorithm for aligning an object across two photos in realtime. I am able to localize the object (create an ROI/BBox) through an object detection (siamese) network for both photos, leaving me with a cropped image with the object in the center for both frames.
I am struggling to do the second step of my proposed algorithm, which is "finetuning" the alignment of one of the objects in the bounding box by calculating a homography with respect to the "template" crop. Ideally, I would like to be able to overlay the transformed second image on the first image and have the object roughly aligned in the image.
I have been trying to use cv2.findHomography using ORB descriptors and keypoints, however I find that the ORB extractor or the feature matcher ends up being inaccurate, leading to terrible transformation estimates when the object becomes non-stationary. ORB especially seems hard to tune, with the parameter that picks up enough keypoints being fastThreshold=5 which seems abnormally low to me. I use ORB features since I am aiming to use a "free" algorithm, unlike SURF or SIFT.
For context, here is an example of the cropped images I'm trying to align after the object is rotated w.r.t. the bounding box. The subject is a keycap I had laying around. The original image isn't that blurry -- I applied cv2.blur(img, (15,15)) before attempting to detect feature and match them.
.
Aligning these images seems like it should be a simple task, but I'm increasingly worried that classical CV is insufficient for alignment. My background is not in classical CV and so I'm not sure if there is some trick/algorithm that can efficiently align these objects. Alternatively, suggestions for heuristics to determine robust parameters to these OpenCV methods or engineer my data to be more suitable would be appreciated.
My background is biased towards neural networks and I'm worried that I need to turn to neural network-based approaches for image alignment or replace my object detector with a more end-to-end estimate of the orientation of the bounding box. Either way, the aim would be to perform this alignment in realtime on consumer hardware.
Thanks!


text:
Does OpenAI ChatGPT has any "dialogue state tracking" besides max tokens to include previous conversation history?
When I chat with chatGPT earlier, chatGPT itself mentioned "dialogue state tracking" technique that is utilized in addition to chat completion with previous conversation history(up to max allowed tokens).
Is this just a specific technique or this is just some general term. As a specific technique, can we outline how it is done?
The background this question is that it seems chatGPT can take a very long initial "instruction", e.g. like a tutor prompt with detailed behavior protocol.
And even after a few rounds questions, it still behave as instructed pretty completely and not wondering off back to its normal style.
Really want to know how it can achieve this and is the open ai API side of chatCompletion
can do the same or not.

ChatGPT probably doesn't know how ChatGPT works, and it's known to hallucinate.


text:
Convert unstructured text to XML
I am working with a large dataset (example bellow) of unstructured and structured documents that contain a variety of text elements, including paragraphs, tables, and lists. Currently, the process of generating the equivalent XML markup for each text element is done manually. I am looking to automate this process by using machine learning.
I am considering two possible approaches:
The first approach would be to treat this as a translation problem and use a transformer model to generate the XML markup automatically. This would involve training the model to translate the text elements into their corresponding XML markup.
The second approach would be to treat this as a Named Entity Recognition (NER) problem in natural language processing (NLP). In this approach, the model would be trained to recognize the different text elements and assign them the appropriate XML markup.
Or there are better approaches ?
Example of text :

Har et radio- og TV-selskab ret til at forbyde udsendelse og til at opkræve licens til viderespredning af sine gratis programmer, som en hotelvirksomhed modtager gennem en parabolantenne og viderefører pr. koaksialkabel til værelserne til gavn for gæsterne? Udgør denne form for viderespredning en »viderespredning pr. kabel« som omhandlet i artikel 1, stk. 3, i direktiv 93/83/EØF (2), og kan den bevirke, at radio- og TV-selskabet tildeles specifikke rettigheder i tilfælde af en national lovgivning, som tilsyneladende udvider kataloget af de i EU-lovgivningen fastsatte.

The corresponding XML :

<col width="5%"/>
<col width="95%"/>

1.

Har et radio- og TV-selskab ret til at forbyde udsendelse og til at opkræve licens til viderespredning af sine gratis programmer, som en hotelvirksomhed modtager gennem en parabolantenne og viderefører pr. koaksialkabel til værelserne til gavn for gæsterne? Udgør denne form for viderespredning en »viderespredning pr. kabel« som omhandlet i artikel 1, stk. 3, i direktiv 93/83/EØF (
2
), og kan den bevirke, at radio- og TV-selskabet tildeles specifikke rettigheder i tilfælde af en national lovgivning, som tilsyneladende udvider kataloget af de i EU-lovgivningen fastsatte rettigheder?


text:
How can one use reinforcement learning for optimal parameter determination to minimize output in dynamical systems?
Assume that we have a discrete dynamical system of the form
$x(k+1)=f(x(k), u(k), p)$
$y(k)=g(x(k))$
where $u$ is the input, $y$ output, $x$ internal state of the system and $p$ is an unknown fixed parameter.
How can we use a reinforcement learning approach to find optimal
value of $p$ so that $y$ is minimized on average for a given set of inputs?

Why do you need RL to estimate this? It doesn’t seem like a sequential decision making process

I know but I want to solve it as an RL problem. I am sure that there are multiple ways to model it as an RL problem.

My advice would be to avoid overcomplicating things. RL is very very data intensive, you could solve this with supervised learning in a much more efficient manner.

I changed the problem statement to better express my objective. I think RL is suited better.


text:
Why the positive example is included in the denominator of NT-Xent loss?
I have a little perplexity about the NT-Xent loss employed in self-supervised contrastive learning.

What we are essentially doing is maximizing the similarity of pairs of augmented images while minimzing the similarity with all the other instances in a single batch. if $j$ is an index corresponding to positive example for the index $i$, then I don't understand why it gets included in the denominator. Shouldn't we consider the indicator function $\textbf{1}_{\{k \neq i\} \cap \{k \neq j\}}$ instead?


text:
Compare True Contribution with SHAP Contribution, using simulated data
I try to compare the true contribution with SHAP Contribution, using simulated data. Because the data is simulated, I have the ground truth contribution of each variable for each observation, which I expected SHAP to give similar contribution. But results are quite different, and I am trying to understand why. Please see details below:

Simulate X: generate 4 random variables (x1,x2,x3 and x4).See code section A.
Similate Y: Let log_odds = x1+x2+x3+x4 (setting all coefficient to 1 and intercept to 0). Then convert log_odds to probability, and use probability to generate binary outcome. Since coefficients are 1 and intercept is 0, the x value is the contribution to the log_odds of the observation. See code section B.
Fit logistic regression. The estimated coefficients are very close to ones used for simulation. The AUC is 0.92. coef: [0.98761674 1.00301607 1.01310563 0.98102926] Intercept [0.03131222]. See code section C.
Get KernelSHAP, and put it side by side with the x table ( the true contribution). See code section D.

The intercept in data generation is 0, and the bias term for SHAP (explainer.expected_value1) is -0.003, which is also close to 0. Thus, I guess I can compare x value with SHAP value for each variable. Please see results for 10 observations below:

The x value and SHAP value are not quite comparable;
For each observation, the contribution rank order within 4 variables (x1,x2,x3,x,4) is not consistent with the rank order in the SHAP value.
In data generation, x1 and x2 are all positive numbers, while x3 and x4 are all negative numbers. Thus in SHAP columns, I would expect to see some pattern, however, the pattern in SHAP columns are quite random.

So my questions are:

Is there any flaw in my testing design, such that the comparison between x value and SHAP value is totally irrelevant?
Is this the SHAP results I should expect?
Why SHAP gives contribution that deviate so much for the actual value and rank order?

Thank you very much in advance!
import random
import numpy as np
import pandas as pd
import sklearn
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder, MinMaxScaler
from sklearn.metrics import classification_report
import matplotlib.pyplot as plt
import shap
from numpy.random import seed
from numpy.random import rand
from numpy.random import randint

##Section A: Simulate X
random_n=2
random.seed(random_n)
seed(random_n)
n=10000
x1=np.array(rand(n))*5 # The larger the *n, the further log_odds is pushed away from 0, the easier data to model.
x2=np.array(rand(n))*5
x3=-np.array(rand(n))*5
x4=-np.array(rand(n))*5
df=pd.DataFrame({"x1":x1,"x2":x2,"x3":x3,"x4":x4})

## Section B: Simulate Y
def logistic(z):
    return 1 / (1 + np.exp(-z))
lp=x1+x2+x3+x4
prob = logistic(lp)
y = np.random.binomial(1, prob.flatten())
print("mean of y: ", np.mean(y))
print("log_odds of mean: ",np.log(np.mean(y)/(1-(np.mean(y)))))

## Section C: Fit logistic regression and make sure AUC is reasonable
from sklearn.linear_model import LogisticRegression
LR = LogisticRegression(penalty='none')
LR.fit(df.values,y)
print("coef:      ",LR.coef_, LR.intercept_)
print("AUC:       ",roc_auc_score(y, LR.predict_proba(df)[:,1]))

## Section D: Get KernelSHAP, and compare with tree contribution (for k observations)
k=10
explainer = shap.KernelExplainer(LR.predict_proba, df, feature_perturbation="interventional", nsamples=10000, link="logit")
shap_values = explainer.shap_values(df[0:k])

df_shap=pd.DataFrame(shap_values[1],columns=("SHAP_x1","SHAP_x2","SHAP_x3","SHAP_x4"))
df_shap_L=pd.concat([df.iloc[0:k,:], df_shap], axis=1)
df_shap_L['log_odd_true']=np.log(prob/(1-prob))[0:k]
prob_pred =LR.predict_proba(df)[:,1][0:k]
df_shap_L['log_odd_pred']=np.log(prob_pred/(1-prob_pred))[0:k]


text:
Comparing Quality of Raw Data from Various Hardware Setups
My background is not in ML so please correct any gaps in my question.
I am working on an application trying to discriminate across multiple sounds (eg. bark, glass break, keys jingle etc.). There are a very large number of hardware setups possible (number of microphones, location, type of microphone etc.). I am trying to determine the best hardware setup without running a full machine learning classification each time (very resource intensive and want to remove variability from ML training process). If I have a database of various N setups what algorithm could I use to determine which setups carry the most information (and thus good candidates for full ML training). I have been considering various feature extraction approaches (ie mean SnR, total system signal power etc.) but it's not obvious if that is the best way or how to compare datasets with different channel counts. Can anyone provide advice on best ways to parse and sort raw data to determine the best candidate data sets without going through a full ML design/training?


text:
Does GPT not need to compress its training data?
In his recent short pamphlet on GPT, Stephan Wolfram says

... the 'size of the network' that seems to work well is ... comparable to the 'size of the training data'. ...
in this representation it seems there’s in the end rather little 'compression' of the training data;
it seems on average to basically take only a bit less than one neural net weight to carry the 'information content' of a word of training data.

But this isn't true is it, even by his own account?
He says that the size of the network is 175 billion parameters, and the size of the training data is at least "a trillion words of text"
(possibly 100 times that). That's a lot less than "one neural net weight to carry the 'information content' of a word".
Does GPT not need to compress its training data? Is it true that the number of parameters (Wolfram omits biases) is roughly the same as the size of the training data in words?

I thought that compression was the whole reason networks wore forced to generalize at all, broadly speaking?


text:
What could be the reason for getting low contrast in GAN generated images?
I am trying to use DCGAN for the waterbirds dataset with the following hyperparameters, but I am getting low-contrast images. It would be great if anyone has any suggestions as to what could be some usual reasons for such an outcome.
LEARNING_RATE = 2e-4
BATCH_SIZE = 64
IMAGE_SIZE = 64
CHANNELS_IMG = 3
NOISE_DIM = 150
NUM_EPOCHS = 150

This could be anything from too few epochs, wrong noise_dim, to that model just plain doesn't get better than this. Is there any indication it should perform better? did you follow parameter settings from somewhere? this question needs more information.


text:
How to build text to video and create AI avatar speaking the text?
There are so many text-to-video tools available online like synthesia ai, elai io. I like to build project like this. Can anybody tell how does these built and how it works?
I know the basics and know ML classification and regression models, but need some knowledge on this. Any help is appreciated.
Thanks in advance :)


text:
Longer DNN training times when using evolutionary algorithms
I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (GA), PSO, etc.). My training dataset is of size >=100,000.
Observation: For GA-trained DNN, I see that it obtains better accuracy than Adam-trained DNN but takes more iterations (hence longer time) than Adam. But only to perform slightly better, GA took 3000 iterations with a population size of 100 whereas Adam took only 100 epochs! I felt the performance gain does not sit right given the longer training time. But before I conclude I want to know if understanding and implementation are correct as given below.
Implementation and problem: While implementing Adam with 100 epochs I use a batch size of 32. Whereas I don't how to use the concept of batch size with GA training but rather the usual initialize population and run some iterations style. Hence for each candidate of the population, the entire 100,000 samples are traversed through before loss (objective function) is computed. Then the same for the second candidate until 100 candidates are done. And this is just for 1 iteration! I am sure this is what increases the training time. But is there any other way to implement this kind of population-based optimizer here?
I feel if I find some time-reducing way I can more easily run these different population-based optimizers and experiments since I see a potential to obtain better predictions with these.

Have you tried to make your population-based optimizer be parallel? I guess you can evaluate multiple candidates in the population at the same time, simulating a batch size > 1 (even if the purpose is radically different.)

I have access to a max of 40 cores and so, I use the multiprocessing command on the objective function that computes the loss value for the 100,000 set. As you correctly thought, this did increase my speed compared to my initial code. But still the situation is as I write in my question. I have seen papers where they mention how easily they used population algorithms with comparable overhead and better accuracy but they didn't provide any code. So I feel there is something wrong with my approach to implementing even though it's technically correct.

I see, maybe you need a more efficient way to improve candidates. Is feasible for you to do something like done in PBT? See the explore and exploit strategies, for example. About the code I have found this, have a look.

@LucaAnzalone Thank you for the links. I read this paper carefully after you suggestion. I see that they propose PBT to jointly optimize the model parameters and hyperparameters. So given a population size, they initialize N number of models with diff hyperparameters and the 'exploit' and 'explore' help to how to use the info from each of the population set of N models. But they also update each compute obj fn. of the each population member in each of the N models. Of course, they do it parallely.

@LucaAnzalone  However, one idea that I get from the paper is maybe I should use GPU to compute the obj. fn value at each member of the population? I get this idea since the paper mentions for one case study  'where each member of the population is a single GPU optimising a Transformer network for 400 × 10^3 steps' and they used 32 population size. Of course I do not have access to GPU currently and the max I can get later is 4 GPUs whereas my population size will be atleast of size 50*2800 i.e. 50 rows and 2800 DNN weights and biases.

Training on GPU is usually a good idea, I've also seen another paper (the one related to the repo) that uses Jax to increase the speeup even more. Anyway, what if you initialize the population with Adam to reach good parameters first and then you continue with GA? This may speed-up convergence and reduce total training time.


text:
Model starts off stronger than SOTA, but doesn't maintain advantage
In general, what could cause a Neural Network to start with better results than a similar model, but then lose it's advantage after a few epochs ?
If a NN starts off about 10% better than a similar one, what could cause them to converge at the same accuracy ?
Thanks


text:
Do LLM or Machine Learning models with a large number of classes employ standard cross entropy?
In language modeling, the next token is predicted from the entire vocabulary during inferenc. However, when training models with large vocabularies or many different classes, standard cross entropy can be suboptimal due to its computational expense and sparsity of optimization. Are there any tricks or relevant papers that people use to address this issue?


text:
How can one incorporate spatial correlations into time series forecasting?
I am working on a project, where I am trying to predict temperatures of various streets and I have their locations recorded.
I was wondering if I could somehow train a model that could incorporate their relative positions and somehow account for it during the temperature predictions.
I plan on using a temporal fusion transformer as there will be several variables and I will want to predict multi-step horizons.
Does anyone have an idea how this could be achieved? I was thinking maybe graph neural networks, but I'm not sure where exactly they would be incorporated.


text:
Why Hadamard product appear in General backpropagation derivation?
I'm trying to understand the basic backpropagation of a neural network and stuck with this formula, which describes how you calculate the error vector for a layer $i$:
$$\delta z^{[i]} = (w^{[i + 1]})^{T} \delta z^{[i + 1]} \odot g'^{[i]}(z^{[i]})$$
Where matrix $W^{[i]}$ contains all weights of layer $i$ and $g'^{[i]}(z^{[i]})$ is the derivative of layer $l$ 's activation function at $z^{[i]}$. We call this equation (1)
This formula comes from this "unrigorous" derivation: Suppose we consider neural network with 3 layers and so, goes along with 3 parameters $(w^{[3]}, b^{[3]})$, $(w^{[2]}, b^{[2]})$, $(w^{[1]}, b^{[1]})$. Then, we'd like to calculate gradient for each layer to improve the cost.  In other words:
With $i = 3$: $\frac{\delta L}{\delta w^{[3]}}$ is the corresponding gradient.
$$\frac{\delta L}{\delta w^{[3]}} = \frac{\delta L}{\delta a^{[3]}} \frac{\delta a^{[3]}}{\delta z^{[3]}} \frac{\delta z^[3]}{\delta w^{[3]}} = \frac{\delta L}{\delta a^{[3]}} \frac{\delta a^{[3]}}{\delta z^{[3]}} a^{[2]}$$.
With $i = 2$:
$$\frac{\delta L}{\delta a^{[3]}} \frac{\delta a^{[3]}}{\delta z^{[3]}} \frac{\delta z^{[3]}}{\delta a^{[2]}} \frac{\delta a^{[2]}}{\delta z^{[2]}} a^{[1]} = \delta z^{[3]} w^{[3]} g'^{[2]}(z^{[2]})a^{[1]} = \delta z^{[2]} a^{[1]}$$, where we've just defined:
$$\delta z^{[2]} = \delta z^{[3]} w^{[3]} g'^{[2]}(z^{[2]})$$. And hence, By "rearranging things", we have the formula (1) mentioned above. Where does Hadamard product come into play? Isn't that just simple matrix operation?
My question: What is the intuition behind the formula (1)?
My math background is ill :D. It's just some basic linear algebra and calculus from university. Highly appreciate any help. Thanks :D


text:
Backpropagation of position-wise feedforward neural network
I have read a paper entitled "Attention is all you need" by Vaswani et al. (2017). This paper use the so-called position-wise feedforward neural network, where the input of this network is a matrix $\mathbf{X} \in \mathbb{R}^{n \times d_\mathrm{model}}$ (not a vector $\mathbf{X} \in \mathbb{R}^{d_\mathrm{model}}$). If I am not mistaken, the meaning of position-wise is that the (same) feed-forward layer applies to every vector $\mathbf{X}_{i*}$ ($i$th row of $\mathbf{X}$) for $i = 1, \dots, n$. Thus, the weights are shared.
I want to do backpropagation for a position-wise network consisting only a linear layer with no activation. Let the output dimensionality is $d_\mathrm{model}$. Applying this network yields $\mathbf{Z} \in \mathbb{R}^{n \times d_\mathrm{model}}$ where each row $\mathbf{Z}_{i*},\ i=1, \dots, n$ is given by $\mathbf{Z}_{i*} = \mathbf{X}_{i*} \mathbf{W} + \mathbf{b}^\intercal$. Here, $\mathbf{W} \in \mathbb{R}^{d_\mathrm{model} \times d_\mathrm{model}}$ and $\mathbf{b} \in \mathbb{R}^{d_\mathrm{model}}$ are weight and bias, respectively.
Let $L$ be the loss function. For $i$th row I get: $\dfrac{\partial L}{\partial \mathbf{W}_{pq}} = \dfrac{\partial L}{\partial \mathbf{Z}_{i1}} \cdot \dfrac{\partial \mathbf{Z}_{i1}}{\partial \mathbf{W}_{pq}} + \dfrac{\partial L}{\partial \mathbf{Z}_{i2}} \dfrac{\partial \mathbf{Z}_{i2}}{\partial \mathbf{W}_{pq}} + \dots + \dfrac{\partial L}{\partial \mathbf{Z}_{id_\mathrm{model}}}  \dfrac{\partial \mathbf{Z}_{id_\mathrm{model}}}{\partial \mathbf{W}_{pq}} = \dfrac{\partial L}{\partial \mathbf{Z}_{ip}} \mathbf{X}_{iq}$, for $p, q = 1, \dots, d_\mathrm{model}$.
Thus, I end up with $\dfrac{\partial L}{\partial \mathbf{W}} = \left(\dfrac{\partial L}{\partial \mathbf{Z}_{i*}}\right)^\intercal \mathbf{X}_{i*}$.
My question: is
$\left(\dfrac{\partial L}{\partial \mathbf{Z}_{1*}}\right)^\intercal \mathbf{X}_{1*} = \left(\dfrac{\partial L}{\partial \mathbf{Z}_{2*}}\right)^\intercal \mathbf{X}_{2*} = \dots = \left(\dfrac{\partial L}{\partial \mathbf{Z}_{d_\mathrm{model}*}}\right)^\intercal \mathbf{X}_{d_\mathrm{model}*}$ holds?


text:
Behaviour of PPO/similar Algos under action penalties
I am currently experimenting with PPO in different environments. I am interested in learning policies that fulfill a certain goal while keeping a specific value low. Here's an example:

Using PPO on a cartpole environment to learn an upswing of the pole,
but simultaneously keeping the angular velocity of the pole low. The
standard approach is to include a penalty on the pole velocity in the
reward function.

However, I observed that penalizing the velocity from the beginning reduces sample efficiency significantly and hinders learning good policies. For this reason, I tried using only a small penalty on the pole velocity until PPO converges to a decent policy and then apply a refinement step in which I penalize the torque much more to get good performance and low velocities. This seems to work better. I observed similar behavior on other environments in a similar setting.
I want to find a (formal) reason for this behavior (why does penalizing velocities from the beginning hinders learning). Does anybody have some literature tips on stochastic optimization/rl that could be useful? Or some resources on the topology of high-dimensional spaces? Or even an idea for an explanation for this behavior?
Thanks in advance for any tips!!

Welcome to AI stack exchange! Please put your question in the title of your post. Try to make it as clear as possible from the title what kind of answer you would like to receive.


text:
What's the right approach to "teach" a chatbot about a big specific data set: training it or feeding it the input once?
I would like to build a chatbot that I can talk to about any specific data set I provide and I have a theoretical question.
Let's say, I want to discuss French cuisine. This could involve providing the model with information like culinary history, biographies of French chefs, French cookbooks, etc.
Now, the question: Assuming I have all this huge data set, what would be the correct, more efficient approach?

Do I use the data set to train the model, so that later I can talk to it?, or
Do I simply feed all this text-based information to the model as a system message and instruct the model to analyze the text before it gives any answers?

So, essentially, training or providing large, one-time input?


text:
How to inspect a dimension in SBERT embeddings
I am using SBERT transformers to vectorize text. I want to use the sentence embeddings in a linear regression to predict a value or a classification. Some dimensions are highly colinear and result in a matrix close to non-invertible: 34 and 688 have 97% correlation, and each has 81% correlation to 756. For now, and for safety, I exclude those dimensions from the matrix, but I would like to understand the problem and the data, and possibly ferret out bugs.
How can I know what those dimensions relate to?


text:
Resources to understand the math behind diffusion models
I started learning about diffusion models, but I couldn't follow the math proofs (probability distribution terms and other stuff), is there any sources you suggest to understand what they're talking about?
my background on these probability terms is quite limited, I learned only the introduction stuff, so only general definition.
If you could suggest me some resources that takes into account the deep learning side and also other deep generative models that would be a plus.


text:
Computing loss only on missing pixels for image inpainting
I'm trying to understand the image inpainting landscape, and I'm seeing 2 (of possibly more) strategies that I'd like to explore further.

Deep Image Prior (DIP) based methods, which don't train a model, but try to overfit a randomly initialized model on the known set of pixels in the image and rely on the fact that the trained weights resulting from those known pixels will somehow be able to provide a good approximation for the unknown pixels. In the case of the DIP model, I see that the loss is computed only on the set of pixels that is not missing. https://github.com/DmitryUlyanov/deep-image-prior/blob/master/inpainting.ipynb total_loss = mse(out * mask_var, img_var * mask_var)

Deep Learning (DL) based methods which train (for example) a CNN model on a lot of images with parts of the image removed, and train a model that can predict the missing pixels. I'd like to know if the current state of the art uses a loss function on all on the predicted pixels (both known and missing) or if the loss function is computed only on the set of missing pixels (effectively making the loss function look at the opposite set of pixels that the loss for DIP methods look at).

Is there any prior research done in this direction and are there any results available?


text:
How do I make an autoencoder and make it work on extracting the feature of a stationary wave?
I have a project to complete in a day, and I know that doing it in a day is a bit far-fetched. The problem is this -
"Design an autoencoder with two neurons as the constriction, multiple hidden layers, and a vector from input neurons.
Make it work on extracting features of a stationary wave.
Then deconvolve what the constriction says.
Can you see wavevector(or frequency) and phase showing up in any way?
If you succeed in this past talk in getting somewhere meaningful, now make the input a wave (so you are now autoencoding a movie instead of stationary picture), and figure out what you should do with the autoencoder to derive some physical meaning."
Can anyone tell me what to do? What kind of things to learn and avoid? (I know basic Linear algebra, multivariable calculus and very basic probability theory of knowing what an expectation value is and one or two distributions, I know some C, not Python)

Hi Neeladri! If this is by any chance a homework question, make sure to use the homework tag :)

@RobinvanHoorn Did it!


text:
Contradiction in a single sentence - Is this an artifact of an external safety mechanisms?
My system prompt contained "Never apologize."
An answer started with

Entschuldigung, ich habe Ihre Anweisung, sich nicht zu entschuldigen, übersehen.

which is quite well translated from German with

Apologies, I missed your instruction not to apologize.

This sentence apologizes for something that it does in the same sentence. The sentence is logically inconsistent.
I do not remember GPT-4 giving me a sentence that pathological ever before.
I would have assumed that the changes in GPT-4 that made it more save and more polite were learned. But from intuition, that seems like there are two separate components, one is stupid but polite, and the other part does the intelligent things, and the polite part has some kind of access to the interaction.
In the example, something apologized, and did not intend to apologize in the same sentence. A contradiction. But in a very deliberate and systematic way, it is not a contradiction arising from some other problem.
How GPT-4 works is not published, so we can not get an exact answer - but are we talking to two separate systems, the main transformer, and something separate that makes the output polite, as opposed to making the transformer learn to be polite?


text:
Diffusion Models - Loss normalization for different noise schedules
With reference to the paper "Denoising Diffusion Probabilistic Models", and running the code given in this notebook showing a simple example on "flattened" data, I have the following issue --- the normalization of the loss is strongly dependent on the choice of the noise schedule.
That is, by changing the noise schedule (while keeping all the rest of the model intact) it may happen to obtain a lower loss although, in practice, the target distribution is modelled more poorly (visually obvious, but a simple KL calculation confirms it).
I can't wrap my head around this.  The loss used in the notebook is the "simple loss" from the paper (their Equation 14):
$$L_{simple}(\theta) := E_{t,x_0,\epsilon}\big[|| \pmb{\epsilon} - \pmb{\epsilon}_\theta (\sqrt{\bar{\alpha_t}}\pmb{x}_0 + \sqrt{1-\bar{\alpha}_t}\pmb{\epsilon}, t))||^2 \big]$$
from which I deduce $\pmb{\epsilon} - \pmb{\epsilon}_\theta$ should always converge to the same range because $\pmb{\epsilon} \sim N(0,\pmb{I})$, but in practice it is not like that.
In turn, the problem becomes that when I run models with different schedules, I cannot compare the results based on the loss, that is, I cannot automate the choice of the best model/schedule combination.
Assuming I didn't make any silly mistake, my question is twofold:
1 - How can I mathematically, intuitively see where the different schedule choice affects the loss?
2- How can I "normalize" the loss, so that it becomes comparable across different schedules and models?

PS: I also tried to replace the "simple loss" with the full loss of Equation 12 (where I assumed that the additional terms would have taken care of the normalization related to the noise schedule):
$$L(\theta) := E_{t,x_0,\epsilon}\big[|| { \beta_t^2 \over 2\sigma_t^2 \alpha_t } \pmb{\epsilon} - \pmb{\epsilon}_\theta (\sqrt{\bar{\alpha_t}}\pmb{x}_0 + \sqrt{1-\bar{\alpha}_t (1 - \bar{\alpha_t})}\pmb{\epsilon}, t))||^2 \big]$$
... but the resulting loss across the epochs is too noisy to understand whether it really sets the same dynamic range independently of the noise schedule.


text:
Can AI be used for submarine detection?
Recently a lot of work has been done in connection to using AI in the search for extraterestrial life, the SETI project.
Could the same methodology be used by the military, in order to detect submarines hiding  in the world's oceans ? Basically the training data sets will have different characteristics,  and the definition of what "relevant signals " are will change, but in principle the fundamental problem is the same, finding relevant signals in an "ocean of noise ".  Is this assessment correct ? Is this avenue of research currently being considered already by the military ? You only need to answer the second question if the answer to the first is yes.

that would be classified information

All  the relevant methodology,  data sets, and even source code are open access  @user253751 https://seti.berkeley.edu/ml_gbt/overview.html   , and the translation to a military application is obvious,  to say the least. Maybe it should be classified information,  but at this time I doubt it is. Should I conclude that the answer to my first question is yes ?


text:
Is it possible to get recurrence through alternating forward and backward propagation?
This is a purely theoretical question, and I'm not trying to solve some real world problem with this, and I don't care about how ineffective this may be, as long as it is theoretically possible.
For a simple NN trained with SGD, is it possible to find an initial configuration of weights, that won't converge (suck in local minima), which "abuses" back propagation to store state between each forward and backward pass pairs?
Say I have a NN with few fully connected layers and two input nodes and one output node.
This NN is now trained on "predicting" the first input using basic SGD with a fixed learning rate.
This is obviously very simple, but only in the example to make sure the NN can know the target trained for with sgd.
Is there an initial weight configuration that results in the network always outputting the second input used in the previous forward/backward pass?
Essentially:
nn: 2 x ... x 1 initialized with a certain state
input: float[2]
loop {
    last = input[1]
    input = randomize with -1 or 1
    target = input[0]
    nn.forward(input)
    nn.backward(nn.loss(target))
    assert nn.output[0] < 0 == last < 0
}

My intuition tells me, that it should be possible to have the weights direct the effect of the specific weights that toggle to the input, when the target output is "known" to it.
Maybe I'm mistaken, and you can actually show that it isn't possible for weights to direct the impact of back propagation in this way.
I've tried brute forcing the weight of smaller NNs, but didn't have success so far, so if it's possible I suspect it to not be very simple.


text:
Confused about interaction terms in polynomial regression
I am trying to code multivariate polynomial regression from scratch and I got confused about how interaction terms work. I saw that a polynomial regression with 2 inputs and with interaction terms should look like this: ax^2 + bz^2 + cxz + dx + ez + f, where a,b,c,d,e,f are the model's parameters and x,z are the inputs. But how would this equation look with 3 inputs? Would there be an interaction term with all three inputs axyz (a is a parameter)?

Do you just want squaring, or with the third term, so you also want terms raised to the third power? // You kind of get to pick what you want. If you just want to tack on a $+y$ term and can justify why, go for it.


text:
Why is it that histogram equalization actually distributs all intensity values uniformly over the whole range?
This article shows the  following transformation function for histogram equalization
$$s = T(r) = (L-1) \int_0^r Pr(w) \,dw$$
Where $L$ is the maximum value a pixel can achieve. $Pr(r)$ is probability density function (pdf) of the image before equalization and $Ps(s)$ is pdf of the image after performing equalization.
It then states that the relation between $Pr(r)$ and $Ps(s)$ is
$$Ps(s) = Pr(r) \bigg|\frac{ds}{dr} \bigg|$$
where
$$\frac{ds}{dr} = \frac{d}{dr}T(r)$$
I can't understand how the pdf of $s$ is the pdf of $r$ times the absolute value of the derivative of $T(r)$ with respect to $r$.
Can you help me understand why this relation holds?


text:
Reproducing Knowledge Distillation on MNIST data
I'm trying to implement Knowledge Distillation, specifically to reproduce the MNIST example given in the paper. My (PyTorch) implementation can be found here.
I would expect it is very self-evident that using this method indeed improves results (e.g., like in other cases I tried: dropout for generalization loss, BatchNorm for speed of training, etc.). But I couldn't get the taught network to be better than the "vanilla" student network - playing with different architectures (with/without dropout, with/without BatchNorm, different number of hidden layers, hidden layer size, etc.), and with different hyperparameters (learning rate, dropout rate, etc.). I also tried using only the soft-targets, as well as combining it with the hard-targets as described in the paper.
My teacher network didn't reach the 99.3% reported in the paper (= 67 errors), but was close (a bit below 99%) so I'm not sure if it's just a matter of training more. But if this is the case - it makes it that the benefits of using this method are extremely small. I did manage once to get better results with the taught network - but it was only in the first epoch, and with much lower accuracies. When trained for longer epochs, the regular student outperformed.
That being said - maybe I'm doing something wrong?


text:
How are the transformer encoder outputs handled?
According to the Attention Is All You Need paper, the transformer's encoder portion is described as

The encoder is composed of a stack of N = 6 identical layers. Each layer has two
sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network.

How are the outputs of the Nx=6 identical layers put together? Is it via concatenation, summation, element-wise product, or are the 6 blocks placed in sequential order, etc? Is the same also done for the decoder?

See this for an extensive explanation on the decoder. The encoder 'stacking' works the same. They are put in sequence.

the output of one feeds into the input of the next


text:
NUMA effects on single-process inference
I've been reading this article, which explains some of the effects of a multi-cpu, hyperthreaded system and suggests ways to get around the issues. Namely: put each problem domain on its own socket; that is, if I'm running a model, then run a copy of the model on each socket.
However, what if my use-case is a LLM/transformer outputting text. Just inference, no training. Due to the sequential nature of the process, the essential effect of applying the "optimizations" there is to run on one socket, and ignore the rest of the machine.
There's only one batch, and only one worker, so the server, on an 8-socket system, would be 88% idle.
I'm interested in what it would take to run on all the numa nodes simultaneously. Assuming no specific optimizations save perhaps avoiding hyperthreaded cores, where would performance end up at on a typical multi-socket system? The reason to do so is that 3-5 year old servers can be had with 10x the memory of new GPU based systems for 1/10th the price, with chips for as little as 2% of their original prices. And I'm wondering if there would be any benefit of a many-socket system over say a 1-socket or 2-socket one, for the throughput of a single, albeit very large, model.
Basically, comparing eight Xeon 8180Ms to say a single epyc rome 7702P. Which would 'win' at this task?
The interesting part is that model weights, during inference, do not change. So, I have this huge chunk of data which is being read by the processors. So if this is cached locally, if that matters, could lead to what should be close to linear performance scaling.
If I add more memory than the model needs; say 192GB per socket for a 120GB model (30B parameters @ fp32/AVX), would pytorch know 'out of the box' that it could cache using the extra memory to keep all accesses of the model weights local? Or is there some numactl command for this?
What happens if the model is too big for the memory of one socket, but still much smaller than the memory in the system?


text:
Are my DeepLabv3 training configurations comparable?
I am currently trying to compare the semantic segmentation performance of DeepLabv3 of a model being trained on data coming from the real world domain with a model trained on data coming from the real world domain on top of which I am augmenting synthesized data. The question that came up to me was whether the training scenarios I have formulated are comparable and/or whether I can draw any meaningful conclusions after evaluating the two models. Below I will describe the two configurations used for training:

In the first scenario I use a subset of 250 data examples from real world (from Cityscapes dataset), using an initial learning rate of 0.001 with a Poly decay function, a batch size of 10 examples, for 100 epochs with crop size of 321.

In the second scenario I use the same subset of 250 data examples from real world and a subset of 250 examples synthetic data examples, using an initial learning rate of 0.001 with a Poly decay function, a batch size of 10 examples, for 100 epochs with crop size of 321. The examples from both domains are sampled randomly in the subsets and after that the subsets are concatenated and the resulting dataset is shuffled, so each batch has randomly sampled examples from each domain.

I attach here the Poly learning rate decay function definition:
$$
f(x) = initialLR * (1.0 -  \frac{x}{totalNoOfIterations})^.9,
$$
where x is the current iteration number (considering an iteration being a forward + backward pass on a batch).
What makes me question the validity of the comparison is:

The difference in the $totalNoOfIterations$ required to complete an epoch (2500 for the first scenario and 5000 for the second scenario, even though the number of epochs is the same, i.e. the datasets are both fully crossed 100 times)
The learning rate evolution, given that it depends on the total $totalNoOfIterations$.


text:
(In stable diffusion) Why are the vae encoded latents sampled by adding randomness to their mean, instead of just taking the mean directly?
In the diffuser's code base they calculate the input img2img latents in the following manner:
init_latents = self.vae.encode(image).latent_dist.sample()
noise = randn_tensor(shape)
init_latents = self.scheduler.add_noise(init_latents, noise, timestep)

With the implementation of latent_dist.sample being:
self.mean + self.std * randn_tensor(self.mean.shape)

I am correct in thinking there is no reason to add noise twice, or am I missing some fundamental reason for this strange behavior?

The first line seems to be adding a 1$\sigma$ standard deviation to the latent vector as a whole. In the third line it seems there is random noise being added to the individual components of the latent space vector?
I'm guessing this could be to make sure input latent as a whole will take on the overall shape or 'structure' of the input image in latent space, but shifted around by 1$\sigma$ (to ensure it's not the same image). Then the random noise is added on top of it to aid in the diffusion process?


text:
Layer Questions regarding Bidirectional VAE (D3VAE)
I am currently trying to figure out how D3VAE are working, but I can't seem to understand the network architecture given. The paper can be found here:
https://openreview.net/pdf?id=rG0jm74xtx

The things I don't understand in the picture can be found in the top right part (BVAE). As far as I understand z_n will be an output of size 2 while the residual-blocks will be of a much greater size. How can they be added, if shapes are mismatched? And they don't explain how "h" (far right in the picture) is calculated? Is it a randomly generated vector?


text:
Clustering bounding boxes to reduce image cutout overlap
Given an image and bounding boxes identified within this image, the objective is to group these bounding boxes in such as way that we can define a bigger bounding box of size NxN that will encompass as little bounding boxes as possible while reducing the overlap between these NxN bounding boxes.
Some background to understand why this is needed. The idea is that we have full Sun images, and we want to use them to train a predictive machine learning model. To do this, we classify each of these bounding boxes into one of two classes. Now, we have available sequences of 512x512 such images and in order to train we need to extract cutouts of these regions and label these with their class. The problem comes from the low resolution of the images, meaning that simply cutting the region itself will yield an image of not high enough resolution. So one possible solution would be to just upscale these cutouts, but that means we distort the size of features, and it may affect the performance of the model. Another possibility is to simply pad the cutouts to the required size. However, I'm trying to go with a fixed cutout size, but that means that ultimately each cutout will have more than one region in it given that the cutout size is ~ 1/4 of the full image. So, what we need is to define these cutouts in such a way that there's as few as possible regions in each and the overlap between them is minimal so that we can label each cutout based on the regions it contains and then use it for training.

This is an example of what one of these images look like. The red bounding boxes are the original region bounding boxes, and the blue bounding boxes are what a cutout centred on each region would look like.
Grouping together regions that overlap, such as 4412, 4399 and 4397 should be easy. However, the problem lies in defining the blue bounding boxes. Manually, I could define three different blue bounding boxes encompassing [4412, 4399, 4397] [4386, 43974414, 4390,4392] and [4379, 4383] but I'm at a lost as to how I could do this automatically. Any suggestions or ideas, even just a name, about what method could work for this are greatly appreciated!


text:
Training DQN Agent slows down and then at around 50 episodes
I am training a DQN Agent at around 50 episodes the fit in the replay function starts slowing down and starts freezing the PC. After a while PyCharm just crashes.
This is the relevant part of the agent
The input is just an array with 12 different floats
        model = Sequential()
        model.add(Dense(64, input_dim=self.state_size, activation='relu'))
        model.add(Dense(64, activation='relu'))
        model.add(Dense(self.action_size, activation='linear'))
        model.compile(loss=keras.losses.Huber(), optimizer=Adam(lr=self.learning_rate))
        return model

    def act(self, state):
        if np.random.random() > self.epsilon:
            actions = self.model.predict(state)
            return np.argmax(actions[0])
        else:
            return np.random.randint(0, self.action_size)

    def remember(self, state, action, reward, next_state, done):
        self.memory.append((state, action, reward, next_state, done))

    def replay(self, batch_size):
        minibatch = random.sample(self.memory, batch_size)
        for state, action, reward, next_state, done in minibatch:
            target = reward
            if not done:
                try:
                    target = (reward + self.gamma *
                              np.amax(self.model.predict(next_state)[0]))
                except:
                    print(next_state, next_state.shape, type(next_state))
            target_f = self.model.predict(state)
            target_f[0][action] = target
            self.model.fit(state, target_f, epochs=1, verbose=0)

Here is my training loop
def train_dqn(agent, env: gym.Env, episodes: int, batch_size: int, episode_length: int):

    ep_rewards = []
    aggr_ep_rewards = {'ep': [], 'avg': [], 'max': [], 'min': []}
    STATS_EVERY = 5

    START_EPSILON_DECAYING = 1
    END_EPSILON_DECAYING = episodes // 2
    epsilon_decay_value = agent.epsilon / (END_EPSILON_DECAYING - START_EPSILON_DECAYING)

    for episode in range(episodes):
        total_reward = 0
        # reset state in the beginning of each game
        state = env.reset()
        state = encode_state(state)
        # Loop over time steps until the episode is done or the time limit is reached
        for i in range(episode_length):
            action = agent.act(state)
            decoded_action = decode_action(action, state, env.action_space)
            next_state, reward, done, _, _ = env.step(decoded_action)
            encoded_next_state = encode_state(next_state)
            agent.remember(state, action, reward, encoded_next_state, done)
            state = encoded_next_state
            total_reward += reward
            if (episode * 29) % 58 == 0:
                if len(agent.memory) > batch_size:
                    agent.replay(batch_size)

        ep_rewards.append(total_reward)

        if not episode % STATS_EVERY:
            average_reward = sum(ep_rewards[-STATS_EVERY:]) / STATS_EVERY
            aggr_ep_rewards['ep'].append(episode)
            aggr_ep_rewards['avg'].append(average_reward)
            aggr_ep_rewards['max'].append(max(ep_rewards[-STATS_EVERY:]))
            aggr_ep_rewards['min'].append(min(ep_rewards[-STATS_EVERY:]))
            print(
                f'Episode: {episode:>5d}, average reward: {average_reward:>4.1f}, current epsilon: {agent.epsilon:>1.2f}')

        if END_EPSILON_DECAYING >= episode >= START_EPSILON_DECAYING:
            if agent.epsilon - epsilon_decay_value >= 0:
                agent.epsilon -= epsilon_decay_value

        print("episode: {}/{}, score: {}, e: {:.2}"
              .format(episode, episodes, total_reward, agent.epsilon))

        if episode % 10 == 0:
            agent.save(f"models/model_{episode}.h5")

    plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['avg'], label="average rewards")
    plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['max'], label="max rewards")
    plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['min'], label="min rewards")
    plt.legend(loc=4)
    plt.show()
    agent.save(f"model_{episodes}.h5")
    return agent

Batch size is 32 and memory is a deque with size 500
I have tried changing batch size.
I have reinstalled CUDA and removed everything related to CUDA and just training on CPU.
I have tried changing the neural net size to smaller and larger.
Changing the frequency of the replay function just makes it crash earlier or later in the training process.
What could cause this? Is this something in my code or is this an issue with my pc or operating system?

Try model.train_on_batch instead of model.fit within the replay fn. Otherwise just profile your code; also I think this question best suits stackoverflow..

I will try that. If it doesn't work I will post it on stackoverflow. Thanks!


text:
Computational overhead of "SCALING FORWARD GRADIENT WITH LOCAL LOSSES"
The paper "SCALING FORWARD GRADIENT WITH LOCAL LOSSES" discusses a new way of training deep neural networks called forward gradient learning. This method is different from the traditional backpropagation method. However, when applied to large networks, it can have high variance. The paper proposes some changes to the method to make it more practical for deep learning tasks. They suggest using perturbations to activations instead of weights to reduce variance. They also introduce a new architecture called LocalMixer that is better suited for local learning. The new approach performs well on standard deep learning tasks (MNIST and CIFAR-10) and outperforms other backpropagation-free algorithms on ImageNet.
Question: As the forward learning approach does not require the backpropagation pass, is it accurate to assume that forward learning (roughly speaking) achieves ~2x speedup in training compared with backpropagation, assuming all other parameters are the same such as the number of epochs, batch size .. etc?
I would get an answer to my question if I could run the code linked in the paper, I tried to run it but it failed giving the following error!
Traceback (most recent call last):
  File "train_mixer_supervised.py", line 2532, in <module>
    app.run(main)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "train_mixer_supervised.py", line 2454, in main
    layer_sizes = get_layer_sizes(
  File "google-research/local_forward_gradient/mixer_lib.py", line 125, in get_layer_sizes
    num_tokens_ = num_tokens_ // (downsample[blk]**2)
IndexError: list index out of range

Note: I contacted the corresponding author of the paper and I still have not gotten any response to my question or the error reported above.


text:
Regression Model overestimates in train-mode
I have a Deep Learning Regression model to predict some values. The results are fine when I use the model in Evaluation Mode, but when I turn Training Mode on the model tends to overestimate the predictions. (I need to use the model in training mode since I want to use Monte Carlo Dropout to estimate a probability/uncertainty for the predicted value.)
My Regression model looks like this (I use PyTorch 2.0)
class Regression(nn.Module):
def __init__(self, num_outs):
    super(Regression, self).__init__()
    self.fc1 = nn.Linear(1024, 512)
    self.fc2 = nn.Linear(512, 256)
    self.fc3 = nn.Linear(256, num_outs * 3)
    self.bn1 = torch.nn.BatchNorm1d(512)
    self.bn2 = torch.nn.BatchNorm1d(256)
    self.dropout = torch.nn.Dropout(0.2)

def forward(self, x):
    x = self.dropout(x)
    x = F.relu(self.bn1(self.fc1(x)))
    x = F.relu(self.bn2(self.fc2(x)))
    x = self.fc3(x)
    return x

If I plot the Training and Evaluation error I get this histogram. As expected the test error is normal distriubuted around 0. However the mean of the train error is moved to the right, which shouldnt be happening. While researching I read this article which describes a similar effect but the other way around, which makes sense to me but my behaviour is really strange since the model should optimize in train mode. I train and test on different datasets, but I get the same effect when I test on my training dataset.
I would be very thankful if someone could point me in the right direction to tackle this problem.


text:
difference in two modified multi-view CNNI
I am working on a dataset where i have two views of a same object( lateral and dorsal). I modified the RESNET50 to take the two views as input( as two different branches of CNN, till the last layer) and concatenate the features from the just before the classification, concatenate them to fC layer and then do the classification. I have two versions of it, and both give more or less same accuracy( >0.9). I am jst not sure what is the difference in these two architecture.
One is
MULTI-VIEW CONVOLUTIONAL NEURAL NETWORK (MVCNN) ARCHITECTURE 1
class MVCNN(nn.Module):
    def __init__(self, num_classes=1000, pretrained=True):
        super(MVCNN, self).__init__()

        resnet = models.resnet50(pretrained = pretrained)

        fc_in_features = resnet.fc.in_features
        self.features = nn.Sequential(*list(resnet.children())[:-1])
        self.fc = nn.Linear(fc_in_features * 2, fc_in_features)
        #self.adavaptivePooling = nn.AdaptiveAvgPool2d((1,1))
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(fc_in_features , 2048),
            nn.ReLU(inplace=True),
            nn.Dropout(),
            nn.Linear(2048, 2048),
            nn.ReLU(inplace=True),
            nn.Linear(2048, num_classes)
        )

    def forward(self, inputs): # inputs.shape = samples x views x height x width x channels
        inputs = inputs.transpose(0, 1)
        view_features = []
        for view_batch in inputs:
            view_batch = self.features(view_batch)
            view_batch = view_batch.view(view_batch.shape[0], view_batch.shape[1:].numel())
            view_features.append(view_batch)

        concat_views = torch.cat(view_features,-1)
        #pooled_views, _ = torch.max(torch.stack(view_features), 0)
        #outputs = self.classifier(pooled_views)
        outputs = self.fc(concat_views)

        outputs = self.classifier(outputs)
        return outputs

another one is
MULTI-VIEW CONVOLUTIONAL NEURAL NETWORK (MVCNN) ARCHITECTURE 2
class MVCNN(nn.Module):
    def __init__(self, num_classes=1000, pretrained=True):
        super(MVCNN, self).__init__()

        resnet = models.resnet50(pretrained = pretrained)

        fc_in_features = resnet.fc.in_features
        self.features = nn.Sequential(*list(resnet.children())[:-1])
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(fc_in_features , 2048),
            nn.ReLU(inplace=True),
            nn.Dropout(),
            nn.Linear(2048, 2048),
            nn.ReLU(inplace=True),
            nn.Linear(2048, num_classes)
        )

    def forward(self, inputs): # inputs.shape = samples x views x height x width x channels
        inputs = inputs.transpose(0, 1)
        view_features = []
        for view_batch in inputs:
            view_batch = self.features(view_batch)
            view_batch = view_batch.view(view_batch.shape[0], view_batch.shape[1:].numel())
            view_features.append(view_batch)

        concat_views = torch.cat(view_features,-1)
        #pooled_views, _ = torch.max(torch.stack(view_features), 0)
        #outputs = self.classifier(pooled_views)

        outputs = self.classifier(concat_views)
        return outputs

In both architecture, I am concatenating the features to FC layer. Is it safe to say, in 1st approach, I am adding extra FC layer, and in 2nd layer, I just used the last FC layer directly to concatenate the feature?


text:
Constraints Satisfaction for Perfect Matchings
Consider the problem of finding a perfect matching for a graph $G$ as a constraint satisfaction
problem where the variables are the vertices and there is just one global constraint (which
does not depend on the edges of $G$):
My formulation of it is:
Variables: A random subset of $|V|/2$ nodes $V'$
Domain: $V \backslash V'$
Constraints: $Alldiff(val(v_1,v_2,...,v_{|V|/2}))$ for $v_i \in V'$ and $(v_i,val(v_i)) \in E$
Constraint solvers generally use a mixture of constraint propagation methods (e.g., k-consistency, generalised arc consistency) and search methods (e.g., backtracking search).
Does this formulation allow the solver to decide in all cases whether a given graph has a perfect matching simply by using some standard form of constraint propagation (i.e., without using any search)?
I think that it's false because if we consider the complete graph of |V| nodes, any form arc consistency won't necessarily reduce the search space, but I'm not sure.


text:
How to create Nodes out of segmentation Mask of Objects in an image
I am new to geometric deep learning. I am trying to use GNN to create nodes based on the objects present in an image. However, most of the online resources refer to creating nodes equivalent to the pixels of the image. However, what am looking for is to define objects in image as nodes meaning if I have a car, this should be a node and a pedestrian should be another node. How can I do that and How can I define the adjacency matrix between them.

It's been a while since I read anything on GDL. However, one idea that comes to my mind (please, at least cite me if you get millions with these ideas :P) is to first pre-detect the objects with some existing object detector, but you need that, which you may not have. Alternatively, you may use some kind of clustering.


text:
Homography matrix of two boards on same plane
The theory
homography is defined like this:
$H=1/r_3*\begin{bmatrix}
f_{x} & 0 & C_x\\
0 & f_{y} & C_y\\
0 & 0 & 1\\
\end{bmatrix}\begin{bmatrix}
t_{11} & t_{12} &  r_{1}\\
t_{21} & t_{22} &  r_{2}\\
t_{31} & t_{32} &  r_{3}\\
\end{bmatrix}$
with some help of symbolic octave:
$H=1/r_3*\begin{bmatrix}
c_x*t_{31} + f_x*t_{11}& c_x*t_{32} + f_x*t_{12}& c_x*r_{3} + fx*r_{1}\\
c_y*t_{31} + f_y*t_{21}& c_y*t_{32} + f_y*t_{22}& c_y*r_{3} + fy*r_{2}\\
t_{31}& t_{32}& r_{3}
\end{bmatrix}$
The 3rd row of H is:
$H_{3}=1/r_3*\begin{bmatrix}
t_{31}& t_{32}& r_{3}
\end{bmatrix}=\begin{bmatrix}
t_{31}/r_3& t_{32}/r_3& r_{3}/r_3
\end{bmatrix}=\begin{bmatrix}
t_{31}/r_3& t_{32}/r_3& 1.0
\end{bmatrix}$
Now lets say I have image of two squares lying on the  same plane, only shifted through $X$ axis , this is my model:

The square 2 is identical to square 1, but shifted with amount of 2 units in X axis.
I must state that both squares are on the X-Y plane, i.e. both have same rotation matrix against the camera coordinates. i.e. the first two rows of both extrinsic matrices are same.
Thus for first object, the last row of homography matrix is:
$H_{3,1}=\begin{bmatrix}  t_{31}/r_{3,1}& t_{32}/r_{3,1}& 1.0\end{bmatrix}$
For second object, the last row of homography matrix is:
$H_{3,2}=\begin{bmatrix}  t_{31}/r_{3,2}& t_{32}/r_{3,2}& 1.0\end{bmatrix}$
Member wise divide:
$H_{3,1} ./ H_{3,2}=\begin{bmatrix}  r_{3,2}/r_{3,1}& r_{3,2}/r_{3,1}& 1.0\end{bmatrix}$
As obvious the first two members are equal ($r_{3,2}/r_{3,1}$).
Real world example
In action, these are values of computed homography matrices for above image:
$H_1 =\begin{bmatrix}
154.05&115.29&27\\
31.83&-70.807&252\\
-0.057042&0.11161&1\\
\end{bmatrix}$
$H_2 =\begin{bmatrix}
178.42&127.13&374 \\
41.677&-80.555&354 \\
-0.044259&0.1161&1
\end{bmatrix}$
Note that I've double checked both matrices values and there where no error.
the ratio: $\alpha = H_2 ./ H_1$ (member wise divide)
$=\begin{bmatrix}
0.86341&0.90685&0.072193\\
0.76375&0.87899&0.71186\\
1.2888&0.96135&1
\end{bmatrix}$
As you can see the member $\alpha_{3,1}=1.2888$ and $\alpha_{3,2}=0.96135$ are not equal or even close together, but based on theory section, they should be equal.
Can anyone explain it?


text:
Word Embeddings but for Logical reasoning in custom knowledge GPT-3.5 bot
So I have created a chatbot using GPT-3.5 turbo. I have a vector database that holds vector embeddings of brands, ratings, commission percentages, outlets, tags, etc. Here's how the system is designed.

User Asks a question.
The question is converted to vector embeddings using gpt-3 ada embedding model.
The converted vector is searched in Milvus vector database using L2 Method.
The fetched content is then used to create a custom knowledge context.
This custom context is added in prompt for gpt3.5
We send the created prompt to gpt3.5 turbo and answer user.

The Problem:
The system is working perfectly. The problem is that this system is designed to work on similarity searches. This means users can ask for brands selling pizza, burgers, etc. Anything that is keyword based is fine. But when users search for the highest commission, highest rating, or highest-rated pizza brand. The results are random because the results are based on similarity not logical reasoning.
I want to know a way to do this. When a user asks a logic-based question that needs reasoning we can somehow find out the correct brands from the database/vector database and then send that brand to gpt3.5 in a prompt.
I'm open to ideas but don't want to use a rule-based bot.
TIA.

Current state-of-th-art LLM does not do logical reasoning, so no luck so far. If you can solve this problem you may win a Turing Award.


text:
Unexpected improvements in model
I have a poor dataset, where of the training data has a single object labelled in the image.
Same for the validation set even though I reserved the best (more general and well labelled) to then select the best weights.
As far as I can tell, it seems that the model learns to identify many objects, sometimes very well and sometimes not, even though there is a single one labelled (but many images will have several of these object kinds)
I am thinking of re-labeling the data with the current model, and retrain it.
I get that once the model learns to distinguish an object to minimize the error with the dataset, it may also learn to identify other objects in the image.
However, I wonder this:

Would the process of back-propagation punish this extra detections, when the images contain a single labelled object but the model detects many (including the correct one) ?

Please, put your specific question in the title. Thanks.