text:
Can we make an algorithm which can solve any high school (science) problem using ML and DL?
I heard that your ML model's quality depends directly on the quality and the quantity of data you use.
So I was thinking that can question answers be used as data to train an algorithm which can solve any high school science problems? Because we do have a gazillion number of high school books with millions of Question-Answers which are both high in quality as well as quantity.
P.S: I don't have any in-depth knowledge in any of the AI fields, so please answer accordingly!

Probably OP heard that AI makes human obsolete so he wants to obtain PhD in AI without learning anything. It is much in agreement with popular version of AI presented in media and by Elon Musk. Maybe even it is Elon who ask? Who knows?

@kakaz, I would love to do PhD, but given I am a high school student, I have some more important short-term goals :)

Focus on these and everything will be all right!

The short answer, I guess, is yes. In theory, you could have a massive, insane network that could take as input any high school science textbook question and output an answer. But think about what that entails:
Take in a question, in any of the various languages that high school science textbooks can be written in. Get all the relevant information out of that. This includes questions like "List four noble gases" as well as ones about applying Kirchoff's laws to a circuit as well as explaining how the ribosome changes its shape to produce proteins. There is soooooooo much information that is contained in just the wording of those questions and so many topics. Then, let's say you have converted the question into a representation that can be "understood" by the algorithm. I want to emphasize that ML models do not "know", "understand", "think", or any other of the words we use to personify them. They are basically just a mathematical function of some level of complexity (usually very complex). That being said, now the algorithm needs to relate this tractable representation of the question to the correct answer.
Since everything before this created some representation of the question, you could view that as a function mapping the input to this representation. Then, everything between intermediate representation and output answer is another function. So, your model needs to learn a single function that is the sum of all human high school level science understanding.
These algorithms you want to build would end up being staggeringly complex. In addition, as you add complexity to an ML model, you need significantly more data to train it. How will you take every high school science book in the world and convert it into digital pairs of questions and answers? How will you handle that some of the answers are incorrect? What is the benefit of this system?
The long answer, I think, is no. It makes no sense to do this. Instead of learning this complex mapping, just build some solvers with equations built in that the user inputs some numbers to and selects the equations. It will take so much less time to build and deploy and I guarantee work better. There are people building ML systems that are trying to get the SAT entirely right (see page 33 of this report). First, I don't think this is what you're asking and second, it still makes more sense, in my mind, to build some solvers. Hopefully, this all makes sense and, if anyone sees things I misstated, let me know.

First of all, thanks for the answer. It is amazing!
Second, you asked what is the benefit of this system? Let us consider subjects like Inorganic chemistry, Zoology, Botany, History, Civics and Political Science ( pick up one subject out of these ). In these subjects, ( at least till the high school level ), there is no IN DEPTH understanding required, as they're mostly facts. Now, if a software has mastered any of the subjects, we could use it as a kind of teacher for the students. Think of it like " Siri for education ".

Even in theory it is completely impossible.

And students would be much more comfortable in asking questions to a software rather than a teacher because they won't ever be judged. No matter how silly their questions are, they could always ask them and get an appropriate answer, which is better both for their curiosity and their education!

@kakaz yeah I do understand. But yeah, it feels nice to think about such stuff you know ;)

Things you name as school problems today, yesterday was serious scientific problems, used to build technology, knowledge and the things you use in real life. Thinking it is possible to train AI to solve it is just equivalent to think it is possible to build machine solving all problems of XIX century science say. It is impossible. If you think other way, just check something about just integration and implementation full algorithm of it. Oner side, clearly to may build just big database to check if given problem is routine one, and check given book-like solution. But no AI is needed there...

@kakaz , take a look at this page: https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/Harvard_University.html . I don't think that it is impossible now! :P

Do you really think it is enough to have big database?

Its true that your AI model's performance depends on the quality of data that you use. However, high quality data alone is insufficient to guarantee that your model will learn effectively and score well on a particular dataset. Other factors such as smarter algorithms and the use of high performance computing infrastructure must be factored in for your AI system to perform well.
Although A.I research has made massive progress in the past decade, ML engineers are yet to build a system that can match the general scope and generalization ability of the human mind. Upto the first decade of the 2000's AI was dominated by expert systems that emulated the decision making ability of an expert. AI at this point couldn't process unstructured data and therefore it lacked the capacity to sit for and pass high school exams.
This was until 2011 when IBM Watson a question answering computer system competed against two former Jeopardy quiz show winners and placed first. IBM Watson was built on top of Deep QA  (a computer system that could answer natural language questions) and UIMA (a software achitecture to process and analyse unstructured information). Below is a link to a paper giving an overview of how IBM's Watson works https://www.aaai.org/Magazine/Watson/watson.php
In 2012 a team led by Geofrey Hinton won the ImageNet competition by exploiting deep convolution networks. This was soon followed by Dahl's team winning the Merck Molecular Activity Challenge using deep neural network architecture. Yann LeCun's work in CNN's, Geoff Hinton's back propagation and Stochastic Gradient Descent aproach to training datasets alongside Andrew Ng's large scale use of GPU's ignited accelerated progress in ML. This was frequently referred to as unreasonable effectiveness of Deep learning.
Following recent advances in fields such as image captioning, natural language processing, information retrieval and computer vision it is highly probable that current generation AI systems can pass high school exams such as SAT.
The Allen AI Institute has made significant progress in developing AI systems that can read, learn and express that understanding through question answering and explanation. Founded by Paul Allen Microsoft's co-founder, the Allen AI Institute's singular focus according to their mission is to conduct high impact research in the field of AI. Below is a news link covering their cognitive system passing high school exams fortune.com/2015/09/21/computer-artificial-intelligence-math/
So far Allen AI Institute has demonstrated a cognitive platform called Geos that is capable of answering geometry questions as well as the average high school student. While another system called Aristo can answer high school science exam questions by leveraging information extraction alongside knowledge representation and reasoning models. You can access AAI's GeoS service here http://allenai.org/euclid/ and Aristo here http://allenai.org/aristo/
Meanwhile researchers working on the Todai project in Japan have demonstrated a cognitive system that is capable of passing the Tokyo University Mathematics entrance exam. My conclusion from the above examples is that possibly we already have AI that can sit for and pass high school exams.

I'm thinking that you could write an AI that takes the question as input, weights it, and googles info based on the first layer of neurons, then takes the first two to three pages of results and spits out an answer. It would be a crapshoot, but maybe you could take the list of results, choose one using another layer, choose the info from the page using a third layer, then answering the question using the info.


text:
What fast loss convergence indicates on a CNN?
I'm training two CNNs (AlexNet e GoogLeNet) in two differents DL libraries (Caffe e Tensorflow). The networks was implemented by dev teams of each libraries (here and here)
I reduced the original Imagenet dataset to 1024 images of 1 category -- but setted 1000 categories to classify on the networks.
So I trained the CNNs, varying processing unit (CPU/GPU) and batches sizes, and I observed that the losses converges fastly to near zero (in mostly times before 1 epoch be completed), like in this graph (Alexnet on Tensorflow):

In portuguese, 'Épocas' is epochs and 'Perda' is loss.
The weight decays and initial learning rate are the same as used on models that I downloaded, I only changed the dataset and the batch sizes.
Why my networks are converging this way, and not like this way?

What do you mean, you settled 1000 categories?

I specified to network/library 1000 different output categories, but in practice I just give 1 to the network. I dont know exactly how Tensorflow/Caffe works, but I think that this value (1000) is used in softmax layer

Ok, so the fact that you always give the same category means the network only has to output the same value for any given input. This is exceedingly easy to learn (all the weights always shift in the same directions) and this make this graph.
Once you have more categories (try 10 for example), you'll see the graph will be closer to expected.

Thats make a lot of sense. But why loss converges faster if my batch is too small? With sizes lower than 32, the networks converges with just 10% of an epoch (aprox. 100 images)

You have less variance when you have a smaller batch size, so the network ignores the noise even faster

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
Searching an AI to recognize and locate persons in a factory
TLDR : Is there an AI available that can recognize employees in a factory and tell when they entered and left pre-defined areas?
I work in a factory where we gather cycle time data from various inputs (computer interfaces, bar code readers and RFID tags). We follow both parts moving along the production line and persons working on those parts because they move from stations to stations during their day and can be more than one at the same place to help one another.
Our goal is to know how many people are working on each part at any given time, but the problem is that it takes time for people to check themselves in when they get to their workstation and they can also forget to do it.
Many employees asked me if I could find a way to track them automatically so they wouldn't need to check in everywhere they go. And then I stumbled upon Microsoft Workplace Safety Demo and Yitu System (this one is scary) but they both seem to be a little overkill for my needs.
After learning about these, here is the ideal AI features that I'd need :

Can use video feeds to detect new people in the workspace and prompt someone (via e-mail or text message) for identification.
Can use video feeds to recognize persons that are already known and document every time someone has entered or left a pre-defined zone (workstation) in the video feed.
Won't force me to have a camera for each workstation.
Allows one workstation to span on more than one video feed.

As of now, I found nothing available that does this, but I may have missed lots of things because this is not what I am used to work with. So maybe you know something that could help me with that?

Are you looking for a turnkey solution or algorithms & ideas to write your own solution?

I'd rather not reinvent the wheel if it is already available to me.

I am definitely not the most qualified person in order to answer this question, but I might be able to give you a couple of buzz words to go and do some more research on your own. A convolutional neural network is usually used for analyzing anything visual. (you can read about it here: https://en.wikipedia.org/wiki/Convolutional_neural_network)
There are a lot of frameworks that allow you to build recognition classifiers that can be loaded into a real-time video feed and give you feedback when it gets a hit.
Some of the ones that I know of are:

https://caffe2.ai/ or http://caffe.berkeleyvision.org/
https://pjreddie.com/darknet/
https://docs.opencv.org/2.4/modules/ml/doc/neural_networks.html

With Caffe being the most popular.
Some articles on how these work:

https://pjreddie.com/darknet/yolo/
http://coding-robin.de/2013/07/22/train-your-own-opencv-haar-classifier.html

Hope this is at least able to get you on the right path.

This is very interesting! YOLO v2 seems very impressive. So what I'd need from there would be to (1) detect the person, (2) send the image to a face recognition software and update the information when it returns something (when they face the camera), (3) extrapolate the persons's feet position on the floor, (4) note the date, and person every time they enter/exit each zone.

That seems correct for you use case, yes.

training a model to identify new persons from video logs seems to be a daunting task.
You will be needing lots of data and computational power for building such a model.
and there is relatively low work going on videos  due to the amount of computational power required , even training a simple video classifier with reasonable accuracy requires lots of expertise and resources (data and computing).
Having said that recognising persons from videos and "new" person identification seems to be even more difficult , you should definitely read some papers on video classification using convolutional neural networks to get an idea of how hard the problem is. frameworks are only mere tools which give you helper functions to build neural networks , so  choice of frameworks is not a question.
The real question is are you sure you want to train a model (CNN) for this task?
Can use video feeds to detect new people in the workspace and prompt someone (via e-mail or text message) for identification
for this you should be training a convolutional network on the factory videos dataset , i.e you the cnn trains on frame level for the videos.
and recognizing and classifying "known" and "unknown" faces from videos is a research problem in its own right

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How does this part of algorithm works?(K-means)
I can't understand the red box. what do does that 1 do before {c(I)=j} ?
Also how does all the algorithms work? May someone explains the to steps more clearly by one example!

1_{c^(i) = j} means if c^(i) is equal to j the value is 1 and zero otherwise.
In summary, k-means try to solve Voronoi cells for each mean point, and then update mean of members of a cell and update Voronoi cells again. This step is repeated until there is no change in members of the cells.

Is mu(j) a vector?
I am confused about I and j. what do counter these 2 letters?

@user8663682 mu(j) is mean of members of the cluster j at each step, and c^(i) is index of the cluster which is assigned to the point of x_i.


text:
What is uncontrolled logging policy?
I am reading Learning from Logged Implicit Exploration Data
It says

Formally, given a dataset of the form S = (x, a, r_a)* generated by the interaction of an uncontrolled logging policy

What is such a policy?

This policy is usually referred to as the Behaviour Policy when talking about off-policy learning. It is the policy that actually takes actions in the world. The policy that one wants to learn, such as a greedy policy in Q learning, is know as the target policy. In this way the behaviour policy “logs” experience that the target policy (or policies) may learn from.


text:
Creating a classifier for simpler classifiers trained on few training samples
Suppose I have a classification problem with a stream of training-samples constantly arriving over time. I cannot keep all training-samples in memory, but I still want to train a classifier that will have the "wisdom" of all samples, and additionally, I want the classifier to become better whenever it gets new samples.
I thought of the following idea. Suppose we have enough memory to keep 100 samples. Then, for each run of 100 samples, we will train a different sub-classifier. We will have a meta-classifier that will classify based on voting between all existing sub-classifiers. Over time, we will have more and more sub-classifiers, so hopefully the meta-classifier will improve with time - it will have a "wisdom of the crowds" effect.
Has this method been tried before? Specifically, has it been tried in a deep-learning sequence-classification setting?

The voting technique that you described is called Ensemble Learning and its improvement over time is guaranteed if each classifier is at least a little better than random.


text:
Is it better to make neural network to have hierchical output?
i'm quite new to neural network and i recently built neural network for number classification in vehicle license plate. It has 3 layers: 1 input layer for 16*24(382 neurons) number image with 150 dpi , 1 hidden layer(199 neurons) with sigmoid activation function, 1 softmax output layer(10 neurons) for each number 0 to 9.
I'm trying to expand my neural network to also classify letters in license plate. But i'm worried if i just simply add more classes into output, for example add 10 letters into classification so total 20 classes, it would be hard for neural network to separate feature from each class. And also, i think it might cause problem when input is one of number and neural network wrongly classifies as one of letter with biggest probability, even though sum of probabilities of all number output exceeds that.
So i wonder if it is possible to build hierchical neural network in following manner:
There are 3 neural networks: 'Item', 'Number', 'Letter'

'Item' neural network classifies whether input is numbers or letters.
If 'Item' neural network classifies input as numbers(letters), then input goes through 'Number'('Letter') neural network.
Return final output from Number(Letter) neural network.

And learning mechanism for each network is below:

'Item' neural network learns all images of numbers and letters. So there are 2 output.
'Number'('Letter') neural network learns images of only numbers(letter).

Which method should i pick to have better classification? Just simply add 10 more classes or build hierchical neural networks with method above?

Item network should be recogniting regional register number standards and typical places of numbers on those standards. Otherwise your 'Item' phase has equal task than the whole system, because 'Letter' and 'Number' do not differ in plates so much you could right away choose between the two without first evaluating the exact object on each case.
Side note: correct me if that is not true on your local plate system

My system tries to find license plate area. In order to do that, it gets input of rectangles that possibly contain digits. Those rectangles form cluster by distance of each other. And it counts how many rectangles containing letter of number are there in each cluster to finally get clusters as part of license plate area. Here's where neural network needed. So there are no information for regional number or place because input is not complete license plate.

Just use one network with a larger Softmax output layer and more hidden units. If you have enough training data, it will work just fine. In fact it could emulate the architecture you propose.

I agree with the above answer. If you want to research this more in-depth look at this paper: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42241.pdf


text:
Where can I get Imagenet test dataset labels for evaluation
I have the imagenet train, validation and test set. I have been able to assign each image in the validation set into its respective class folders with the help of some online resources. However, for the test set, I could not find the labels.
All the model evaluation experiments on Imagenet dataset I’ve seen people do only use a fraction of the validation set. I have not been able to find one that used the test set or a fraction of it.
I really want to use the imagenet test set to evaluate my pytorch model. How do I go about it or where can I get the labels for the test set?

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why can't language models, like GPT-3, continuously learn once trained?
GPT-3 has a prompt limit of about ~2048 "tokens", which corresponds to about 4 characters in text. If my understanding is correct, a deep neural network is not learning after it is trained and is used to produce an output, and, as such, this limitation comes from amount of the input neurons. My question is: what is stopping us from using the same algorithm we use for training, when using the network? That would allow it to adjust its weights and, in a way, provide a form of long-term memory which could let it handle prompts with arbitrarily long limits. Is my line of thinking worng?

I'm not well acquainted with machine learning and the involved literature, so this is a very newbie question. I tried to make it as high quality as I could, but I'm certainly limited on how well I can make questions about a subject I don't know. Hope this kind if question is acceptable here.

In theory, there is nothing stopping you from updating the weights of a neural network whenever you like. You run an example through the network, calculate the difference between the network's output and the answer you expected, and run back propagation, exactly the same as you do when you initially train the network. Of course, usually networks are trained with large batches of data instead of single examples at a time, so if you wanted to do a weight update you should save up a bunch of data and pass through a batch (ideally the same batch size that you used during training, though there's nothing stopping you from passing in different batch sizes).
Keep in mind this is all theoretical. In practice, adjusting the weights on a deployed network will probably be very difficult because the models weights have been exported in a format optimized for inference. And it's better to have distinct releases with sets of weights that do not change rather than continuously updating the same model.
Either way, changing the weights continuously would not affect the "memory" of the network in any way.  The lengths of sequences that sequence-to-sequence models like transformers or RNNs can accept is an entirely separate parameter.

Thanks for all the info. The only thing I don't get is why changing the weights continuously wouldn't allow it to have memory and learn new things. Isn't that how humans learn; by changing the weights of some specific neurons?

GTP-3 is trained on 45 TB data, so any additional examples you might give it will not make much difference https://www.springboard.com/blog/data-science/machine-learning-gpt-3-open-ai/

@MaiaVictor The memory in sequence-to-sequence models is more akin to working memory in humans: memory that is stored not in neuron synapses, but in the activity of neuron. The long term synaptic memory you're referring to is used by the network to learn the overall task. But this doesn't mean changing the weights will let you feed in arbitrary length sequences: those sequences are stored in the activity of the network's neurons. In RNNs, this working memory is modeled explicitly by gates in each neuron (look up LSTM or GRU). In transformers, it's much more complicated.

Keep in mind that while what Stefan said is true, you can make a big difference in the quality of GPT3's: not by training the network itself on new examples, but by including examples at the start of your prompt. This is called prompt engineering, good examples are available here https://beta.openai.com/examples/

@Stefan isn't this just strongly hinting that training is terribly inefficient? Humans have seen way more than 45 TB of raw data in their lifetime, yet they can read a single sentence such as "jellyfish are biologically immortal" and learn a new concept for the rest of their life. Somehow humans are extremely good at storing new "insightful" information they're exposed very briefly, while ignoring massive "bland" amount of information (like all you see while commuting).

With $175$ Billion parameters, GPT-3 is remarkably large and powerful, but it has several limitations and risks associated with its usage. The biggest issue is that GPT-3 can't continuously learn once trained?. It has been pre-trained(as the name ~ Generative Pre-trained Transformer), which means that it doesn't have an ongoing long-term memory that learns from each interaction.
In addition, GPT-3 suffers from the same problems as all Neural Networks: their lack of ability to explain and interpret why certain inputs result in specific outputs.
Another reason could be that the model has reached a point of diminishing returns, meaning that any additional training is unlikely to result in significant improvements.

"A significant concern when building AI models like these is diminishing returns—that is, you cannot simply scale the model up forever. At some point, some factor(s) of the model will plateau, whether it’s the information generated, the dataset size, the training regime, etc".

However, at the level of GPT-2, there was no indication that this plateau had been reached. Thus, the “bigger and better” tactic continued, bringing us GPT-3". So, it may also be possible that the model has simply reached a plateau in its learning and is unable to make any further progress.

References:

OpenAI GPT-3: Everything You Need to Know
an overview of GPT-3: AI of the future
GPT-3


text:
References on Theoretical Bandit Problem
I am going to start learning the bandit problem and algorithm, especially how to bound the regret. I found the book ``Bandit Algorithms'' but it is not easy to follow. It is based on advanced stochastic processes and measure theory in some cases. I am wondering if there are any lecture notes, or courses to start.

I like this very detailed blog post. https://mpatacchiola.github.io/blog/2017/08/14/dissecting-reinforcement-learning-6.html. It's equivalent to a chapter or two of an introductory text, and covers a couple other adjacent but also very important topics.

Thanks for your comment. It actually covers just a few topics and it is not theory-based. I am looking for something like a course for Ph.D. level students who will do research on Bandit.

You're going to need to learn measure theory, advanced probability and stochastic processes. The book neurodynamic programming covers some stochastic convergence proofs but you need some results for martingales.
In essence theoretical RL is extremely difficult, even the bandit problem is extremely.

Thanks. I would be thankful if you can recommend some references to cover these topics. These topics are so broad and may need many books. Is there any book covering the essentials of them?


text:
Is AI the right tool for solving walking?
Modern robots walk like they had a heart attack and then a stroke... twice. Everyone in AI fiercely believes that the higher number of neurons (or the quality of training with smaller number of neurons) is the solution and things will definitely improve after pumping in more neurons and more time.
Everyone talks about humans and how many neurons we have and how our babies need time to learn to walk.
Meanwhile horse babies are born and start running right away after a minute or so. They have less neurons than we do.
Ants jump out of their eggs and zoom around like crazy, their brains have only 250k neurons.
Xeno bots were moving around repurposing their protrusions for swimming, sensing the environment, changing directions and such - those are individual cells, no brains, no neurons.
Is AI really a way to solving organism movement problem? It seems like a completely wrong tool for the task.

In principle, for any complex problem there is only one solution: AI.
Humanoid robots are just starting. Yes, they are here let say 30 years in some cases but it was only hard-coded attempts to make robots.
First true case of useful robot was introduced by Tesla one year ago and first demo showed yesterday.
This is the first real case when real AI was used and not just hard-coded  programming. Tesla AI - which has been developed for cars - FSD.
You can watch on youtube how increasingly better FSD has become in last few months. Everything is recorded. The same speed of improvement will be visible on Tesla Bot.
So yes. This is extremely complex problem therefore only AI can solve this.
Boston Dynamics tried this hard-coded way (20 years of trying) and they failed.
Tesla nailed it when they was able to put together walking robot in 8 months.
And the positive impact is that AI of Tesla Bot will impact FSD and FSD will impact AI of Tesla Bot. The synergies will run in both directions. It's inevitable.
> Meanwhile horse babies are born and start running right away after a minute or so. They have less neurons than we do.
Number of neurons is correlated to number of organs and complexity what organism is capable of + number of environments where can life. Variation of movements for humanoids are far more complex than variation of movements and capabilities of horses.
And movements for humanoids must reflect also general thinking - how to use tools. Humanoids are tool makers and every tool category has different set of movements. So if you want to solve walking for humanoids you have to build in tool usage what has impact on walk movements.


text:
Is this a better formulation of the Turing test?
Would the Turing test be better formulated by assessing whether the creator (as opposed to a third party) was able to tell the difference between their program and a human?
A magician, through the use of misdirection and slight of hand, could make an audience believe they are witnessing true magic, however the magician would only be able to make themselves believe if in fact the magic was real.
Would formulating the Turing test as such put a greater focus on creating “real” AI as opposed to the illusion of AI?

For the purpose of having AI on par with humans and releasing it to the market is not that relevant that the creator, a single person, can still not be fooled.
Moreover this would create a conflict of interests, as the creator may pretend to be fooled.
In conclusion: no.

Interesting twist on the classic.  @Rexcirus makes a good point about the market, and indeed, the rest of the world.  His 'no' is well-felt.  To pile on a little, I would say that it is very easy for humans to fool themselves.
Fooling yourself is normally prevalent in youngsters, and tends to wear out as they get older and interact with people who say things like "you're fooling yourself".  So in that sense, it may be more of a "learning process for developers." Although today there are many an LLM dev who seem reluctant to listen to that kind of feedback.
I would offer that your "personal Turing test" is a good starting point for an AI dev.  In the sense that if their product cannot pass that test, then they can begin the difficult journey of understanding why.  If they opt to skip that test, it's no big deal.  They'll find the wall eventually!
Excellent point about magicians!


text:
Carla simulator and gpu
I am trying to buy a HP laptop with iris intel graphic card to run the Carla self driving simulator. Can carls run on iris or I need to buy a laptop with nividia gpu. Thx u

While Carla was developed by Intel their GitHub recommends a powerful Nvidia GPU.
Comparing a mid-level NVidia GPU vs an Intel IRIS the NVidia comes out ahead. You could use an Intel IRIS, but the performance wouldn't be very satisfactory.
An Intel Arc A730M-Powered Laptop would be a minimum laptop configuration, with an Intel GPU, for running Carla; still not anywhere as well as a top of the line GPU (which is necessary for AI), but I understand that cost is also a consideration for many people.

All Deep Learning tasks generally require a NVIDIA GPU, and it is generally recommended to use them. It is given on carla website that a Nvidia GPU is recommended. Currently, Intel or other GPUs are not much compatible with tensor cores or deep learning due to lack of some structure.
Hence, a Nvidia GPU is recommended.
However, you can also use online GPUs which can provide you a better and configurable experience.


text:
Does iterative deepening depth-first search expand at most twice as many nodes as breadth-first search?
My understanding is that iterative deepening search is roughly equivalent to breadth-first search, except instead of keeping all visited nodes in memory, we regenerate nodes as needed, trading off memory for time.
The $(i+1)$th layer of a tree has about as many nodes as the entire rest of the tree before it*. Thus, on each iteration, iterative-deepening effectively has to expand the $(i+1)$th layer twice whereas breadth-first search only has to expand it once. So IDS takes at most twice as long as BFS. Is this correct?
* A proof of this statement is as follows: in a tree with branching factor $b$, the $i$th layer has $b^i$ nodes. The number of nodes up to and including the $i$th node is given by the $i$th partial sum of the geometric series, $\frac{b^{i+1} - 1}{b-1}$, which is strictly less than the number of nodes in the $(i+1)$th layer ($b^{i+1}$)
Please let me know if all this is correct and whether clarifications are needed anywhere to make it more rigorous.

That is correct. Iterative deepening depth-first search expands at most twice as many nodes as breadth-first search since it only needs to keep track of the current generation of nodes, and not all visited nodes.


text:
Policy Gradient Methods when using a fixed initial sequence of actions
I am implementing a Policy Gradient agent based on IMPALA. Specifically, I'm working on DeepNash, but that is not considerably different from vanilla IMPALA for the purposes of this question.
In my implementation, I am running vectorized games that I can run batched on my GPU using convolutions and such, similar to this https://towardsdatascience.com/learning-to-play-snake-at-1-million-fps-4aae8d36d2f1?gi=7ef24f81a7a6
However, because of a technical limitation, I have to fix the first $n$ actions of every environment in the batch to be the same. To explain further, these initial $n$ actions consist of placing 'walls', and the placement of walls determines a set of 'vision' mask tensors that filter the observations of the environment. It is not possible to store a separate set of vision masks for each environment, so I've opted to use the same walls/masks for each environment in the batch.
This would be a fine compromise, if not for the fact that it is seems to be off policy, since we are not actually sampling the net works output during this initial stage. In fact, it could be described as selecting from a policy that is identical to the nets except in the initial stage, where it is deterministic. From this viewpoint, it seems natural to use off-policy correction. Indeed, DeepNash already uses IMPALA, so I was hopeful that v-trace would 'absorb' this compromise. However, the policy that is generating the trajectories is deterministic, so this causes singularities in the $\log \frac{\pi (a)}{\mu (a)}$ terms of the v-trace estimate ($\mu$ is the actor policy).
My intuition tells me that it wouldn't be catastrophic if I were to use the trajectories in the batch as if they were on-policy. It seems tantamount to a difference in learning rate at 2 distinct phases of the game: initial and non-initial. Is this correct? Any advice?


text:
Does more data increase training accuracy in neural networks?
I was wondering what is the performance benefit of feeding more data to a machine learning model like a neural network? Like I know one of the benefits is that it increases generalization - testing accuracy, but I was wondering if does it affect training accuracy of the model?

The question use specific terms in a vague way so let me set some very basic ground definitions first. It might sounds trivial but please bear with me cause it's easy to give reasonable answers that in reality make no sense.
data: any unprocessed fact, value, text, sound, or picture that is not being interpreted and analyzed. It can be real, i.e. gathered trough real observations/experiments or synthetic, i.e. artificially constructed based on some hand crafted distribution.
training/test data: different subset of data, the difference being that testing are not used to train or tune the model parameters and hyperparameters. Important to note is that we don't always have knowledge about the real distribution of the data, meaning that the distribution of our training data many times do not match the distribution of testing data.
accuracy: it's a specific metric used to evaluate only a specific subset of machine learning tasks among which (and mostly) classification. Is also a pretty unreliable metric in some circumstances like multi class classification or unbalanced datasets.
generalization: the ability of a model to perform well on unseen data. In principle it has nothing to do with the metrics used to evaluate a model, even though metrics scores are the only tool we have to assess it.
You ask if using more data can increase training accuracy and you already pointed out that using more data is meant to increase generalization, which you consider equal to testing accuracy. You`re right when you say that adding more data serve the purpose of increasing generalization, but as I wrote in the definition of generalization, in principle we can't always expect a linear relation between a model generalization and its metrics scores, and in fact adding data might as well decrease model generalization in some situations.
As a basic example let's consider an imbalanced dataset with 90% instances belonging to class A and 10% instances belonging to class B. A model will easily learn to overfitt the data predicting only class A, still reaching 90% training accuracy. In test phase we might even observe a similar score if the distribution match the 90/10 ratio of training data. To prevent overfitting and increase generalization, we add instances of class B to make the dataset balanced, i.e. 50% instances class A 50% instances class B. Suddenly the model works perfectly, and reach 100% training accuracy. We see tough that in test phase the accuracy drops to 40%. How come? If the test indeed had the same 90/10% distribution among classes A and B, training the model on a 50/50% distribution teach the model to overpredict class B, so the model is now predicting class B, which is why we added more data, but it's now predicting too many times class B, leading again to poor generalization.
Of course this is a toy scenario, but be aware that classic data augmentation and synthetic data always introduce the problem of introducing biases in the training distribution. Also, note that using a different metric like f score alongside accuracy would let you catch immediately if the model is generalizing more or if it's only an artifact produced by the accuracy metric (like the initial 90% score when totally overfitting).

Thanks for the answer. The main reason I'm asking this is to assist in faster debugging. Like basically what I was wondering was if I run experiments to see how best to modify my loss functions or my model architecture, could I run it on a small dataset to see the changes, then run it on the actual dataset. Like if those debugging changes would be present regardless of the dataset size.
That way I get to do model debugging and experimentation faster if changes are still present in a smaller dataset as smaller datasets run faster.

Using more training samples decreases the chance of over-fitting. However, I think, it may not occasionally result in a decrease in the training error, maybe the opposite (look at the loss function definition). for instance, if you have only very few samples a very deep network is capable of memorizing them all which makes training error = 0.


text:
Why VQ-VAE instead of VAE?
From the paper on VQ-VAE, it said that the vector quantized variational autoencoder (VQ-VAE), differs from VAEs in two key ways:

the encoder network output discrete, rather than continuous, codecs
the prior is learnt rather than static

The paper further says, using the VQ method allows the model to circumvent issues of "posterior collapse".
So can anybody explain the main use of vector quantization in VQ-VAE, and how does it improve the VAE model in a mathematical or some kind of rigorous way so that I can understand it?
Also explain points 1, 2, and "posterior collapse" in detail with mathematical notions if possible.


text:
Compare Stable-Baselines3 vs. Tianshou
What would you recommend between Stable-Baselines3 and Tianshou for applied research in Reinforcement Learning?
Can anyone provide a comparison of the strengths and weaknesses of each library? Or at least some criteria for choosing one over the other?
It seams like Tianshou is much more flexible and generalist but I have not been able to find a benchmark that clearly establishes this.

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.


text:
Training a neural network in full batch training
It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that it is possible to train the network on one datapoint at the time. It is quite rare, on the other hand, to see research papers, or state of the art models, that are trained on a whole dataset simultaneously, i.e., in full-batch training.
For a research project, it would be useful for me to come up with a list of applications/models where training a neural network in full batch training is preferable with respect to using multi-batches. An example I have found is the COIN and COIN++ papers, that train on full batch training in order to memorise datapoints.
Question:
Do you know other applications where performing full-batch training is preferable to use mini-batches? Which ones?

I think the closest approach to what is described in the papers you linked is Neural Style Transfer. But I see also some (potential) misunderstanding regarding the full batch training, so let me elaborate a bit.
In the papers you linked and the neural style transfer one a model is trained on a single image. Not a single image at the time, literally on a single image. So I wouldn't call it full batch training, since there's no batch at all, and there can't be one. But I guess this is open to interpretation and it depends on how you define a batch in the first place.
it would be useful for me to come up with a list of applications/models where training a neural network in full batch training is preferable with respect to using multi-batches
Aside from these examples (i.e. overfitting or fine tuning on a single training instance) I doubt you'll find papers where people use full batch training. Not only full batch training comes with all sort of computational issues (like keeping in memory all gradients for an entire dataset) but even from a mathematical point of view mini batch and stochastic gradient descent works better than full batch and (classic) gradient descent. I did link some references about this topic in another answer.

Thank you for your answer and reference! It looks interesting. A little comment: while it is true that the COIN paper trains on a single image, it is encoded as W x H different datapoints, one per pixel. This is why I called it full batch. More generally, yes, I'm interested in all of these very edge cases used in practice.


text:
Batching together similar length sequences to avoid padding and packing
I am training an RNN in PyTorch to produce captions for images. It's a pretty standard architecture – the image is processed by a pre-trained InceptionV3 to extract features, the recurrent module processes the words seen so far and then its result is merged with image features and fed through a linear layer to produce a probability distribution over the vocab to predict the next word. A network trained this way can auto-regressively predict a whole caption for an image (it's seeded with an artificial  token at the beginning to have a preceding sequence to start off with).
To train the network every caption is decomposed like this in the training dataset:
X, y
<START>, a
<START> a, man
<START> a man, is
<START> a man is, sitting
<START> a man is sitting, on
<START> a man is sitting on, a
<START> a man is sitting on a, chair
<START> a man is sitting on a chair, <END>

The result is that there is a lot of input sequence length variation in the training data which is a problem since we would need a lot of padding in batches. I am aware that there are techniques of counteracting this (pack_padded_sequence in PyTorch) but I came up with another idea.
I am modifying the Sampler object that's used to provide indices for the DataLoader to construct batches in such a way that the Sampler orders all of the example indices by input sequence length, groups it into batch-size chunks, and shuffles the chunks before feeding them to the DataLoader. As a result the sequences in individual batches are almost always the same length – not because of any padding but because of the fact that we ordered the examples by length before grouping them. There will be some batches that will encompass examples on a border of length groups so some padding will be necessary but it will be minimal (just one zero for a group of lengths [5, 5, 5, 6, 6, 6] for example) and the number of such groups would be negligible (~0.002% of the batches in my case).
My question is – does this method introduce some kind of drawback that I am not aware of? I find it simpler than having completely random batches and padding and packing them. I can't tell however, if it would introduce some kind of bias to the model. My intuition tells me that this should not be problem since the sequence lengths for batches would vary greatly across training (since I am shuffling the index chunks), but I wanted to get a second opinion. I guess the question boils down to -> is sequence length variation within the batch important for proper training? Or, for this specific case -> is it important that the net process examples decomposed from a single caption within the same batch or close to each other time-wise?
I'd be happy to train the network with both solutions and check the results myself but it's quite time-expensive and I'd prefer to avoid some stupid stuff if I can.

You can read that this was done in "Attention is All You Need", for Transformers: "Sentence pairs were batched together by approximate sequence length.".
But, with RNN you don't need any padding at all when using packedSequences, so I don't think your idea will be efficient, if I understand correctly.


text:
Very low accuracy (0.11) and it remains constant after few epochs on MNIST database
I am developing my ANN from scratch which is supposed to classify MNIST database of handwritten digits (0-9). My feed-forward fully connected ANN has to be composed of:

One input layer, with 28x28 = 784 nodes (that is, features of each image)
One hidden layer, with 1,2,3,4 or 5 nodes (shallow network)
One output layer, with 10 nodes (one for each predicted class)

and has to compute gradient w.r.t. weights and bias thanks to backpropagation algorithm and, finally, it should learn exploiting gradient descent with momentum algorithm.
The loss function is: cross-entropy + softmax, since the task is about classification.
Each hidden neuron is activated by the same activation function, the sigmoid; meanwhile the output's neurons are activated by the identity function.
The dataset has been divided into:

60.000 training pairs (image, label) - for the training
5000 validation pairs (image, label) - for evaluation and select the network which minimize the validation loss
5000 testing pairs (image, label) - for testing the model picked using new metrics such as accuracy

This is the curves I get about training and validation loss:  and these are some logs about their loss and accuracy on the validation set:
E(0) on TrS is: -0.09995663952048875  on VS is: -0.10000215508367913  Accuracy: 9.120000000000001 %
E(1) on TrS is: -0.09997782058505714  on VS is: -0.10000721545529842  Accuracy: 9.120000000000001 %
...
E(47) on TrS is: -0.1000167196491167  on VS is: -0.10001587893489938  Accuracy: 11.42 %
E(48) on TrS is: -0.10001676221437443  on VS is: -0.10001594086326895  Accuracy: 11.42 %
E(49) on TrS is: -0.10001680503178402  on VS is: -0.10001599995199295  Accuracy: 11.42 %

Validation loss is minimum at epoch: 22

Finally, if I test the network at epoch 22, which is the epoch where the validation loss is minimum, I get the following result:
Accuracy score on test set is: 11.28 %

These are my questions: as you can see, the accuracy remains (almost) constant and it is very low. It has a low value on the test set as well and on the training set. Is that normal? If it is not, how can I fix?
This is my code:
import numpy as np
from scipy.special import xlog1py
from scipy.special import expit
from matplotlib import pyplot as plt
from mnist.loader import MNIST

def softmax(y):
    e_y = np.exp(y - np.max(y))
    return e_y / e_y.sum()

def cross_entropy(y, t, derivative=False, post_process=True):
    if post_process:
        if derivative:
            return y - t
        return -np.sum(np.sum(xlog1py(t, softmax(y)), axis=0))

def sigmoid(a, derivative=False):
    f_a = expit(-a)
    if derivative:
        return np.multiply(f_a, (1 - f_a))
    return f_a

def identity(a, derivative=False):
    f_a = a
    if derivative:
        return np.ones(np.shape(a))
    return f_a

def accuracy_score(targets, predictions):
    correct_predictions = 0
    for item in range(np.shape(predictions)[1]):
        # print(predictions[:, item])  # dbg
        argmax_idx = np.argmax(predictions[:, item])
        # print("argmax idx", argmax_idx)  # dbg
        # print(targets_test[:, item])  # dbg
        if targets[argmax_idx, item] == 1:
            correct_predictions += 1
    return correct_predictions / np.shape(predictions)[1]

def one_hot(targets):
    return np.asmatrix(np.eye(np.max(targets) + 1)[targets]).T

def plot(epochs, loss_train, loss_val):
    plt.plot(epochs, loss_train)
    plt.plot(epochs, loss_val, color="orange")
    plt.legend(["Training Loss", "Validation Loss"])
    plt.xlabel("Epochs")
    plt.ylabel("Loss")
    plt.grid(True)
    plt.show()

class NeuralNetwork:

    def __init__(self):
        self.layers = []

    def add_layer(self, layer):
        self.layers.append(layer)

    def build(self):
        for i, layer in enumerate(self.layers):
            if i == 0:
                layer.type = "input"
            else:
                layer.type = "output" if i == len(self.layers) - 1 else "hidden"
                layer.configure(self.layers[i - 1].neurons)

    def fit(self, X_train, targets_train, X_val, targets_val, max_epochs=50):
        e_loss_train = []
        e_loss_val = []

        # Getting the minimum loss on validation set
        predictions_val = self.predict(X_val)
        min_loss_val = cross_entropy(predictions_val, targets_val)

        best_net = self  # net which minimize validation loss
        best_epoch = 0  # epoch where the validation loss is minimum

        # batch mode
        for epoch in range(max_epochs):
            predictions_train = self.predict(X_train)
            self.back_prop(targets_train, cross_entropy)
            self.learning_rule(l_rate=0.000001, momentum=0.9)
            loss_train = cross_entropy(predictions_train, targets_train)
            e_loss_train.append(loss_train)

            # Validation
            predictions_val = self.predict(X_val)
            loss_val = cross_entropy(predictions_val, targets_val)
            e_loss_val.append(loss_val)

            print("E(%d) on TrS is:" % epoch, loss_train, " on VS is:", loss_val, " Accuracy:",
                  accuracy_score(targets_val, predictions_val) * 100, "%")

            if loss_val < min_loss_val:
                min_loss_val = loss_val
                best_epoch = epoch
                best_net = self

        print("Validation loss is minimum at epoch:", best_epoch)

        plot(np.arange(max_epochs), e_loss_train, e_loss_val)

        return best_net

    # Matrix of predictions where the i-th column corresponds to the i-th item
    def predict(self, dataset):
        z = dataset.T
        for layer in self.layers:
            z = layer.forward_prop_step(z)
        return z

    def back_prop(self, target, loss):
        for i, layer in enumerate(self.layers[:0:-1]):
            next_layer = self.layers[-i]
            prev_layer = self.layers[-i - 2]
            layer.back_prop_step(next_layer, prev_layer, target, loss)

    def learning_rule(self, l_rate, momentum):
        # Momentum GD
        for layer in [layer for layer in self.layers if layer.type != "input"]:
            layer.update_weights(l_rate, momentum)
            layer.update_bias(l_rate, momentum)

class Layer:

    def __init__(self, neurons, type=None, activation=None):
        self.dE_dW = None  # derivatives dE/dW where W is the weights matrix
        self.dE_db = None  # derivatives dE/db where b is the bias
        self.dact_a = None  # derivative of the activation function
        self.out = None  # layer output
        self.weights = None  # input weights
        self.bias = None  # layer bias
        self.w_sum = None  # weighted_sum
        self.neurons = neurons  # number of neurons
        self.type = type  # input, hidden or output
        self.activation = activation  # activation function
        self.deltas = None  # for back-prop

    def configure(self, prev_layer_neurons):
        self.set_activation()
        self.weights = np.asmatrix(np.random.normal(-0.1, 0.02, (self.neurons, prev_layer_neurons)))
        self.bias = np.asmatrix(np.random.normal(-0.1, 0.02, self.neurons)).T  # vettore colonna

    def set_activation(self):
        if self.activation is None:
            if self.type == "hidden":
                self.activation = sigmoid
            elif self.type == "output":
                self.activation = identity

    def forward_prop_step(self, z):
        if self.type == "input":
            self.out = z
        else:
            self.w_sum = np.dot(self.weights, z) + self.bias
            self.out = self.activation(self.w_sum)
        return self.out

    def back_prop_step(self, next_layer, prev_layer, target, local_loss):
        if self.type == "output":
            self.dact_a = self.activation(self.w_sum, derivative=True)
            self.deltas = np.multiply(self.dact_a,
                                      local_loss(self.out, target, derivative=True))
        else:
            self.dact_a = self.activation(self.w_sum, derivative=True)  # (m,batch_size)
            self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weights.T, next_layer.deltas))

        self.dE_dW = self.deltas * prev_layer.out.T

        self.dE_db = np.sum(self.deltas, axis=1)

    def update_weights(self, l_rate, momentum):
        # Momentum GD
        self.weights = self.weights - l_rate * self.dE_dW
        self.weights = -l_rate * self.dE_dW + momentum * self.weights

    def update_bias(self, l_rate, momentum):
        # Momentum GD
        self.bias = self.bias - l_rate * self.dE_db
        self.bias = -l_rate * self.dE_db + momentum * self.bias

if __name__ == '__main__':
    mndata = MNIST(path="data", return_type="numpy", mode="randomly_binarized")
    X_train, targets_train = mndata.load_training()  # 60.000 images, 28*28 features
    X_val, targets_val = mndata.load_testing()  # 10.000 images, 28*28 features

    # Getting the test set splitting the validation set in two equal parts
    # Validation set size decreases from 10.000 to 5000 (of course)
    X_val, X_test = np.split(X_val, 2)  # 5000 images, 28*28 features
    targets_val, targets_test = np.split(targets_val, 2)

    targets_train = one_hot(targets_train)
    targets_val = one_hot(targets_val)
    targets_test = one_hot(targets_test)

    net = NeuralNetwork()
    d = np.shape(X_train)[1]  # number of features, 28*28
    c = np.shape(targets_train)[0]  # number of classes, 10

    # Shallow network with 1 hidden neuron
    # That is 784, 1, 10
    for m in (d, 1, c):
        layer = Layer(m)
        net.add_layer(layer)

    net.build()

    best_net = net.fit(X_train, targets_train, X_val, targets_val, max_epochs=50)

    predictions = best_net.predict(X_test)
    print("Accuracy score on test set is:", accuracy_score(targets_test, predictions) * 100, "%")

Note that:

The performance are similar even though I set 5 hidden neurons
instead of 1
Accuracy on training set is very low as well (around 11 %)

5 nodes is basically nothing, have you tried with 500? If the accuracy remain the same then I would start worrying about the code implementation.

@EdoardoGuerriero accuracy is low

@EdoardoGuerriero I would disagree with the fact that the number of nodes is not enough. According to the original mnist paper a naiive linear classifier got an error rate of roughly 12%. Note that error rate is different than accuracy. Also if accuracy is 11%, then that means it's basically picking a number at random.

Any solution? @juicedatom


text:
How to normalizing various elements of the reward function?
Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,s_2]$ and $B$ can represent the phenomenon that two consecutive actions shouldn't be too far apart $|a_t - a_{t+1}| < \epsilon$, for some small $\epsilon$. A trivial reward mechanism can be:
step(prev_a, a,epsilon, min_s, max_s):
    s = env(action)
    if not min_s < s < max_s:
        r -= s
    if ||a-prev_a|| > epsilon:
        r -= a

As $A$ and $B$ are from different worlds (different physical units, if you will), they both have different ranges. For example, a state $s$ may obtain values that are at most $10$, though an action $a$ may have larger values like $100$. Hence, penalizing by subtracting the state or the action from the current reward may lead to the preference of the agent to only make sure the action condition is set, as this one translates to more future rewards.
How can this issue be addressed? I assume some normalization should be added, though I'm not quite sure how.
Any ideas?

There are several ways to normalize various elements of the reward function. One way would be to divide all values by the largest value in the set. This would ensure that all values are between $0$ and $1$. Another way would be to subtract the mean from all values and then divide it by the standard deviation. This would ensure that all values have a mean of $0$ and a standard deviation of $1$.
A trivial reward function for this scenario would be to simply assign a negative value to $R$ whenever either $A$ or $B$ occurs. However, this may not be the most effective way to penalize the behavior. A more effective approach might be to weigh the penalties differ depending on how severe the violation is. For example, if $A$ occurs, the penalty could be $-1$, but if $B$ occurs, the penalty could be $-10$. This would ensure that more severe violations are penalized more heavily than less severe violations.
This reward function appears to be penalizing the agent for taking actions that result in the state not being within some boundary, as well as for taking actions that are too far apart from the previous action.
One way to avoid this issue would be to normalize the values of $A$ and $B$ before subtracting them from the current reward. This would ensure that both $A$ and $B$ have an equal impact on the reward function, regardless of their range.


text:
Training loss is decreasing very slowly while learning MNIST database
I am developing my ANN from scratch which is supposed to classify MNIST database of handwritten digits (0-9). My feed-forward fully connected ANN has to be composed of:

One input layer, with 28x28 = 784 nodes (that is, features of each image)
One hidden layer, with any number of neurons (shallow network)
One output layer, with 10 nodes (one for each predicted class)

and has to compute gradient w.r.t. weights and bias thanks to backpropagation algorithm and, finally, it should learn exploiting gradient descent with momentum algorithm.
The loss function is: cross_entropy on "softmaxed" network's outputs, since the task is about classification.
Each hidden neuron is activated by the same activation function, I've chosen the sigmoid; meanwhile the output's neurons are activated by the identity function.
The dataset has been divided into:

60.000 training pairs (image, label) - for the training
5000 validation pairs (image, label) - for evaluation and select the network which minimize the validation loss
5000 testing pairs (image, label) - for testing the model picked using new metrics such as accuracy

The data has been shuffled invoking sklearn.utils.shuffle method.
These are my net's performance about training loss, validation loss and validation accuracy:
E(0) on TrS is: 798288.7537714319  on VS is: 54096.50409967187  Accuracy: 12.1 %
E(1) on TrS is: 798261.8584179751  on VS is: 54097.23663558976  Accuracy: 12.1 %
...
E(8) on TrS is: 798252.1191081362  on VS is: 54095.5016235736  Accuracy: 12.1 %
...
E(17) on TrS is: 798165.2674011206  on VS is: 54087.2823473459  Accuracy: 12.8 %
E(18) on TrS is: 798155.0888987815  on VS is: 54086.454077456074  Accuracy: 13.22 %
...
E(32) on TrS is: 798042.8283810444  on VS is: 54076.35518400717  Accuracy: 19.0 %
E(33) on TrS is: 798033.2512910366  on VS is: 54075.482037626025  Accuracy: 19.36 %
E(34) on TrS is: 798023.431899881  on VS is: 54074.591145985265  Accuracy: 19.64 %
E(35) on TrS is: 798013.4023181734  on VS is: 54073.685418577166  Accuracy: 19.759999999999998 %
E(36) on TrS is: 798003.1960815473  on VS is: 54072.76783050559  Accuracy: 20.080000000000002 %
...
E(47) on TrS is: 797888.8213232228  on VS is: 54062.70342708315  Accuracy: 21.22 %
E(48) on TrS is: 797879.005388998  on VS is: 54061.854566864626  Accuracy: 21.240000000000002 %
E(49) on TrS is: 797869.3890292909  on VS is: 54061.02482142968  Accuracy: 21.26 %
Validation loss is minimum at epoch: 49

(note that training loss seems to be much higher than validation loss and for this reason they seems to be straight horizontal lines in the first image)
This is my code:
import numpy as np
from scipy.special import expit
from matplotlib import pyplot as plt
from mnist.loader import MNIST

def relu(a, derivative=False):
    f_a = np.maximum(0, a)
    if derivative:
        return (a > 0) * 1
    return f_a

def softmax(y):
    e_y = np.exp(y - np.max(y))
    return e_y / e_y.sum()

def cross_entropy(y, t, derivative=False, post_process=True):
    epsilon = 10 ** -308
    if post_process:
        if derivative:
            return y - t
        sm = softmax(y)
        sm = np.clip(sm, epsilon, 1 - epsilon)  # avoids log(0)
        return -np.sum(np.sum(np.multiply(t, np.log(sm)), axis=0))

def sigmoid(a, derivative=False):
    f_a = expit(a)
    if derivative:
        return np.multiply(f_a, (1 - f_a))
    return f_a

def identity(a, derivative=False):
    f_a = a
    if derivative:
        return np.ones(np.shape(a))
    return f_a

def accuracy_score(targets, predictions):
    correct_predictions = 0
    for item in range(np.shape(predictions)[1]):
        argmax_idx = np.argmax(predictions[:, item])
        if targets[argmax_idx, item] == 1:
            correct_predictions += 1
    return correct_predictions / np.shape(predictions)[1]

def one_hot(targets):
    return np.asmatrix(np.eye(10)[targets]).T

def plot(epochs, loss_train, loss_val):
    plt.plot(epochs, loss_train)
    plt.plot(epochs, loss_val, color="orange")
    plt.legend(["Training Loss", "Validation Loss"])
    plt.xlabel("Epochs")
    plt.ylabel("Loss")
    plt.grid(True)
    plt.show()

class NeuralNetwork:

    def __init__(self):
        self.layers = []

    def add_layer(self, layer):
        self.layers.append(layer)

    def build(self):
        for i, layer in enumerate(self.layers):
            if i == 0:
                layer.type = "input"
            else:
                layer.type = "output" if i == len(self.layers) - 1 else "hidden"
                layer.configure(self.layers[i - 1].neurons)

    def fit(self, X_train, targets_train, X_val, targets_val, max_epochs=50):
        e_loss_train = []
        e_loss_val = []

        # Getting the minimum loss on validation set
        predictions_val = self.predict(X_val)
        min_loss_val = cross_entropy(predictions_val, targets_val)

        best_net = self  # net which minimize validation loss
        best_epoch = 0  # epoch where the validation loss is minimum

        # batch mode
        for epoch in range(max_epochs):
            predictions_train = self.predict(X_train)
            self.back_prop(targets_train, cross_entropy)
            self.learning_rule(l_rate=0.00001, momentum=0.9)
            loss_train = cross_entropy(predictions_train, targets_train)
            e_loss_train.append(loss_train)

            # Validation
            predictions_val = self.predict(X_val)
            loss_val = cross_entropy(predictions_val, targets_val)
            e_loss_val.append(loss_val)

            print("E(%d) on TrS is:" % epoch, loss_train, " on VS is:", loss_val, " Accuracy:",
                  accuracy_score(targets_val, predictions_val) * 100, "%")

            if loss_val < min_loss_val:
                min_loss_val = loss_val
                best_epoch = epoch
                best_net = self

        plot(np.arange(max_epochs), e_loss_train, e_loss_val)

        return best_net

    # Matrix of predictions where the i-th column corresponds to the i-th item
    def predict(self, dataset):
        z = dataset.T
        for layer in self.layers:
            z = layer.forward_prop_step(z)
        return z

    def back_prop(self, target, loss):
        for i, layer in enumerate(self.layers[:0:-1]):
            next_layer = self.layers[-i]
            prev_layer = self.layers[-i - 2]
            layer.back_prop_step(next_layer, prev_layer, target, loss)

    def learning_rule(self, l_rate, momentum):
        # Momentum GD
        for layer in [layer for layer in self.layers if layer.type != "input"]:
            layer.update_weights(l_rate, momentum)
            layer.update_bias(l_rate, momentum)

class Layer:

    def __init__(self, neurons, type=None, activation=None):
        self.dE_dW = None  # derivatives dE/dW where W is the weights matrix
        self.dE_db = None  # derivatives dE/db where b is the bias
        self.dact_a = None  # derivative of the activation function
        self.out = None  # layer output
        self.weights = None  # input weights
        self.bias = None  # layer bias
        self.w_sum = None  # weighted_sum
        self.neurons = neurons  # number of neurons
        self.type = type  # input, hidden or output
        self.activation = activation  # activation function
        self.deltas = None  # for back-prop

    def configure(self, prev_layer_neurons):
        self.set_activation()
        self.weights = np.asmatrix(np.random.normal(-0.1, 0.02, (self.neurons, prev_layer_neurons)))
        self.bias = np.asmatrix(np.random.normal(-0.1, 0.02, self.neurons)).T

    def set_activation(self):
        if self.activation is None:
            if self.type == "hidden":
                self.activation = sigmoid
            elif self.type == "output":
                self.activation = identity  # will be softmax in cross entropy calculation

    def forward_prop_step(self, z):
        if self.type == "input":
            self.out = z
        else:
            self.w_sum = np.dot(self.weights, z) + self.bias
            self.out = self.activation(self.w_sum)
        return self.out

    def back_prop_step(self, next_layer, prev_layer, target, local_loss):
        if self.type == "output":
            self.dact_a = self.activation(self.w_sum, derivative=True)
            self.deltas = np.multiply(self.dact_a,
                                      local_loss(self.out, target, derivative=True))
        else:
            self.dact_a = self.activation(self.w_sum, derivative=True)  # (m,batch_size)
            self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weights.T, next_layer.deltas))

        self.dE_dW = self.deltas * prev_layer.out.T

        self.dE_db = np.sum(self.deltas, axis=1)

    def update_weights(self, l_rate, momentum):
        # Momentum GD
        self.weights = self.weights - l_rate * self.dE_dW
        self.weights = -l_rate * self.dE_dW + momentum * self.weights

    def update_bias(self, l_rate, momentum):
        # Momentum GD
        self.bias = self.bias - l_rate * self.dE_db
        self.bias = -l_rate * self.dE_db + momentum * self.bias

if __name__ == '__main__':
    mndata = MNIST(path="data", return_type="numpy", mode="randomly_binarized")
    X_train, targets_train = mndata.load_training()  # 60.000 images, 28*28 features
    X_val, targets_val = mndata.load_testing()  # 10.000 images, 28*28 features

    X_train, targets_train = shuffle(X_train, targets_train.T)
    X_val, targets_val = shuffle(X_val, targets_val.T)

    # Getting the test set splitting the validation set in two equal parts
    # Validation set size decreases from 10.000 to 5000 (of course)
    X_val, X_test = np.split(X_val, 2)  # 5000 images, 28*28 features
    targets_val, targets_test = np.split(targets_val, 2)
    X_test, targets_test = shuffle(X_test, targets_test.T)

    targets_train = one_hot(targets_train)
    targets_val = one_hot(targets_val)
    targets_test = one_hot(targets_test)

    net = NeuralNetwork()
    d = np.shape(X_train)[1]  # number of features, 28*28
    c = np.shape(targets_train)[0]  # number of classes, 10

    # Shallow network with 1 hidden neuron
    # That is 784, 1, 10
    for m in (d, 1, c):
        layer = Layer(m)
        net.add_layer(layer)

    net.build()

    best_net = net.fit(X_train, targets_train, X_val, targets_val, max_epochs=50)

What I have done:

Set 5 instead of 1 hidden neuron
Add many hidden layers
Decrease/increase learning rate (l_rate) value
Decrease/increase momentum (and set it to 0)
Replace sigmoid with relu

but there still is the problem.

I wouldn't use epsilon smaller than 1e-10.  Why is this pure-python and not something like Tensorflow or PyTorch?  There is an outstandingly large difference between running this in a compiled and optimized framework that runs on a GPU, and running it hand-coded in an interpreted language that runs on the CPU.  Compiled vs. interpreted I've seen yield 1000x speedup, and cpu compiled vs. gpu compiled i've seen yield a 25x speedup.  My thought: check your error, because I'm not sure you are getting categorical cross-entropy right? https://www.tensorflow.org/datasets/keras_example

@EngrStudent Because the assignment is about neural network from scratch: no PyTorch and other frameworks.

The way to do this is build up a step at a time.
Can you build a classifier that assigns whether a number is positive or negative?  Make a single input, a single hidden node with random weights, and a single sigmoid output.  The output attempts to answer the question "is it positive?".
Should be a subset of code you already have, quick to make.  Will let you know that your stuff works.  Then you can work on applying it to larger inputs.
Also, you don't submit the package code or results alone, but use them as a sanity check.  If you can't make it work there, you can't make it work.

which version of 'mnist' are you using because version '0.2.2' in python 3.8 doesn't seem to have "loader".


text:
How large of a neural network can you run on a modern graphics card?
I'm trying to get a general idea of what is a typical size for neural networks that run on a single graphics card.
What I mean for instance is if I have data with N total inputs, and I sent it through a network of M hidden layers with $N_m$ nodes, and Y outputs. What is the maximum size that a modern GPU can handle running in a reasonable timeframe?
I'm just looking for a sort of ballpark understanding of what values are typically used for node number and hidden layer number.

"Run" means different things.  "Large" means different things.  "Modern" is as Yoda describes the future: "always in motion".  "Reasonable timeframe" is qualitative.  That said, I have run training on ~50M parameter models in under 48 hours requiring 100k epochs.  If you can convert to lower precision, that can help.  If you use a good framework, that can help.  If your training set isn't too big, that can help.

TL;DR AT BOTTOM
Depending on your subjective opinion on what a "modern" GPU is, your answer will vary.
The recent NVIDIA 3000's and 4000's are considered to be "the best" but I cannot find a objective source that gives actual statistics, so this should be taken with a grain of salt.

If I have data with N total inputs, and I sent it through a network of M hidden layers with Nm nodes, and Y outputs. What is the maximum size that a modern GPU can handle running in a reasonable timeframe?

This depends on your AI model. Are you using Tensorflow or the hundreds of other open source projects or are you building from scratch? All AI models try to get the job done, only some do it in a fast timeframe as the main goal.
In my subjective experience with my "modern" laptop (depending on the definition here) a text generating AI can have up to two million nodes without the laptop being laggy. I have seen much more such as ten million nodes on a "normal" (again, subjective!) laptop. Image processing  and more data-full tasks are where millions of nodes are impractical. You would ideally need more than a "modern laptop." I don't play in this realm and I can't find reliable stats, but for sure you could do more than ten million.
TL;DR: Depends on your idea of a "modern device." I have seen up to ten million nodes on a laptop for text related AI, but no stats exist for GPU's all accross the board.
Benchmark next time you use the AI model in question to get a good idea of what you can handle.

There are some benchmarks and additional information in these links: https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/#GPU_Deep_Learning_Performance  https://www.run.ai/guides/gpu-deep-learning/best-gpu-for-deep-learning

@Bapcap, thank you for your answer this is what I was looking for.
So for this text AI that does two million nodes that you mentioned, is that two million total nodes? Any idea what the dimensions are for the layers?  (how many nodes per layer and how many inputs)

@StevenSagona I cannot remember nodes per layer as it differs depending on your model/software. But yes, two million total nodes


text:
Is Relativistic GAN better than WGAN-GP?
I am currently reading the ESRGAN paper and I noticed that they have used Relativistic GAN for training discriminator. So, is it because Relativistic GAN leads to better results than WGAN-GP?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

For anyone looking for answer, as explained in Paper: The relativistic discriminator: a key element missing from standard GAN!,
Yes,  Standard RaGAN with gradient penalty generate data of better quality than WGAN-GP while only requiring a single discriminator update per generator update (reducing the time taken for reaching the state-of-the-art by 400%).The  images generated by RaGAN are of significantly better quality than the ones generated by WGAN-GP and SGAN with spectral normalization.


text:
How to discover/approximate the causations/correlations between multiple time-series and related open source libraries?
I have the following time-series data with two value columns.
(t: time, v1: time-series values 1, v2: time-series values 2)
 t | v1 | v2
---+----+----
 1 |  1 |  0
 2 |  2 |  2
 3 |  3 |  4
 4 |  3 |  6
 5 |  3 |  6
 6 |  4 |  6
 7 |  5 |  8
(7 rows)

I am trying to discover (or approximate) the correlation between the $v1$ and $v2$, and use that approximation for the next step predictions.
Please note, the most obvious correlation is $v2(t)=2.v1(t-1)$.
My question is, what are the algorithms to employ for such approximations and are there any open source implementations of those algorithms for SQL/python/javascript?

The algorithm which is best to implement this problem is the hidden Markov model because hidden Markov model is best for generative sequence and openMarkov
and also you can implement this in pytorch,jax,tenserflow(all are open source).

I believe you may want to look at Autoregressive Models (AR) like ARMA models. Python has implementations of all AR models. Python statsmodels have all the necessary tools you may require.

There are a number of algorithms that could be used for such approximations, including linear regression to learn the relationship between $v1$ and $v2$. You could then use that relationship to predict $v2$ for new values of $v1$., k-nearest neighbors, and support vector machines. There are also a number of open-source implementations of these algorithms available, including scikit-learn, Weka, and TensorFlow.
there are a few ways to go about this:

Look for existing research on the topic. This will give you an idea of which methods have been used successfully in the past and what kind of results have been achieved.

Use a data mining tool such as Weka or R to explore the relationships between the time-series data. This will allow you to try out different methods and see which ones work best on your data.

Use a machine learning algorithm to learn the relationships between the time-series data. This can be more challenging, but if you have a large amount of data it may be the best option.

I am currently working on anomaly detection for time series.
Recently, I've read about so-called "dynamic time series" (dtw). This method finds similarity for time series. As far as I read, it is nicely applicable for shifted time series (your time series is shifted and scaled).
I hope that was helpful.


text:
Learning curve converges with huge errors
I am training an auto-encoder over $10^4$ epochs. I get a converging learning curve. However the error at the last stages stays huge $\sim10^{15}$. What does this mean? does it mean that my auto-encoder isn't trained properly? Is it a question of the error function?
The error function I am using is: $\mathcal{L} = \|X-\hat X\|_F^2$ where $X\in\mathbb{R}^{n\times d}$ is the original features (with $n$ the number of rows and $d$ the number of features), $\hat X\in\mathbb{R}^{n\times d}$ is the reconstructed features (by the auto-encoder) and $\|.\|_F^2$ refers to the Frobenius Norm.

It could mean anything. Start tweaking parts of the network and see the effect that has on the learning curve.

There are a few potential explanations for this. One possibility is that your model is overfitting to the training data. This means that it is memorizing the training data instead of learning to generalize from it. Another possibility is that your error function is not well-suited to the task at hand. This could be due to the fact that it is not a convex function, or it could be because it is not differentiable. Finally, it is also possible that your learning rate is too high, causing the model to diverge instead of converge.
Yes, the error function you are using is known as the Frobenius Norm. It is a measure of the distance between two matrices. In this case, it is measuring the distance between the original features and the reconstructed features. The fact that the error is staying huge at the last stages could be due to overfitting or a poor learning rate, as mentioned before.


text:
possible to combine multiple labeled objects as one object?
So I have labeled the entire skeletal muscles in detail. For example instead of just labeling shoulders I have labeled:

Rear Delt
Middle Delt
Front Delt

but now you want all of the delts to be combined as one object, shoulder. Is there anyway to take all of the previous labeled images of laptops and combine them as one object?
The why:
We have a lot of objects that are close to one another and it's making the model pretty big and complex. I want to drastically reduce the number of objects in the model but I don't want to lose over 15,000 labeled instance segmentation images.
thank you in advance

There are probably many ways to combine classes.  Here is my suggestion:

Figure out the mask pixel values for each of the classes you want to pool (e.g. Rear Delt = 1, Middle Delt = 2, front delt = 3)
Next choose a "recode" pixel value of the pooled class that is not used for any other class (e.g., make all of them 1).
Loop through each mask file to (re)assign the "recode" value to a pixel if the pixel value is equal to the value determined in #1 above.

As an aside, you should confirm that your recoding worked by visualizing the masks before and after the recode.
Also, don't save the new mask file over your original mask file.  You never know when you may need the original again.
Finally, there is a SO post on vectorizing this process.


text:
Is the decoder in a transformer Seq2Seq model non parallelizable?
From my understanding, seq2seq models work by first computing a representation of the input sequence, and feeding this to the decoder. The decoder then predicts each token in the output sequence in an autoregressive manner. In this sense, it's limited to processing one time step at a time, as the next token in the sequence depends on the previous (non parallelizable). This is different from the encoder, as it's able to process many time steps in parallel. Is my understanding correct?

Yes, the decoder in a transformer Seq2Seq model is non-parallelizable. Yes, that is correct.

During training, the decoder can be trained in parallel (and that's one of its advantage over LSTM) :
You input <s> I love you, and the decoder learns to produce I love you </s>. For each token, the model learns to predict the next token (for example given <s> the model learns to predict I). This is possible thanks to attention.

But at inference time, you are right : the model has to predict the first token, then put this token in the decoder input and predict the second, etc...
So yes, at inference time a decoder is much slower (because we go through the whole decoder several times : one time for each token) than an encoder (which do a single forward pass).


text:
Using a pre-trained model to generate labels to data to then train a model on
I'm trying to set up a pipeline for my ML models to automatically re-train themselves whenever concept drift occurs to recalibrate to the new output distributions. However, I can't get ground-truth from my data without manual labeling, and I want an automated MLOps pipeline, so that no human-in-the-loop labeling is required.
I was noticing that using zero-shot learning using BART was actually giving pretty impressive results with my label set on sample data I want to classify on.
Suppose I were to use BART to manually label data, say, if it scored my label with a score of 0.95 or better, and then just used that labeled data to train my models.
Is this "cheating"? Am I trying to cut a corner here that can't be cut? Will this create a meaningful model for my text data, or just cause my trained model to try to become a crude version of the BART model that provided the labels in the first place, and therefore just be worse than strictly using the BART model?

Seems possible: Google search for: automatic labeling artificial intelligence BART finds: https://github.com/CristianViorelPopa/BART-TL-topic-label-generation - leaving BART out of the search terms finds quite a few articles.

If using BART is already giving you good results, why do you need a new model?
Not a rhetorical question. You might have good reasons for that. Training a model with less parameters optimized only on few classes is an example. But if you can't came up with good reasons then the answer is pretty straight forward: just use BART. Reason being that a model trained on another model annotations without any human in the loop will never perform better than the model used to automatically train the data. The first reason being error propagation, like in any other area of empirical science. And on a more technical level you can't really use tricks like using only predictions with a high confidence scores, cause deep learning models are basically never calibrated, hence you can't trust them and a human would still have to check them manually.

I see. The motivation for training a model from the BART predictions was so I could avoid concept drift, by constantly re-training the model based on new data BART predicts on. How do I deal with concept drift using BART, a pre-trained model?


text:
Loss Function for Binary Classification with Multiple Correct Choices
I have a binary classification problem, where there are multiple correct predictions, however, I would consider the prediction to be correct if the highest confidence prediction of a 1 is correct.
I have had limited success with a CNN using the Hinge loss function. I think the reason is that, this loss function is calculating the loss between every element in y_true and y_pred.
Example:
I want my model to confidently predict one of these cells as a 1. There probably isn't enough information in the input data to accurate predict every 1, especially in the areas far from input cells with a positive value, therefore I would be happy if the model is conservative and predicts a 0 where it is unsure.
input_data

y_pred:

I believe the model is being optimized to classify as many cells correctly as possible.
It's not possible to correctly predict the cell in the bottom right corner in this example. Therefore the model should take the conservative approach and predict a 0.
y_true

A perfect loss function in my mind would:

Heavily penalize the case where (y_true, y_pred) = (0,1) (false
positive)

Heavily reward the case where (y_true, y_pred) = (1,1) (true
positive)

Moderately penalize the cases where (y_true, y_pred) = (1,0) (false
negative)

Moderately reward the cases where (y_true, y_pred) = (0,0) (true
negative)

Or
Calculate the loss only on the cell with the highest value in y_pred
Can anybody suggest a suitable loss function?
Let me know if I have omitted any required detail


text:
Semi-supervised learning algorithms creating redundant data
If I'm generating pseudo-labels that I'm confident are correct for my dataset due to high confidence scores or something else, how can I expect that the new data I'm labeling won't be redundant? To my knowledge, if a datapoint is labeled automatically, unless novel data is generated like in this paper I don't see how the data that can be labeled will be particularly useful, as it will admit no new patterns to learn from, as otherwise the data wouldn't be labeled with a high confidence. Is the point here to make sure the labeled dataset you start with is sufficiently large that it covers all the kinds of patterns you expect from the data for the model to need to be able to identify?

If you are confident that your pseudo-labels are correct, then you can expect that the new data you label will not be redundant. however, If you want to avoid having redundant data in your labeled dataset, then you need to make sure that it is large enough to cover all the relevant patterns.
One way to avoid creating redundant data when using data augmentation is to use a technique called transfer learning. This involves training a model on one dataset and then transferring the learned knowledge to another related dataset. This can be done by fine-tuning the model on the new dataset or by using the model to extract features from the new dataset that can be used to train a new model. Transfer learning can be used to reduce the amount of data needed to train a new model, and can also improve the performance of the new model.


text:
Order of features learned by DNNs during training?
I'm looking for papers probing into the question of what features get learned when (or equivalently what subproblems get "solved" when) during the training process. For example, a paper showing that a Convnet trained on MNIST learns to distinguish 0 from 1 before it learns to distinguish 0 from 8.
The one example I can think of off the top of my head is the Grokking paper, but that's looking at a slightly different (and less intuitive) phenomenon. Thanks!

The order in which features are learned by DNNs during training is typically random & depends on a number of factors:

Including the specific architecture of the DNN
The type of data being used for training
& the optimization algorithm being used

However, it is known that features are learned in a hierarchical manner, with lower-level(simpler) features being learned first, followed by higher-level(complex) features.

There is some evidence that certain features are learned early on in the training process, while others are learned later. For example, a study by Karpathy et al. (2014) found that a convolutional neural network (CNN) trained on the MNIST dataset tends to learn to distinguish between the digits 0 and 1 before it learns to distinguish between the digits 0 and 8. However, it is not clear how generalizable these findings are to other tasks and datasets.
A good general resource on how different features are learned at different stages in neural networks is the Deep Learning book by Goodfellow, Bengio, and Courville. In particular, Chapter-6 (Feature Representations) and Chapter-7 (Neural Networks) cover this topic in detail.
Artificial Intelligence: A Modern Approach by Russell and Norvig (2010) covers this topic in Chapter 18 on Learning Complexity. The chapter starts with a discussion of the concept of concept learning and the ways in which different concepts can be learned. It then covers the question of how well different concepts can be learned, and how this affects the overall learning process.
One paper that explores this question is "Visualizing and Understanding Convolutional Networks" by Matthew D. Zeiler and Rob Fergus.

In this paper, the authors visualize the features learned by a convolutional neural network as it is trained on different datasets. They find that the network gradually learns increasingly complex features, starting with simple edge detectors and then moving on to more complex shapes and patterns.

Another paper Why does deep and cheap learning work so well? by Henry W. Lin, Max Tegmark, and David Rolnick

In this paper, the author tries to answer the question of why deep learning works so well, despite being "cheap" (in terms of the amount of data and computation required). He argues that deep learning is able to learn a wide range of features, from low-level features (e.g. edge detectors) to high-level features (e.g. object detectors), and that this enables it to achieve better performance than shallower methods.

This is a really thorough wide-ranging response, and I really appreciate the time you spent crafting it. I've not going to accept it as the answer just yet though, because I'm particularly interested in empirical investigations of what content is learned at different points in the training process, as opposed to what concepts are represented in which layers. This could be relative to image classification, language generation, whatever, I'm just looking for papers that have attempted to quantify it in any way.

Thanks for your appreciation, answer acceptance is secondary... not an issue :) just trying to help...here is another paper  "The Layer-wise Relevance Propagation Toolbox for Artificial Neural Networks(https://www.researchgate.net/publication/309201826_The_Layer-wise_Relevance_Propagation_Toolbox_for_Artificial_Neural_Networks) paper discuss about LRP method of decomposing the output of an ANN into the contributions of each input feature.

There are a few different ways to investigate what content is learned at different points in the training process. One approach is to look at the weights of the neurons in the different layers of the network. This can give you some information about which concepts are represented in which layers. Another approach is to look at the output of the network at different points in the training process. This can give you some information about what content is learned at different points in the training process.


text:
Are custom tokens better than punctuation pseudo-tokens for LLMs?
I've seen two approaches for introducing custom tokens for transfer learning with large language models like Bert or GPT3. Some approaches introduce new tokens into the vocabulary and learn embeddings from scratch. This is the "traditional" approach. However, I've seen other papers that imitate custom tokens with the use of punctuation, e.g. "<custom-token>". In this case the model is not learning any new tokens, but is learning to connect subword tokens and punctation already in its vocabulary. I think this approach is often used with GPT3, as the closed API prevents learning new tokens from-scratch.
Has any research benchmarked whether one approach is better than another, when both options are available?


text:
Why training the same model on the same data can be slower on better card?
Can someone explain why training CNN model (in my case DenseNet201) on the same data, and the same data processing pipeline can be slower on better GPU (RTX3090) than worse one (RTX3060), with the same other parameters (exactly same PC just with new GPU)?
In both cases I used same batch size and other settings. The only way to make training faster on 3090 was to actually increase the batch size, which was too big for 3060. But I still don't understand why the same training params wouldn't produce better results.
Even though big part of the training is reading data from disk and data augmentation (albumentations in my case) it's still the same setup, so even if GPU work is smaller part of one entire epoch, it still should be a bit faster, right?

Every training will be slightly different because of the statistical matter of neural networks.
The question is, how much is your better?
Then, newer does not imperatively mean better. It means more hardware and computational power, but not necessarily how it is used for specific tasks.
Maybe, the 3090 can't leverage its power in this specif task properly.

But isn't image classification with CNN model a pretty standard task in DL? From your answer I reckon that if a person or a company wants to increase their computational power to speed up their work, they will have to buy couple of different GPUs and test each of their different task's models on each of these? Is this really how it works? Am I suppose to have 3 different PCs and for each task always find the best one? Buying card that has more computational power isn't enough anymore?

@GKozinski I can't really say more. I have no insight in your set-up etc. I just gave you some general hints. Maybe check your drivers, or the task is so trivial that the higher power of the 3090 becomes irrelevant.


text:
Can I pick up possible calendar events from a text file
I am new to this community and not even too sure if this is the right place for such a question.
I want to know how I could implement (possibly with already made solutions) a system which could take in a (possibly long winded) text file and deduct from there, events and meetings which a user could add (similar to what Siri does when it finds a possible event).
Is there something for this that already exists?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

If you're fine training a model of your own, I would suggest try building a NER prediction model on your data. It would help to start with a certain type of data which follows similar pattern throughout. Once you're confident, you can extend it to many more patterns.
Or if you're lucky and find pre-trained NER models to be able to identify the event-date relationships already, you can just use them out-of-the-box.
Interestingly, I tried some QA models to simply ask the event & time of event given some context, and deepset/roberta-base-squad2 works pretty well for simple examples.

Context: I have a flight to catch at 6 AM tomorrow.
Question: What is the topic of event?
Answer: a flight to catch at 6 AM tomorrow
What is the date of event?
6 AM tomorrow

Context: The next set of features are scheduled to be discussed at meeting on 25th Dec 2022.
Question: What is the topic of event?
Answer: The next set of features
Question: What is the date of event?
Answer: 25th Dec 2022

Of course your data could be very different, but the pre-trained models should be a good starting point for ideas.