text:
Which type of Machine Learning is used in robots?
Which type of Machine Learning is used in robots?
Is it supervised learning or unsupervised learning or Reinforcement learning? Especially the robots that were sent in space?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

There are lots of types of robots. Many don't use machine learning at all (it's actually quite dangerous to use black-box learners on machines that could cause damage if incorrectly operated), and I'm pretty sure you will also find plenty that use each type of ML depending on purpose. Many will use multiple types of ML, for different sub-tasks withinn the robot. One way to narrow this question down would be to ask about a specific type of robot

The general question asks for a general answer. There's the risk that this will lead to many different answers, though, which we want to prevent. I think a good answer to this question would be general and maybe provide 1-2 specific examples. Anyway, yes, some robots use ML, in labs, at least. I am pretty of this because I know people that did research on this. Now, I don't really know about the robots for space. So, what I'd suggest is that you at least narrow down the type of robot, although you're already saying "especially the robots that were sent in space", which might still be too broad

So, please, if you're interested only in robots that were sent into space, edit your question, including the title, to be specific. Otherwise you will get answers for all types of robots and this might be too broad. If you always aks multiple questions, one for each type of robot you have in mind.

Are you interested only in ML applications in robotics or also want to known about the "engineering way" (pretty much non-ML) of doing robotics?

I'll just vote to close this because I think this will generate a lot of different answers.

Robotics primarily uses RL and supervised learning. To give examples:
Supervised Learning is used for vision and perception
Reinforcement Learning is used for control
Unsupervised Learning for robotics is less commmon, but I am sure exists.
Source: I work in a robotics lab and use these daily.

Nice reply. Voted


text:
Are there strictly deterministic LLMs?
LLMs are understood to generate non-deterministic outputs.
Are there LLMs out there that are capable to producing deterministic outputs for any given input given fixed parameters (like e.g temperature)?
I heard that llama.cpp - if run on a CPU instead of a GPU appears to generate deterministic outputs.

When LLMs "generate output" they are sampling from a probability distribution. In principle, it should always be possible to get the same samples from a probability distribution. Read more here: https://community.openai.com/t/is-there-a-way-to-set-a-a-random-seed-for-responses-with-temperature-0/4164/3

Any LLM that exists could easily be modified to be deterministic. At the present, they sample from a probability distribution for the next word. It is a trivial change to make, so that instead of sampling from this distribution, they always pick the word deemed most likely.
This has nothing to do with running on a GPU vs CPU: the non-determinism is not a function of hardware but software.
In many LLMs, this tradeoff is governed by a temperature parameter (it is called temperature for historical reasons) that parameterizes how much randomness will be in the LLM.
Reference: LLM Text Generation: Why is Determinism so Hard to Achieve?

I think OP wants deterministic sampling behavior for any temperature setting (including high temperature). Setting a random seed will do the trick.

yes - any idea why (to my knowledge) no LLM offers to define a randomness seed to make it deterministic? IMO this would have plenty of useful applications.

I don't think this is correct. You can set temperature to 0 and even set a seed (if available), thereby always selecting the most likely next token, but that won't get you the same output from GPT or Claude.

Here's a link to some discussions on HN https://news.ycombinator.com/item?id=37446724

LLMs are mathematical functions executed on floating-point calculations.  The software and hardware implementations cause non-deterministic side-effects.
In most cases, these non-deterministic side-effects are desirable to produce outcomes that converge towards better overall performance.
That being said removing the non-deterministic behavior is some cases can be incredibly difficult since this behavior may be buried in some framework.  For example, weight initialization is performed by selecting random small values for regularization in neural network implementations.  This is a desirable effect to prevent vanishing or exploding gradients, but that is not to say that those random values couldn't be explicitly reused in some use case if it was absolutely necessary.
When people talk about using CPU, vs GPU, it typically references computational speed increases but GPUs include a myriad of computational optimizations and some GPU optimizations for picking random numbers do exist and may be used.  There are also floating point anomalies in some computation environments but that should not be attributed towards being "non-deterministic".

To answer your question "No there arent any".
Executing an LLM such as llama.cpp on a CPU may indeed lead to more predictable outcomes due to the way it handles floating-point calculations. To see the difference for yourself, simply obtain a copy of llama.cpp and adjust the temperature and seed parameters as needed, or apply both if relevant, and conduct your tests. The same experiment can be replicated with a model running on a GPU to compare results.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is a pipeline in machine learning?
I have heard the term "pipeline" used in many different contexts. Now I'm trying to bring some clarity to the terminology: What exactly is a "pipeline" in machine learning?

A data pipeline consists of 3 main steps

data collection (e.g. you collect images of cats from different sources)
data transformation (e.g. you make the images all have the same dimensions and maybe you want to convert all of them to grayscale, etc)
data storage (so that you don't lose the data and can later reuse)

In machine learning, you need data to train the models. So, you could adopt a data pipeline, but not necessarily. It depends on your use case. For example, maybe you don't need to collect the data because you can download it from the Internet (although we could consider this download the data collection itself), or maybe you don't need to store it (in a database) because you will use it only once. However, you will probably need to transform it.
Anyway, data pipelines are not specific to machine learning. You can also develop them for data analysis or visualisation (so without training any ML model). There may also be other types of pipelines (e.g. people may refer to the steps they followed to train a certain model as a pipeline - see e.g. how InstructGPT was created here - that's also a pipeline, in the sense that it's a chain of operations, where the output of the previous one is feed to the next one), but data pipelines are definitely the ones I am most familiar with, because I also work with them, and they have a more or less standard definition.
This IBM article provides a more detailed explanation.

A "pipeline" typically refers to a chain of methods where the output of the one is used as the input of another method.
This could be, e.g., a "preprocessing pipeline" where several data cleaning and data preparation steps are sequentially performed.
A more complete "ML Pipeline" could consist of data prepocessing, sampling of the data (e.g. for obtaining folds for cross validation or resampling) and a ML training and testing. Often, such a pipeline also records all used parameters.
Other than that there is no strict definition, as far as I know.

A "pipeline" in machine learning refers to a sequence of data processing and modeling steps that transform raw data into predictions or insights. Pipelines often include stages such as data preprocessing, feature extraction, feature scaling, model training, and model evaluation[1].
You may see in the context of the transformers library, the pipeline class simplifies the process of using pre-trained models for common NLP tasks like text classification, named entity recognition, and text generation. It provides an easy-to-use interface that handles tokenization, input processing, and output decoding in a boxed fashion.
Considering the Pipeline class in scikit-learn, the Pipeline class helps automate and organize machine learning workflows by allowing users to define a sequence of data transformations and models. It simplifies the process of applying multiple preprocessing steps and ensures that the same preprocessing is applied to both training and test data.
It is nothing distant from the definition in the context of generic computing.

is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion.

You might say that ML pipeline is a natural extension of data pipeline, which may have been more known and understood better.

[1] See: https://neptune.ai/blog/ml-pipeline-architecture-design-patterns

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why does model overfitting lead to poor generalization?
If a model overfit to the training data, why does it generalize poorly?
Consider the basic problem of a noisy 2d dataset where I am fitting polynomials. A good model would be a parabola and a line would be underfitting. As I increase the polynomial's power, I end up overfitting more and more to the data. However, why is the data unlikely to follow this overfitting behavior? The only knowledge I have is of this data. It would be reasonable to assume the better the model does on this data, the better it would do on testing dataset.
How do I increase the complexity of my model or training data such that accuracy can only be improved? For example if I have a given model that does generalizes well, how do I keep this good generalization while it fits better to the training data?

I think you're asking different questions here. The answer below answers to your question in the title. Please, consider removing your other questions from this post and ask them in separate posts, if you want specific answers to them. Thank you.

The issue is the generalisation. If your model fits perfectly to your training data, it then depends whether your training data is a good reflection of the actual data you will encounter 'in the wild'. If it was a perfect sample, and no other values ever occur, then there is no problem.
But, as is more likely, the training data is only a small and not 100% representative sample. Now your model fits this data perfectly, but previously unseen (noisy) data is different, and doesn't fit, because your model is too specific.
With classification you want to allow for some variability, because real-world data is never perfect. Your accuracy (of classifying items) might improve if your model is more forgiving (ie less specific).

But the model is fitting to everything, the noise and also the actual data that it should be learning. So why does the noise ruin the generalization?

@JobHunter69 The noise will be random, and thus will mask the 'actual' (idealised) data. But you don't know what is noise and what is the data, so the noise will be treated as part of the data and reduce accuracy. With more data, this effect diminishes.

If the model fits both to the noise and also the data, why does the noise mask the data? The model should still be able to learn the data

The model cannot distinguish between the two.

Note that if you know the function your data was generated from, you do not even need machine learning. For example, if you generated the data with $y = f(x) = x^2$, you already have the function to predict future $y$s given the $x$s, i.e. $f(x) = x^2$.
So, we usually apply machine learning when we don't know the target function, i.e. when you don't know the shape of some function you want to learn, although you can make assumptions about it, which may be wrong. For example, you could assume your function is a parabola $x^2$, but it could also be any other function that may have data in common with $x^2$. So, if your limited training data suggests that the function is a parabola, but in reality is not, you will learn a parabola (overfit), instead of learning the actual target function (i.e. generalise). It can be a big problem if the target function is actually very different from the function you learned, which may be the case if you used small or non-representative training dataset, because your model will give wrong predictions for unseen data.
So, you almost never want to overfit.

Oliver's answer is correct. I'm just giving a slightly different answer.


text:
Which techniques are best suitable for explainable AI for LLM models
I am currently working with large language models like llama and mistral, interested in techniques for making these models more explainable. I am looking for some tools or techniques which can help me with insights of how their models comes up with the predictions.
Which are the tools and techniques for interpreting the decision-making process?

Simple explanations of decision-making in LLMs are usually false.
Getting a correct explanation of how e.g. LLaMA-2-7b solves some task can be difficult. Here's my mental model of how tough it is in early 2024:

could likely be done by reusing existing code over a few weeks of work: indirect object identification, modular addition
research project: factual Q&A on a given topic, e.g. capitals of states, or birthdates of famous figures
tough research project: anything involving composition of multiple facts stored in model's weights
impossible with today's technology: e.g. solving coding problems, refusing to answer harmful requests

My understanding is that, in early 2024, there does not exist a task where looking up how a 7B LLM does it is trivial, in the sense that you could get a confidently correct explanation in a few hours, using openly available tools.
I recommend taking a look at TransformerLens; this repo is used by many recent mechanistic interpretability. I guess (but have no inside info) that Google DeepMind and Anthropic have better tools internally.

This is an active research area, and we can be confident that we don't know the best techniques yet. There are overviews of the field on Wikipedia, in this July 2023 special issue, and in this tag on the AI alignment forum. This blog post talks specifically about LLM interpretability.
The most promising technique I know of for LLMs comes from Anthropic in their Towards Monosemanticity paper. Scott Alexander wrote an explainer with pointers to important background papers. Unfortunately, it doesn't scale to Llama/Mistral levels yet.
For more near-term applications, attention heatmaps should be scalable with some work. Here is a 2019 blog post constructing one for BERT and a GitHub repo applying it to Llama.


text:
How to learn text style in an article using LLMs?
What is the best way to learn text style in an article? By text style I mean special characteristics and patterns inherent to different authors/group's writing style. For-example, author attribution problem is about learning who wrote an article.
I've seen some researchers applying VAEs and GANs to text to learn the style distribution of sentences in an article (Some references: here, here, here). However, VAEs and GANs usually have worse performance in language compared to vision tasks.
My question is how to capture style information using LLMs? Is averaging the last hidden states of sentences/paragraphs based on the "max_length" attribute of the model a good approach? Is there a better way to capture style information with LLMs?


text:
How do we know that $c(n,a,G) = h^{\ast}(n)$, in the proof that if a heuristic is consistent then it is admissible?
I found a proof that if a heuristic $h$ is consistent, then it is admissible, but I'm confused by one of the steps in the proof.
The proof is by induction on $k$, the number of actions from a node $n$ to the goal node $G$.
Base step: $k = 1$, i.e., node $n$ is one step from $G$. Since the heuristic function $h$ is consistent,
$$h(n) \leq c(n, a, G) + h(G)$$
Since $h(G) = 0$,
$$h(n) \leq c(n, a, G) = h^{\ast}(n)$$
Therefore, $h$ is admissible.
Induction step: Assume: If $h(n′)$ is consistent, then $h(n′)$ is admissible, for all nodes $n′$ that are $k$ steps from $G$.
$$h(n) \leq c(n, a, n') + h(n')$$
Since $n′$ is $k$ steps from $G$ and $h$ is admissible for node $n′$ that is $k$ steps from $G$,
$$h(n') \leq h^{\ast}(n')$$
$$\implies c(n,a,n') + h(n') \leq c(n, a, n') + h^{\ast}(n')$$
$$\implies h(n) \leq c(n, a, n') + h^{\ast}(n') = h^{\ast}(n)$$
hence, $h$ us admissible for node n.
I'm specifically confused about these lines:
$$h(n) \leq c(n, a, G) = h^{\ast}(n)$$
and
$$h(n) \leq c(n, a, n') + h^{\ast}(n') = h^{\ast}(n)$$
How do we know that $c(n,a,G) = h^{\ast}(n)$? Wouldn't it depend on the action $a$ chosen?
Also, how do we know: $c(n, a, n') + h^{\ast}(n') = h^{\ast}(n)$?

$c(n, a, G) = h^*(n)$ is true because $h^*$ is the optimal heuristic, i.e. it gives you the minimal/optimal cost from $n$ to the goal $G$, which, in that base case, is exactly $c(n, a, G)$, i.e. the edge distance from $n$ to $G$.
Now, I understand the confusion arises from the parameter $a$, the action.
Here, $a$ is arbitrary, but, in this case, there's only one action that should lead to $G$, which I suppose is a reasonable assumption, otherwise you would have multiple arrows from $n$ to $G$ (which are neighbours), which I've never seen.
I don't even know why they use this parameter $a$. I think you can safely ignore it, unless you want to consider cases where there are more than one edge between two neighbours $n$ and $n'$, which is useless because you can always consider the cheapest edge.


text:
Resulting quantiles from Quantile Regression DQN
In my QR-DQN application, the resulting quantiles for a state s and action a take the form of the blue line in the figure. The method works well in expected values and trains effectively. However, I know that in my problem the return distribution for (s, a) is multinomial with PMF only at {-2, +2}. Thus, I would expect the quantiles to take the form of the green line.
Is my expectation of the resulting quantiles correct? If yes, why does QR-DDQN behave this way?


text:
Should I define my problem as image segmentation or detection?
I have a problem and have to decide wether it's an object detection or object segmentation problem. I want to use Yolov8 for training. We already have hundrets of images but they aren't labeled yet. So before we start labeling we have to clearfy what we want. Maybe you can help me. I will provide the facts:

8 classes of objects
every class occurs 0 to n times per image
objects can be partially hidden by other classes or random objects (sometimes only 10% are visible)
if they are partially hidden, the visible part is contiguos, so no object is splitted apart
the objects are stacked, so they will touch each other (maybe it's a stack of objects of the same class)

As the result I want to know how often the classes occurs on the image. Like class1: 0 times, class2: 3 times, class3: 1 time, ....
At first it sounds like a detection problem. But since the objects are stacked and yolov8 uses rectangles for training, 90% of the labels will contain (small) parts of other classes - or only very small fractions of the classes can be labeled. If using augmentation like rotation, it will be not possible to avoid class-mixtures in labeling.
What does it sounds like? Segmentation or detection. Are there pros/cons?
I lean towards segmentation.
And:
Does it make a big change for yolo if I use segmentation instead detection? I know, the pixels were classified but has the labeling process to be different?
Thanks in advance

The scenario you're describing seems to initially suggest an object detection approach. However, given the complexity of objects being stacked and the necessity to count individual instances accurately, object detection alone, even with a robust model like YOLOv8, may not deliver the level of precision you need.
For greater accuracy, particularly where objects are partially obscured or in contact with one another, instance segmentation could be the more appropriate technique. Although it is more computationally intensive, instance segmentation models can provide the pixel-level detail required for precise identification and counting of object instances in challenging scenarios.

(source)

If I understand your requirement correctly, you're looking to tally the occurrences of each class in an image. I would suggest considering a detection approach as it might provide a more effective solution. Segmentation assigns labels to individual pixels, and in the presence of obstructions, it can be challenging to determine whether clusters of pixels belong to the same object or represent distinct instances of the same class.
Opting for a detection method offers additional advantages in your scenario. Since you lack a labeled dataset, the labeling process becomes more efficient with bounding box annotations compared to pixel-wise classification. Additionally, detection allows for accommodating obstructions by creating larger bounding boxes for obstructed objects.
I hope this perspective is helpful for your use case.

Can you please demonstrate how to label the red object? https://s20.directupload.net/images/240111/mivn9jno.png - for segmentation I would create a polygon follwing my red 'drawing'. for detection I would create a rectangle. but then the other object will be inside my label-rectangle


text:
Which RL algorithms can be used in an environment where actions have to be performed only in specific situations?
I am wondering which RL algorithms can be used in an environment where actions have to be performed only in specific situations. For example, on a conveyor belt on which a box that fulfills certain conditions must be sorted out. A signal must then be sent at the right time to sort it out correctly. If the signal comes too early or too late, it would be wrong.
Is it only the reward function that enables the agent to learn that or do I need to consider some special neural networks such as LSTM or do I need to consider certain RL algorithms?
Does somebody have some experience in this kind of scenario and can give suggestions?

Do you want the agent to learn both what signal to send and the timing at which to send it? Or do you just want to guarantee that the algorithm has known complexity so you can make sure that the signal is sent at the right time (i.e., the algorithm will take at most X ms to process its input and update its actions based on the last feedback, so it will be ready in time for the next item on the conveyor)?

Also, (from the "related" tab), consider PID. https://ai.stackexchange.com/q/12472/61494. When to take an action sounds like the kind of well-defined problem suited to PID. With a simple memory that recalls which action was chosen, you could decompose the system into "Which action should I take" and "Did I take action successfully?" You can use these to send feedback to the "action chooser" (RL) and individual "action taker" (PID) controllers. Things are a bit more complex if the action to take depends on how much time will be required, but I think still decomposable.

The act of pushing something off a conveyor is probably best handled by something real-time (microcontroller, PLC, etc).  When a box passes a gate and the pusher is armed, fire the piston which pushes.  We understand how to fire a piston, no ML needed.  The logic for when to arm the pusher could be learned by an RL algorithm.
The complexity of the model will primarily be driven by how Markovian the problem is.  A problem is said to be Markovian if the decision can be made by looking at the information in the present situation only (no history needed).  For Markovian problems a simple feed-forward network may be enough.  The output being a signal either to arm or disarm the pusher.
If the decision depends on history, the decision to push the present box depends on the boxes which have passed before it, then one may consider an LSTM so the network may efficiently integrated the information necessary to solve n-step markov problems or non-markovian problems.
On the off chance one REALLY wanted to do the whole process by ML, construct it like a videogame.  Pushing correct packages off gives a reward and incorrect a negative reward.  A DQN family algorithm would likely get the gist after several million frames.  Predicting future position based on current movement is non-markovian, the original DQN paper used three frames into a convolutional network.  LSTM would also work there.


text:
Why the space complexity of depth-first search is O(bm)?
In "Artificial Intelligence: A Modern Approach", the authors say that the space complexity of depth-first search is proportional to O(bm), considering b as the branching factor and m the maximum depth of the graph. But it is not obvious why, for me.
The algorithm for searching graphs involves storing the explored nodes in the explored set, and the nodes open, but not explored yet, are stored in the border set.
Let us consider the following graph, with the start node in green and the end node (the goal) in red.

The solution is given by the red path.
But, in my perspective, in this scenario, in the worst case, the algorithm would try every path from the root to each leaf, and store all the nodes in the explored set.
Thus, if we consider this, the space complexity is, at least, proportional to $O(b^m)$.
Is there something wrong with this reasoning?
I'm including another image to represent better the searching process on a graph.

Best regards.

If you have looked through an entire branch, you only need to mark off the top node for that branch.

Thanks for your contribution! Would you be able to elaborate your answer a bit more? Without further explanation, I'm afraid that your answer wouldn't really help to clarify things for anyone who would ask the question in the first place.

But notice that the algorithm proposed in the book stores all the explored nodes, in order to avoid visiting them again. And notice that although my example is in a tree, we are interested in searching on graphs.

To visit a node again, you have to visit the node's parent. If you're at the node's parent, you do need to mark off the node. But if not, you only need to mark off the parent. You can continue upwards: if you're not at the grandparent, you only have to mark off the grandparent. You end up only needing to mark off nodes adjacent to your path, which is $O(bm)$ in space. This works for any graph.

@programjames
This is not true if we are searching in a graph. In a graph, a given node can be a successor of several different nodes. I have included a novel image to represent the searching process on a graph.
But, it is important to notice that the algorithm presented in the book that I have cited says explicitly that we need to store all the explored nodes.

But you still only need to mark off a boundary, not every node in the area.

But how can we avoid expanding again an already explored node without tracking (and storing in memory) the nodes that were already explored? We don't know in advance what are the nodes that were already explored and will be visited again. This is not clear for me.
But, as I was said, the book says that the explored nodes are stored. So, let us assume that we really need to store them. In this case, my reasoning is ok?


text:
Is there a standard nomenclature for model names suffixed by strings such as "Q4_K_S"?
Many models specify details using prefixes such as Q6_K or Q4_K_S.
It seems obvious that the Q portion refers to quantization; what about the other characters?
Is there a standard nomenclature reference for this?

Can you please point us to a reference/link/webpage that uses this notation?

Just about any huggingface model page, such as this one: https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF


text:
Does Breadth-First Graph Search need to store all all unique generated states?
I am reading Artificial Intelligence: A Modern Approach by Russel & Norvig. As I understood it, a breadth-first graph search stores explored states instead of explored nodes (where "nodes" are the actual nodes generated by the search algorithm i.e. the "nodes" contain the state). However on pg. 82 the book states

"For breadth-first graph search in particular, every node generated remains in memory."

But, by my understanding the only nodes we store in memory are the nodes in the frontier(as defined in the book). Instead it is the unique states that have been generated that we must store. Note that this is not a trivial difference since there are problems which can have finite states but infinite nodes.

States are represented by nodes in the graph. In other words, a state is a node in the graph. So, I don't think there's any difference, but some authors maybe make a difference (e.g. because you don't need to store all information associated with some state while searching).
BFS explores by level (hence the name breadth, in contrast to depth-first search, which goes deep and then backtracks), in the sense that it will first explore all nodes that are $w$ away from the start node before exploring all nodes that are $w + 1$ way, if you think of the edges of the (unweighted) graph as having a weight of 1, which you can do.
BFS indeed needs to store a frontier in memory, which is a FIFO data structure (i.e. you first remove from the frontier the oldest node there in constant time) of nodes we can explore next.
If you use BFS to find some path, you need some way to compute or keep track of it. If you apply BFS, you can generate a tree, where each node has a parent, which can be used to retrieve the path, i.e. you just backtrack from the goal.
Note that graphs can have cycles and you can get to a node in different ways. So, you also need to mark a node as explored in some way, which may be with some variable you associate with the state/node (CLRS approach) or with an explored set (AIMA approach), which may thus contain all nodes, which makes BFS an instance of graph search (see also this).
The running time also depends on how you represent the graph (e.g. adjacency list and adjacency matrix) or how you analyse the algorithm. See also the CLRS book for a different perspective on this topic.


text:
Why is R(s) more restrictive than R(s, a) in an MDP?
I am quite new to RL. I would like to know why a state-dependent reward function R(s) is more restrictive than a state-action-dependent reward function R(s, a)? And why is it that a policy can be optimal for the latter R(s, a) but not for the former(R(s).

Does this answer your question? How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

you have in front of you 10 slot machines

you can play with any of them, and each of them have a specific winrate (reward function)

the only state of this MDP is the initial state, the one where you are in front of these slot machines

how do you model the reward?
a. $R(s)$: well, the state is only one, so the distribution of the rewards does not depend on the action you take/ slot machine you play
b. $R(s,a)$: well, now you have actions, so you can learn statistics about $R(s,a)$ (aka statistics about each slot machine), and thus take actions according to these statistics (policy)


text:
Is there any well-established work that allows robots to communicate their decision-making using natural language?
I am searching for a well-established work that allows robots to communicate their decision-making using natural language.
For example, a robot's explanation could be "I did [task1] because [reason1] and [reason2]."
I know there is parallel research across different fields but I was wondering if anyone knows a good work with reusable code.
Thanks!

they did something like this in this big project https://ai.meta.com/research/cicero/diplomacy/


text:
Is there validation data in K-fold cross-validation?
We know that in machine learning the dataset is divided into 3 parts: training data, validation data and test data.
On the other hand, K-fold cross-validation is defined as follows:

the dataset is divided into K number of different sectors. One section is used for testing and the rest for training. The results of these K-iterative tests are then averaged to get the final accuracy.

What happens to validation dataset in K-fold cross-validation? Is there such a dataset?

Before doing the $k$-fold cross-val you divide all your dataset into train and test splits (e.g., 80-20 in proportion), then the $k$-fold cross validation is performed on the training split as follows:

The training data is split into $k$ folds, where at each training iteration a fold is held out for validation: so $k-1$ folds are effectively used for training. The process repeats $k$ times, each time using a different fold for validation.
The $k$-fold cross-val approach is useful when your dataset is small, also it can be used as an easy way that provides $k$ (slightly different) models to build an ensemble (since each iteration trains a new model on a different split of the training data), which is supposed to generalize better. Moreover, since you have $k$ models you can use them to estimate the variance of their predictions, which can be used as a way to estimate uncertainty.

I think it is like this. You divide the dataset into K-folds. 1 fold is used for test. Then, you divide the remaining K-1 folds into training and validation based on a chosen percentage.

This is a nice description of k-fold CV, but I don't think this answers the actual question.  Do we need to differentiate between testing a validation datasets in k-fold CV?

@nbro Well, the fact is that in classical ML there is no notion of validation set, which instead is present in modern ML and deep learning. In the first case, you perform K-fold cross-val on the entire dataset without the train-test splitting beforehand: so one fold is used for testing. But in general, especially when you need to tune hyper-parameters, performing k-fold cross-val on the training-split, and after having the real evaluation on the test-set is the safest option to not underestimate the generalization error.


text:
What is the current state of AI regarding features of Moravec's paradox, such as sensorimotor & perception skills?
Moravec's paradox is the observation in artificial intelligence and robotics that, contrary to traditional assumptions, reasoning requires very little computation, but sensorimotor and perception skills require enormous computational resources.
Given that there have been strong advancements in computer vision in the past few years, what is the current state of AI when it comes to giving a physical robot the possibility of identifying and performing unstructured physical tasks in the real world?


text:
What happens when the probability of either one of the policies is 0 in Importance Sampling?
I have a general question about the methods that use importance sampling in RL. What happens when the probability of either one of the policies is 0?

Can you be more specific? Can you provide some formulas? You can use mathjax.

Background: Importance sampling is used in many off-policy RL algorithms when the data is generated with one policy, yet it is being used to update another policy. The policy generating the data is usually called the behavior policy and the policy being updated is the target policy. Since the distribution of data that is generated by the behavior policy is not necessarily the distribution of data that would be generated by the target policy, using the data from the behavior policy might lead to inaccurate estimates when updating the target policy. To mitigate this issue, it is common to introduce the importance sampling ratio
$$\rho_{t:T-1}\ \dot{=}\ \prod_{k=t}^{T-1} \frac{\pi(A_k|S_k)}{b(A_k|S_k)},$$
where $\pi$ is the target policy, $b$ is the behavior policy, $t$ is the current time step, and $T$ is the terminal time step. For further questions about this derivation and how the importance sampling ratio can be used to update the target policy, see the discussion near equation 5.3 in Sutton & Barto.
(Case 1) Target Policy is 0: In the case where one of the terms $\pi(A_i|S_i)$ is $0$ in the numerator of the above importance sampling ratio, then the entire product is $0$. Intuitively, this sample should have minimal influence in updating the target policy since it has zero probability of being generated by the target policy. As an example, in weighted importance sampling (see equation 5.6), this data sample will have no effect on the estimate.
(Case 2) Behavior Policy is 0: In the case where one of the terms $b(A_i|S_i)$ is $0$, the above product would be undefined. Intuitively, if the behavior policy was used to generate the data, then the data must have a nonzero chance of being generated by the behavior policy in the first place; otherwise, we reach a contradiction that results in the undefined product. See Section 5.7 of the same reference for a discussion regarding the design of useful behavior policies to mitigate this and other issues.

Very simply, one of the requirements of off-policy RL to converge, is that the behavioral policy $b$ has at least the same support of the target policy $\pi$, thus:
$$
\forall s \in S \forall a \in A \,\,\,\,\,\,\,s.t\,\,\,\,\, \pi(a|s)>0\Rightarrow b(a|s)>0
$$
Thus, you must have a policy that would do at least everything that your knew policy would do
In the case $\pi(a|s) = 0$, then you just don't learn anything


text:
How can we create an encoding scheme that captures these relationships between the suits of cards in poker?
In poker, a standard deck of 52 cards is used. Each card has one of 13 ranks, ranging from Ace to Deuce, and one of 4 suits: Spades, Hearts, Diamonds, or Clubs.
The rank of a card is crucial. For instance, holding an Ace and a Jack is significantly different from holding a King and a Jack, or an Ace and a Seven.
However, the suit of a card doesn’t hold individual importance. What matters is whether the cards share the same suit or not. For example, holding the King of Clubs and the Jack of Diamonds is equivalent to holding the King of Hearts and the Jack of Spades.
Poker players often talk about whether their cards are suited (sharing the same suit) or offsuit (having different suits). They describe the cards on the board as monotone, two-tone, or rainbow, and talk about holding a flush draw or backdoor flush draw. These terms don’t refer to the suit of an individual card, but rather to the pattern of suits among all the cards.
So, when encoding cards for a game of poker, it’s important to reflect these aspects. Instead of encoding the suit of each card, we should encode the relationships between the suits of all the cards. This approach captures the essence of how suits function in poker.
So, how can we create an encoding scheme that captures these relationships between the suits of cards in poker?

https://ojs.aaai.org/index.php/AAAI/article/view/20394/20153

@Alberto, can you add your thought about the article?  I think that this article does not answer this question because it do not use any encoding for suits.  See Effective Game State Representation: "1) there is no abstraction of the card information thus reserves all the game information"

This is normally handled by considering hand isomorphisms. So in pre-flop Texas holdem there are 169 isomorphisms, consisting of 13 pairs of the same rank, 78 suited pairs of different rank and 78 unsuited pairs of different rank. You can take a similar approach to post-flop boards. Iirc there are 1755 isomorphic flops, 16432 turns, and about 127000 rivers, quite a few of which are nut rivers. Google "A fast and optimal hand isomorphism algorithm"


text:
What is 'system card'?
What is 'system card' in these context:

https://ai.meta.com/blog/system-cards-a-new-resource-for-understanding-how-ai-systems-work/

Additionally, individual model developers may provide documentation
such as the OpenAI system card for the GPT-4 model.

Source: https://learn.microsoft.com/en-us/training/modules/responsible-generative-ai/3-identify-harms
By the way, what is 'model cards'?


text:
What is the concept of pruning a tree in Machine Learning regression problems?
What is the concept of pruning a tree in Machine Learning regression problems? I am confused and a simple explanation would be great.

The concept of pruning a tree is mainly to avoid overfitting in either decision tree (DT) classifier or regressor learning when there is noise in the data or the number of training samples is too small to produce a representative sample of the true target function.
Furthermore DT regressor usually needs to transform possibly many features' continuous ranges to dynamically created possibly multiple ad hoc Boolean test attributes based on their information gains without firm theoretical basis, and each of its leaf nodes contains a constant value that serves as the predicted value which is typically the mean or median of the target values of the training data in each leaf node partition, therefore this effectively creates a noisy training dataset even the untransformed raw dataset is noise free. Thus for regression problem DT learning is very prone to overfitting and the concept of pruning becomes extremely important to reduce its inherent high variance and MSE/R-squared of its performance metrics.
Of course the learning algo can stop growing the tree earlier before it reaches the point where it perfectly regresses all the training data, but it's difficult to estimate precisely the minimum tree depth or leaf nodes.

Prepruning methods share a common problem, the horizon effect. This is to be understood as the undesired premature termination of the induction by the stop () criterion.

Thus post-pruning (or just pruning) is the most common way of simplifying trees. For DT regressors you just replace below classification accuracy confusions of validation data with MSE or R-squared

Here, nodes and subtrees are replaced with leaves to reduce complexity. Pruning can not only significantly reduce the size but also improve the classification accuracy of unseen objects... The procedures are differentiated on the basis of their approach in the tree (top-down or bottom-up)... One of the simplest forms of pruning is reduced error pruning. Starting at the leaves, each node is replaced with its most popular class. If the prediction accuracy is not affected then the change is kept. While somewhat naive, reduced error pruning has the advantage of simplicity and speed.

For a detailed pioneering example of the popular reduced error pruning see Quinlan, J. R. (1987), 'Simplifying decision trees', International Journal of Man-Machine Studies 27, 221-234.


text:
Do neural network weights need to add up to one?
The idea that weights determine how much influence each input value from the current layer will have when calculating the input to the following layer reminds me of when my professors would say that our final grade for the course was broken down something like this:

Item
Percent of Grade

Midterm Exam 1
20%

Midterm Exam 2
20%

Final Exam
50%

Classwork
5%

Homework
5%

Total
100%

My question is then do all the weights need to add up to 1.0 (i.e. 100%)? If not, why?

Why "If not, why?" and not "If yes, why?"

@Stef I think I just assumed both were implied when I was thinking of the question, didn't think much of it, but valid point

;) You'll notice one of the answers begins with "No, the weights do not need to add up to one. There isn't really a reason to do that." and another with "No. Why should they?". There are situations where imposing the constraint "must sum to 1" is useful, but the weights of a neural network is not such a situation. Since it's not needed, we don't impose it. Scolpe's answer gives one argument why imposing it would be a bad idea. One place where this constraint would be useful is in the outputs of a classification neural net, where the outputs do represent probabilities (but not the weights)

I don't understand what the relationship between the grades and the weights you thought could possibly exist. It seems that something is missing from this question. Some people in the answers even mention probability distributions, which you don't mention. This seems like a random question. Maybe a question that has not been asked here, or maybe yes. Still, the motivation for this question is unclear to me. It's unclear where the doubt is. It's unclear why you're asking this question. Still, it makes us think why. So, why not challenge the status quo?

@nbro I didn't originally think of weights in this way. I'm a visual learner and trying to think of real-world things I could compare weights to to help me visualize them and this idea came up.

Simply, the answer is "no".

No, the weights do not need to add up to one. There isn't really a reason to do that. Weights as "contributions" may not be the best way to think about things here -- you're trying to learn a function that transforms the input values into your desired output values. Say you're learning a linear model and your data is 2D and drawn from a line with slope 2, then you'd want your learned weight to be 2.
Additionally, say you have a feature that is negatively correlated with the output. That is, that a higher value in the input results in a lower outputs value. If all your weights are non-negative, you wouldn't be able to encode this at all!
Finally, as the output would simply be a weighted average of the inputs, the scale of the outputs would be constrained by the scale of the inputs. Consider:
$$f(\mathbf{x}) = \Sigma^d_{i=1} w_i x_i$$
with
$$\Sigma^d_{i=1} w_i = 1$$
$f(\mathbf{x})$ will always be less than or equal to $\max(\{w_1, ..., w_d\})$. For a simple linear model, you can definitely think of examples where the scale of the output is much larger than that of the input features (for example, number of bathrooms in a house versus its price). Although you can mitigate some of this by rescaling your data (as I will talk about shortly), there is really no reason to artificially constrain your model like this.
A deep neural network can be thought of as stacks of these linear models, with the output of the previous layer being the input of the next layer. Although, these intuitions aren't at all perfectly transferable to deep neural networks, we would still be constraining the expressiveness of the model in a similar way: an increase in an input dimension can't result in a decrease in the output value and the scale of the output is constrained by the scale of the input.
Normalization in Neural Networks
You do, however, often need to perform normalization, but this normalization is done on the data and the outputs of hidden layers instead of the weights. If you add a weight penalty to your linear model for example, you want to make sure that each dimension is in around the same range, to not overly penalize features that tend to have larger weights. In neural networks, you also often normalize weight values for hidden layers, the most common variant is Batch Normalization.
However, the normalization often used in these cases is a bit different from what you mentioned. Also known as standardization, this kind of normalization subtracts each sample by the mean (over a set of samples) and divides by the standard deviation (over that same set of samples).
The exact reason why this is effective in neural networks is an active area of research. A common explanation is batch normalization reduces the effects of internal covariant shift, where changes in weights during training results in changes in the output distribution of a layer, which can harm performance as the next layer "expects" the original distribution. Centering and scaling the input to have zero mean and unit standard deviation reduces these changes. This, however, is disputed by this paper, which argues that batch normalization helps by making the loss landscape smoother.

Nothing to do with it! Let me illustrate this concept using one of the simplest models in machine learning: Linear Regression. In linear regression, the goal is to find the coefficients of a hyperplane that best fits the given data, represented by the equation $Ax=b$. Here, we have free variables that represent the slopes along each axis and an independent term for the height (that in neural nets is called the Bias). Now, to address your question: Do the coefficients need to add up to 1? The answer is no.
You might be tempted to interpret the concept of weights with the concept of probabilities, but they are not the same! They are merely coefficients in a high-dimensional, nested function.

Here's a 3D example of linear regression. The blue points represent the data, and the red plane is the best-fitting hyperplane according to the linear regression model.
The equation of the fitted plane is approximately:
$y=2.79⋅x_1+2.01⋅x_2+4.11$
As you can see, the coefficients are  $2.79$ for $x_1$ $2.01$ for $x_2$ , and the intercept is $4.11$. If we sum these together, we get $2.79 + 2.01 + 4.11 = 8.91$, which is clearly not equal to $1$.

Couldn't you simply reparameterise the coefficients to [0.2817, 0.2026, 0.4157] and have them add up to 1 again?

I think the main point here is the term "weights" has nothing to do with the term "weights" which represents a percentage.

@ToivoSäwén If we parameterize the coefficients to the values you mentioned in order for them to add up to one, we will not end up with the same hyperplane. In the example I gave, we had the following linear equation:
y = 2.79x₁ + 2.01x₂ + 4.11
If we divide the entire equation by 8.91, we get:
y' = 0.31x₁ + 0.225x₂ + 0.416
This changes the y value to y'. Therefore, we would have to change our data "y" value in order for it to work. I believe you are confusing normalizing a vector with the property that all its components add up to one.

do all the weights need to add up to 1.0 (i.e. 100%)?

No.

If not, why?

Why should they?
In the case of school grades, the justification might be that each component (midterm exam, homework etc.) is graded separately on a same scale and the final grade, which must lie on the same scale, is computed as the weighted average of these components.
There is no such requirement regarding weights in neural networks.
What you might want to enforce is that the outputs in the last layer sum to one, to obtain probabilities of class memberships (this, obviously, applies only to classification problems and not to regression). Softmax is typically used for that purpose.

No, weights in that case do not constitute a formal probability distribution and do not need to be add up to 1. In fact, if they would — it could cause problems with underflow. Imagine a (nonsensical) neural network — n inputs, followed by 10 layers, each containing 10k parameters. This MLP would possess 100,000 parameters, excluding biases and the output layer. Given the novel architectures, it is not even a large number of parameters. Now, we normalize all the weights (layer or network-wise) so they form a valid probability distribution. It is easy to see, that the forward and backward passes could be problematic as the value of each parameter would be close to 0. Also, each gradient could be negligible small - a phenomenon that is observable now in NN due to various reasons Vanishing Gradient Problem.
How do we interpret the activity of the Neural Networks? It is a reasonable question — there is a field of research that is called an Explainable Artificial Intelligence and deals with the interpretation of many deep learning architectures. Given that deep learning basically provides for creating a sequence of nested functions that can approximate arbitrary difficult non-linear functions it is a non-trivial task. If the weights would sum up to 1 — it could greatly help the progress. Unfortunately, this is not the case ;)


text:
What is the difference between A/B testing and Reinforcement Learning?
I was learning ML, and I learnt a new section called, Reinforcement Learning. After some research on web, I found that it is a trial and error technique by which the agent learns from the environment by some interactions (actions), and it get rewarded if the action is accurate.
I wonder why it is better than A/B testing ? We can simply perform A/B testing between the variants of any product and can calculate the Interaction Rate, and can choose which one of the variant is better.

What is the difference between A/B testing and Reinforcement Learning?

Why RL is better than A/B testing?

Please, next time, put your specific question in the title. Thanks.

What is the difference between A/B testing and Reinforcement Learning?
A/B testing can be thought of as a very basic subset of reinforcement learning. A/B testing is usually more closely related to context-free bandits (also a kind of subset of reinforcement learning), because it typically does not consider anything stateful about user interactions (e.g. what site the user arrived from, and demographic data from user profile), or any time domain changes (e.g. what has been presented and clicked/not clicked before), although both are possible whilst still following a simple "collect data, predict best choice later" design.
The main differences are:

Training of simple A/B testing is batched, whilst reinforcement learning, and bandit learning algorithms are online.
That means with reinforcement learning and bandit algorithms, decisions can be made in real time based on knowledge so far. You can choose, by tuning the algorithm, to optimise for better results during learning, reducing the risk of losing revenue from possible bad choices.
Reinforcement learning (unlike A/B testing or bandit algorithms) can attempt to optimise longer-term goals, if there is some kind of state that can be tracked, and that is influenced by decisions that have been made. So for example you can optimise for overall revenue rather than short-term results from each site visit independently.
Outside of promoting user choices on the web, and other "single interaction, single choice" scenarios, reinforcement learning is a superior theoretical model for agents which play games, control robots or manage any long term interactive system. RL solves problems involving sequential action choice, whilst A/B testing cannot really do that as a framework.

Why RL is better than A/B testing?
It isn't inherently better at all things. A/B testing may be an appropriate model and system design for some scenarios.
However, RL is the more general purpose learning system, whilst A/B testing is a simpler tool that is more specific.
Usually the "upgrade" from simple A/B testing would be to look into context-free or contextual bandits. These can improve revenue in advertising or product promotion scenarios due to the online learning factor. These improvements occur independently of whatever batch learning timing you may have set for offline training of A/B testing, and can be left on continuously in case something changes after the initial trial. Bandit algorithms add complexity, but are otherwise a drop in replacement for existing A/B systems. In that sense they can be viewed as better than simple A/B testing.

A/B testing takes more time because it is trained in offline before deployment and Reinforcement Learning techniques (e.g, Upper Confidence Bound) can be deployed directly, because it is trained by trial and error method , right ?

But it A/B testing is very simple because we just have to calculate the no. of votes for each variations and can deploy them afterwards

@mainakmukherjee - it is not just "takes more time". This time has often cost you money directly because one or more of the choices is not optimal, and A/B testing won't allow you to adjust the ratios in response to early stages of learning. In addition, if you train/deploy and then consider all trials done, you may miss out on changes to preferences for any reason. You can adapt A/B testing for those things (check early results, always have a 10% chance of showing worst afterwards and monitor routinely), but the more you do so, the more you have effectively changed A/B testing into RL.


text:
What is considered the pre-fill, and what is considered the decoding phase in this process?
I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically.
I used to think that a sequence of tokens is generated in, and a sequence of probabilities for the next token in the sequence is generated as output. This generated output token is then appended to the sequence of tokens, and fed in the model again.
However, I know understand that there's a key value cache that's generated for each sequence of tokens fed in. This key value cache stores a precomputed matrix that can be used in future computations to prevent having to recompute previously seen tokens.
So, the new workflow is feed in token list -> generate next token and key value cache -> feed in next token -> generate next token and key value cache for current token
What is considered the pre-fill, and what is considered the decoding phase in this process? Does the prefill phase involve feeding through the token list one by one passes to generate the kv cache? Why does the prefill phase take significantly longer?

From Efficiently Scaling Transformer Inference:

Each example ... has L_input tokens of input text, and generates L_gen
tokens of output text. ... Since prefill can run in parallel over
L_input, but decode must run sequentially over L_gen, the two phases
have different performance characteristics and we analyze them
separately.

This means the KV cache can be filled fully in parallel for all L_input tokens. In contrast, in the decoding phase the output has to be computed autoregressively, i.e. one output token after the other because the current output token is dependent on the last output token.


text:
How is LLM generated content moderated?
I'm looking for references (articles) about how LLM generated content is moderated.
From a technical point of view, what makes the difference between a so-called "uncensored" LLM, such as Pygmalion 7B, and a "censored" one?
Does a LLM always generate text that gets moderated later, or is it pre-trained / fine-tuned to generate moderated content by default?

Is LLM like ChatGPT able to generate content that needs to be moderated? I am thinking about this from training point of view, is it possible that LLM generates sensitive or illegal content from training set not containing such?

The better formulation of this question is “How do LLMs get aligned with human values?”. Which humans and what values is an interesting societal question. Let’s say, for the purpose of this conversation, it’s with a set of humans who don’t like to hear curse words.
There are 4 main forms of achieving alignment.  The layer at which each is implemented is shown in [brackets].

[model] Pre-train your model on the data set you want to learn from. If you don’t want curse words, strip all the training data of curse words laden content, and the model is better aligned with your “no curse words” values.
[model] Fine-tuning & reinforcement learning w/ human feedback (RLHF).  If you have a base model that is good in performance but needs alignment, you can take a smaller set of data than what was used to pre-train, and do a post-train step.  This fine-tuning data is often very carefully curated to achieve specific outcomes or match a specific data distribution.  Fine-tuning on that data often “re-wires” the neurons (via updating the weights of the neurons) to reduce the curse words.  Reinforcement learning can further achieve alignment using a more complicated process of creating a reward model that the base model is further tuned on.
[application layer] At the prompt layer, few-shot learning and system prompts can be used to prevent the model from using certain words, by way of giving examples.  This is also called “in-context learning” and can be used for many outcomes, alignment is one of them.
Post-generation of the model, the final result can be passed through another model or ensemble of models. These are typical classifier models that are looking for specific things. You could put any kind of “model” in this step from something naive as keyword filtering, to more sophisticated models like “hate speech”, “violence” or “sexually explicit material’ detection. The best practice is to pass prompt inputs (from users) through moderation systems, but the outputs from the LLM can also be checked.

TL;DR; There is no model inherently censored, as big models have big training data which can't be all modified to not include explicit content. Also, we don't want to not-include the explicit content lest we want to take away the abstractions that the NN might learn from that explicit content also. There are multiple layers to causal reasoning involved to design commercially available products like OpenAI's models.
That's an interesting one. Let's first differentiate between different terms being used. A foundational model is a model that is pre-trained on a set of training data and can be fine-tuned to perform downstream tasks, whereas consequentially a fine-tuned model is one that can perform specific tasks like summarization, topic extraction, NER, etc.
Now, let's talk about the GPT-3 family for a bit which houses both the Davinci model and ChatGPT (GPT-3.5) model. If you read about it a little bit, OpenAI lets you fine-tune Davinci but not ChatGPT. Davinci was fine-tuned to perform better generations/completions. On the other hand, ChatGPT was fine-tuned to perform humanesque chat conversations and consequentially started outputting more uncensored content (because of-course human texts tend to get witty).
Then comes the concept of causal reasoning. You can understand it most easily as a conditional statement which is an added layer over the output of the language model. Essentially it forces the model to take certain actions given certain scenarios. One example might be a prompt saying to use a different selection strategy (it becomes greedy at temperature=0) if the next generated token leads to a vulgar word/slang. Another might be pre-added instructions to model input requesting it to return 'I can't respond to this query as it pertains to copyright information'. It's basically an elaborate if-else statement that works with language models.
Coming back to our example of OpenAI's models, while not completely uncensored, DaVinci is susceptible to producing more uncensored content than ChatGPT as evidenced in their blog asking users to use a layer of causal reasoning to best meet their market requirements and that OpenAI will not be held liable to any explicit content generated.
UPDATE (21 July 2023) - According to this UC Berkley study, this causal reasoning might be actually reducing the effectiveness of ChatGPT due to increasing limitations on the content that it can produce.

Are you saying it is not doable to prevent sensitive training data, using amount of data as an excuse, from entering the learning process?

I think it's more practicality than the amount of data that I mean here. Of course, you can engineer methods to reduce bias (which of course is necessary), but it shouldn't come at a cost of decreased associations which might in itself be a bias. Ultimately, sensitive information like vulgar words to socially unacceptable ideas should be included in the training data and the model should later be fine-tuned to behave a certain way. We want a foundational model to be able to understand as much as it can.

It is a practicality for sure, and I can only image how much training data there is available and used, when it comes to more larger LLM. Still I wonder maybe social media companies like Facebook have means to recognise offensive language from the stream of text they keep receiving. And if this is doable, why not filter offensive language from LLM training similarly too, just before it enters the training loop.


text:
How to handle penalty and reward occurring simultaneously
Assume the following scenario:

We have an agent that acts on an environment where the agent should never take an action that results in him leaving the environment. For example, imagine an agent playing tennis and he should never run over the lines that determine the field.
However, it may occur that the agent has to cross the lines in order to hit the ball (let's stick to the tennis game). In that case he would get a reward for hitting the ball and a penalty for leaving the arena.

I wonder what approach would be better:
Only give the agent a reward while he is in the arena, so in our example he would not get any reward while outside of the arena.
Or sum up the penalty and reward, so the agent always gets the reward, regardless if he hurt the constraints or not.
I thought that giving him the reward, even when outside of the arena, could encourage him to keep hurting the constraints. On the other hand, I fear that not giving the reward, he would miss out on learning how to hit the ball.
Which approach would be superior?

the sign of the reward should tell the agent if it was something wanted or not, and the magnitude how wanted or not it is (then there are more advanced concepts like advantage function that changes this, but still)... the reward function shaping is a very important part as it's the "supervision" induced to the agent, thus if hitting a ball has a reward bigger than the penalty for the line, the agent will learn that it's better to hit the ball (and yes, theoretically you should sum... if you will 5 dollars and you loose 3 dollars, it's the same as winning 2 dollars)


text:
What is function calling in OpenAI's ChatGPT models?
OpenAI has recently added something that it calls function calling to its ChatGPT models/API.
What is meant by function calling in the context of a large language model? Does it allow us to invoke functions and feed them back immediately?
What limitations does it have? What sort of applications does it enable?

This question is about a specific API, so it's off-topic.

There are  other questions about chatGPT on here. I'll try and rephrase though.

Models and APIs are not the same thing. An API has nothing to do with AI, unless that API was e.g. created by an AI. Function calling seems to be just an API feature. If function calling is something that the model itself is able to do, then fine. If they created a new model that can produce JSON, then your question is more likely to be on-topic here.

According to this https://openai.com/blog/function-calling-and-other-api-updates, they fine-tuned the model, so it's not just an API feature.

Indeed before it could just output langauge now it is outputting parameter values.

If you examine the announcement on https://openai.com/blog/function-calling-and-other-api-updates function calling does not actually give LLMs the power to invoke functions directly.
Instead it gives them the power to populate JSON models of the arguments to functions.
Hopefully ones for which it has a good model.
So there is a new output from such models. In addition to language they can output parameters. Though this is not such a stretch given that they can already output code which include functions and parameters.
These can be checked to see if they follow the rules of the function itself in the same way that written output must conform to the rules of the language.
It is an obvious evolution as:

It plays to the models strengths - creating seemingly sensible patterns
Its what many applications of chatGPT are already doing.

The model itself will not invoke the functions but its easy to imagine invoking a function which will provide new prompts.
I think its very wise to break the feedback loop here as immediate invocation of arbitray functions could have undesirable consequences.
I expect we will see some consequences from people experimenting with that in the near future.
The limitations are less obvious but you can imagine it falling prey to its own hallucinations.

What sort of applications does it enable?

Here is an example of an application that function calling enabled: PDFTriage: Question Answering over Long, Structured Documents

We propose an approach called PDFTriage that enables models to retrieve the context based on either structure or content. Our experiments demonstrate the effectiveness of the proposed PDFTriage-augmented models across several classes of questions where existing retrieval-augmented LLMs fail.

OpenAI can call Functions you provide in your API. This function can provide an extra information that is now available on web, or it can execute a task such as registering to an event.
You can check this article which proviodes further details and example project with function calling feature of OpenAI:
OpenAI function call example


text:
Is it possible to have a ChatGPT bot that can answer questions about sensitive information without using external servers?
We have a lot of internal documentation that contains sensitive information.
Is it possible to have a ChatGPT bot that can answer questions about this information, to my used by users with clearance for this information, without posting the information to external servers?
I can find documentation on embeddings, but these seem to require posting info to OpenAI for it to be processed and stored there.

Yes it is possible. This is a popular use-case these days. check out privateGPT.

You can run open source models locally without sharing your data to any server. Nowadays to perform well you need to use GPUs for efficient running of te queries, either locally or on private clouds. You may prefer services that provide private instances of large models such as GPT / Claude (available on Azure / AWS) or use smaller models with e.g. RAG to optimize the results for your domain and data.


text:
Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?
The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without hitting obstacles. Those locations that need to be visited are fruits that requires the agent to fertilize them. There are different fruits but the agent only have to fertilize specific types. The only available information to the agent is its current location and the type of the fruits to be fertilized. It is possible to know the location of the target crops but not the obstacle beforehand. The fixed information in each episode is the size of the field/grid-world, the type and locations of the fruits, the initial state of the agent.
My question is  - can I train the agent with a DQN algorithm, avoiding obstacles and still finding the optimal path to finish the task?

Could you please give some more information regarding the rules that the obstacles follow? In your previous post you said that the agent can see obstacles, but only nearby ones - is that still the case? Also, it will make a difference to the best approach to use depending one when new locations for obstacles are generated, and what kinds of rule are used to generate those new locations. For example there is a big difference between random positions chosen once at the beginning of each episode, and positions that change on every time stamp. Also between completely random and items that move

Obstacles are generated at the beginning of each episode and will not each time step. But it does not matter as the agent does not have access to the whole field. It is a simulation which is supposed to be realistic. But for simplicity we assume that the obstacles when they are generated they will not move for the whole episode. About the seeing part, in the training phase at the moment I assume that by giving high penalties, the agent will learn the location of the obstacles and/or not to hit it. But it is not in the state representation of the agent

This is technically a partially observable MDP (POMDP), because some of the environment state is not visible to the agent. However, there are degrees of non-Markov states, and the environment you describe is pretty close to Markovian, with new details being revealed to the agent effectively at random as time progresses.
So the good news is that a DQN-based agent should be able to cope. One caveat is that you should code the state so that it keeps a memory of the obstacles seen so far (the state is the agent's knowledge of the map and its position on it). Another is that there is no RL solution that will be optimal from the perspective of an observer that knows the location of obstacles in advance. The best any agent can do will be a policy that does well on average given the limited knowledge that the agent has. If a set of obstacles is generated that forms a dead end, the agent may waste time steps discovering that, whilst an observer who can see the obstacles could find a more optimal path that doesn't need to approach the blockage.
There's a difference here between writing an agent to make use of knowledge you as designer has, as opposed to one that learns the hidden state details by itself. It is possible to design agents - DQN or otherwise - that figure out how to represent discoveries in POMDPs by themselves, but it's far easier to code an update to a state/memory yourself if you can and assuming your goal is to solve the environment, not to develop the most sophisticated general learning agent possible.

Thank you for your reply.  According to your explanation, the best way to proceed is to somehow encode this information related to the obstacles' locations into the code? Otherwise I should go with POMDPS? But then in testing the algorithm how can we add this extra knowledge? What if we make the problem really simple and not looking for the shortest path and instead focusing on fertilizing the target fruits and avoiding the obstacles. I think with my current state representation it cannot solve it. The current state includes the agent location and target fruit type.

@Mamora Make the state the agent's current knowledge of the map, including position of any obstacles it has discovered. If the space is very small, and number of obstacles fixed, you can do this by enumerating all possible states (see the Open AI Taxi environment for something similar - here's a worked solution: https://towardsdatascience.com/solving-the-taxi-environment-with-q-learning-a-tutorial-c76c22fc5d8f ). If the environment is larger, e.g. 16x16 grid with 10 or more movable obstacles, then a grid map "image" and CNN for the agent is the way to go.

The environment is quite big like 300*250 cells and one or two stationary obstacles but being located randomly.

So the obstacles are likely to be larger shapes, covering many pixels, and the agent only observe the parts of them that it is adjacent to? 300x250 is a bit large for a starting problem - it may take some effort to tune the CNN and representation. All of this is really different set of questions/answers though - I'd encourage you to try, and perhaps ask again if you get stuck. If I just predict what your challenges are in advance, you miss the chance to try stuff and learn about it even if it doesn't quite work. Plus I could well be wrong

Thanks @Neil Slater for your answer and the effort. The obstacles are covering several pixels and the idea is that the agent only sees the adjacent parts of it. I am still stuck :D mostly for the state representation. I was thinking about how to proceed, one way is that because the fruits' location and the grid size are known in advance, I can give the robot the initial path to do the task and then whenever it encounters an obstacle it should learn to find a detour to finish the task. But then the state representation should be the whole field? It is too big and I want not to use CNN

@Mamora I have successfully coded an agent working in 2D space without using a grid or CNN. Remember you don't have to input the state representation directly to the neural network, you can use feature engineering to make things simpler. Istill have questions though, such as how the agent moves, and you haven't already asked about how to implement your state. Your question will need to give full information about the environment, not general issues you face or are not sure of

The action is four direction which moves the agent one cell in the selected direction and it is deterministic. The state at the moment includes current location (cell coordinates) and % of the target fruits being fertilized. I also added the location of the obstacle, which is a rectangle (2 corners of the obstacle is given to the agent). But still it is not enough. There is not much left about the environment. Agent has to move around to collect the target fruit, not hitting the obstacle. Fertilizing wrong fruit also has punishment but way less than hitting obstacle.


text:
Meaning of a symbol in trace
I am reading this paper Not All Samples Are Created Equal: Deep Learning with Importance Sampling. In the paper there is a deviation shown below.

I can understand everything except the $V$ symbol in the trace. Any help will be appreciated. Note I am only seeking to know the meaning of the symbol in this context.

It could be the variance of the random variable, but this is just a guess. I didn't read the paper. You need to check the definition of the variance and see if things are correct mathematically.


text:
In logistic regression, do I try to fit the graph perfectly or mimimize the error in the predicted probabilities?
In linear regression, I train the model so the graph runs best through the data points, so the geometric distance between f(x) and $y^i$ is minimized.
Now, is it correct that in logistic regression I do not try to fit the graph perfectly through the data but instead try to mimimize the error in the predicted probabilities? Why is that a difference? And what happens when I plot the resulting graph to the dataset. Does it still look like it's perfectly fitted through the data, and why?

I believe you are correct. In linear regression you are predicting a continuous numerical value but in logistic regression you are trying to perform classification tasks so you want to minimize the error but you also don't want to over fit to the data as well.

But then what does it try to minimize if not the distance between f(x)-y

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I apply the z-score normalisation directly?
I want to implement a neural network in Pytorch for medical image segmentation. I should normalise my data.
Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I apply the z-score normalisation directly?

What is the best approach if the dataset comes from a single source of data?

What if the samples come from multiple sources?

There is, beside of numerical losses not difference between directly using z-normalization and first min-max and then z-normalization.
Explanation
Both are affine transformations and a combination of affine transformations is again an affine transformation. Since there is only one(*) affine transformation that results on zero-mean and variance of one, the z-normalization will (kind of) undo all previous affine transformations to achieve this goal.
(*): In fact there are two and you could get the other by multiplying every value with -1. But this is just a silly detail and does not matter for this question.


text:
What are other examples of theoretical machine learning books?
I am looking for a book about machine learning that would suit my physics background. I am more or less familiar with classical and complex analysis, theory of probability, сcalculus of variations, matrix algebra, etc. However, I have not studied topology, measure theory, group theory, and other more advanced topics. I try to find a book that is written neither for beginners, nor for mathematicians.
Recently, I have read the great book "Statistical inference" written by Casella and Berger. They write in the introduction that "The purpose of this book is to build theoretical statistics (as different from mathematical statistics) from the first principles of probability theory". So, I am looking for some "theoretical books" about machine learning.
There are many online courses and brilliant books out there that focus on the practical side of applying machine learning models and using the appropriate libraries. It seems to me that there are no problems with them, but I would like to find a book on theory.
By now I have skimmed through the following books

Pattern Recognition And Machine Learning
It looks very nice. The only point of concern is that the book was published in 2006. So, I am not sure about the relevance of the chapters considering neural nets, since this field is developing rather fast.

The elements of statistical learning
This book also seems very good. It covers most of the topics as well as the first book. However, I am feeling that its style is different and I do not know which book will suit me better.

Artificial Intelligence. A Modern Approach
This one covers more recent topics, such as natural language processing. As far as I understand, it represents the view of a computer scientist on machine learning.

Machine Learning A Probabilistic Perspective
Maybe it has a slight bias towards probability theory, which is stated in the title. However, the book looks fascinating as well.

I think that the first or the second book should suit me, but I do not know what decision to make.
I am sure that I have overlooked some books.
Are there some other ML books that focus on theory?

Some of the books that you mention are often used as reference books in introductory courses to machine learning or artificial intelligence.
For example, if I remember correctly, in my introductory course to machine learning, the professor suggested the book Pattern Recognition And Machine Learning (2006) by Bishop, although we never used it during the lessons. This is a good book, but, in my opinion, it covers many topics, such as variational inference or sampling methods, that are not suited for an introductory course.
The book Artificial Intelligence. A Modern Approach, by Norvig and Russell, definitely does not focus on machine learning, but it covers many other aspects of artificial intelligence, such as search, planning, knowledge representation, machine learning, robotics, natural language processing or computer vision. This is probably the book that you should read and use if you want to have an extensive overview of the AI field. Although I never fully read it, I often used it as a reference, as I use the other mentioned book. For instance, during my bachelor's and, more specifically, an introductory course to artificial intelligence, we had used this book as the reference book, but note that there are other books that provide an extensive overview of the AI field.
The other two books are not as famous as these two, but they are probably also good books, although their focus may be different.
There are at least three other books that I think you should also be aware of, given that they also cover the actual theory of learning, aka (computational) learning theory, before diving into more specific topics, such as kernel methods.

Machine Learning (1997) by Tom Mitchell
Foundations of Machine Learning (2nd edition, 2018) by Mehryar Mohri et al.
Understanding Machine Learning: From Theory to Algorithms (2014) by Shai Shalev-Shwartz et al.

You can find more books on learning theory here.

Pattern Recognition And Machine Learning is a great theoretical book. I don't know anything better on standard ML. I read several pages from it myself and all my colleagues researchers suggest to look there if you are not sure about some concepts. The 2 problems with it are that it's huge and it doesn't cover almost all deep learning models known for today.
So, in addition, I'd suggest you look at Deep Learning by Ian Goodfellow et al.
Your concerns about not studying topology, measure theory and group theory are groundless. These sections of math aren't prerequisites in any way, they aren't even discussed anywhere I know.
Actually, ML theory is more like probability theory and statistics. Especially, statistical learning theory (which is nothing more than probability theory and statistics). I haven't read any books on SLT so have a look at this answer.

"Deep Learning" is not a good book.

@Kevin, why isn't it a good book? I haven't read it yet but I have seen it recommended many times so I am curious.

For a more updated list I'd suggest these:

Deep Learning: Foundations and Concepts (2024) by Bishop (same author for the classic)
Reinforcement Learning: An Introduction by Rich Sutton
Multi-Agent Reinforcement Learning: Foundations and Modern Approaches (2024)
The Principles of Deep Learning Theory
Foundations of Computer Vision (2024)

There's also a list of 27 papers (reportedly shared by Ilya Sutskever with John Carmack).

"If you truly master these, you’ll understand 90% of what matters in modern AI."


text:
What are some examples of functions that machine learning models compute?
My simple understanding of AI is that it is based on a mathematical model of a problem. If I understood correctly, the model is a polynomial equation and its weights are calculated by training the model with data sets.
I am interested to see a few example polynomial equations (trained models) which are used in certain problem areas. I tried to search it, but so far could not find any simple answers.
Can anyone list a few examples here?

Hello. Welcome to AI SE! Can you please cite the article, paper or book that says that "the model is a polynomial equation". We need more context to answer this question properly.

It's just my simple understanding. For example there will be a polynomial equation (hyperplane) that will classify the labelled data-set into n-groups.

I have changed a little bit your question to make it more acceptable. Let me know if this version is still consistent with your original question.

If I understood correctly, the model is a polynomial equation

No, it's not true that all machine learning (ML) models compute (or represent) a polynomial function. For example, a sigmoid is not a polynomial, but, for example, in a neural network, you can combine many sigmoids to build complicated functions that may not necessarily be polynomials.
We usually distinguish between linear (straight-lines) and non-linear functions (rather than polynomials and non-polynomials). In some cases, it is straightforward to visualize the function that your model computes: for example, in the case of linear regression, once you learned the coefficients (i.e. the slope and y-intercept), you can plot the learned straight-line function. In other cases, for example, in the case of neural networks, it is not fully clear how to visualize the function that your model computes, given that it is the composition of many non-linear functions (typically, ReLUs, sigmoids or hyperbolic tangents).
If you are interested in solving problems with polynomials, take a look at polynomial regression.

and its weights are calculated by training the model with data sets.

Yes, in machine learning, we want to find a function that "fits the given data", and the specific meaning of "fitting the data" depends on the specific machine learning technique.
For simplicity, let's focus on supervised learning, a machine learning technique where we are given a labelled dataset, i.e. a dataset of pairs $D = \{(x_1, y_1), \dots, (x_N, y_N)\}$, where we assume that $f(x_i) = y_i$, for some typically unknown function $f$, and $y_i$s are the labels (the outputs of $f$) and $x_i$ the inputs of $f$. The goal is to find function $g_{\theta}$ that approximates well $f$. I will soon describe what the subscript $\theta$ represents.
For simplicity, let's assume that $f$ is a linear function (i.e. a straight-line). So, we can define a linear model $g_{\theta}$ that we can use to find a function that approximates well $f$. Here is the linear model
$$g_{\theta}(x) = ax + b,$$
where

$g_{\theta}(x)$ is the output
$x$ is the input
$ax + b$ is the linear function (a straight-line)
$a$ is the slope (a parameter, aka weight)
$b$ is the $y$-intercept (another parameter)
$\theta = \{ a, b \}$ (the set of parameters of the linear model)

Why is this a model? I call this a model because, depending on the specific values of the parameters $\theta$, we have different specific functions. So, I am using the term "model" as a synonym for a set of functions, which, in this case, are limited by the definition $ax + b$ and the specific values that $a$ and $b$ (i.e. $\theta$) can take.
So, what do we do with this linear model? We want to find a specific set of parameters $\hat{\theta}$ (note that I use the $\hat{ }$ to emphasize that this is a specific configuration of the variable $\theta$) that corresponds to a linear function (a straight-line) that approximates $f$ well.  In other words, we need to find the parameters $\hat{\theta}$, such that $g_\hat{\theta} \approx f$, where $\approx$ means "approximately computes".
How do we do that? We typically don't know $f$, but we know (or assumed) that $f(x_i) = y_i$, so the labeled dataset $D$ contains information about our unknown function $f$. So, the idea is that we can use the dataset $D$ to find a specific set of parameters $\hat{\theta}$ that corresponds to some function that approximates $f$ according to the information in $D$.
This process of finding $\hat{\theta}$ based on $D$ is often denoted as "fitting the model to the data". There different ways of fitting the model to the data that differ in the way they compute some notion of distance between the information in $D$ and $g_{\hat{\theta}}$. I will not explain them here because this answer is already quite long. If you want to know more about it, you should take a book about the topic and read it.

What are some examples of functions that machine learning models compute?

I don't have specific examples, but you can easily try to fit a linear regression model to some labelled data, then plot the function that you found. You could use the Python library sklearn to do that.


text:
What is wrong with equation 7.3 in Sutton & Barto's book?
Equation 7.3 of Sutton Barto book:
$$\text{Equation: } max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi| \le \gamma^nmax_s|V_{t+n-1}(s) - v_\pi(s)| $$
$$\text{where }G_{t:t+n} = R_{t+1} + \gamma R_{t+2} + .....+\gamma^{n-1} R_{t+n} + \gamma^nV_{t+n-1}(S_{t+n})$$
Here $V_{t+n-1}(S_{t+n})$ is the estimate of $V_\pi(S_{t+n})$
But the Left Hand Side of the above equation should be zeros as, for any state s, $G_{t:t+n}$ is an unbiased estimate of $v_\pi(s)$ hence $\mathbb{E}_\pi[G_{t:t+n}|S_t = s] = v_\pi(s)$.

In general, $\mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s)$. $v_\pi(s)$ is defined as $\mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$, so you should be able to see why the two are not equal when the LHS is an expectation of the $n$th step return. They would only be equal as $n \rightarrow \infty$.


text:
Are there neural networks with 3-dimensional topologies?
The topologies (or architectures) of the neural networks that I have seen so far are only 2-dimensional. So, are there neural networks whose topology is 3-dimensional (i.e. they have a width, height, and depth)?

Yes. Convolutional neural networks are usually 3-dimensional. In fact, they usually deal with images (e.g. RGB images), which can already be 3-dimensional.


text:
What is the intuition behind the number of filters/channels for each convolutional layer?
After having chosen the number of layers for a convolutional neural network, we must also choose the number of filters/channels for each convolutional layer.
The intuition behind the filter's spatial dimension is the number of pixels in the image that must be considered to perform the recognition/detection task.
However, I still can't find the intuition behind the number of filters. The numbers 128 and 256 are often used in the literature, but why?

The channel sizes 32, 128, etc. are used because of memory and efficiency. There is nothing holy about these numbers.
The intuition behind choosing the number of channels is as follows-
The initial layers extract low-level features- they consist of edge detectors, etc. There aren't many such features. So, we won't gain much by adding a lot of filters (of course, if we use a 3x3 filters on an RGB image, we would have $2^{27}$ different filters even if our neurons have only 0 and 1 as their values. However, most of them are quite similar/meaningless for our job). Using a lot of filters might even lead to overfitting.
The latter layers are responsible for detecting more nuanced features, like elbows/nose shape from the lower level features extracted previously. So, we might do better if we increase the number of channels. Also, note that the resultant layers become more and more sparse as we go deeper.
Though it might differ in applications like super resolution image, in general, the number of channels stays the same or increases when we go deeper.
A nice experiment would be to try and increase the number of channels until you get no more benefit from it. I believe there was a paper that did exactly this (please cite it if someone remembers). You could even try to visualise the filters at this stage and see if the filters are similar or not.

You say "So, we might do better if we increase the number of channels.", but why is that the case? Why would we need more filters to detect these high-level features? Are there more high-level features than low-level ones? I think that this answer would also be improved, in terms of reliability, if you cite 1-2 papers that use a CNN that has more filters as we go deeper, but the spatial dimensions decrease. For instance, you can take u-net (which has an encoder part that follows the pattern that you describe). I am sure you can find other better examples online.

Yes. Increasing the number of channels won't have much effect since you keep increasing the channel dimension and keep reducing the height and width of the image. It would be useful to look at 1x1 filters that were used to resolve this issue.

A couple of good reads are EfficientNet paper and understanding how CNN's work by referring this blog post


text:
Do the rows of the design matrix refer to the observations or predictors?
I attempt to understand the formulation of dictionary learning for this paper:

Depression Detection via Harvesting Social Media: A Multimodal Dictionary Learning Solution
Multimodal Task-Driven Dictionary Learning for Image Classification

Both papers used the exact formulation in two different domains.
Based on my understanding, in common machine learning, we formulate our matrices, from vectors, as rows to be observations, columns to be predictors.
Given a matrix, $A$:
\begin{array}{lcccccc}
& p_1 & p_2 &  p_3 &  p_4 &  p_5 & \text { label } \\
o_1 & 1 & 2 & 3 & 4 & 1 & 1 \\
o_2 & 2 & 3 & 4 & 5 & 2 & 1 \\
o_3 & 3 & 4 & 5 & 6 & 2 & 0 \\
o_4 & 4 & 5 & 6 & 7 & 3 & 0
\end{array}
So, using a math notation and excluding the label, I can define this matrix, $A = [o_1, o_2, o_3, o_4]  ∈ R^{4×5}$, as $A = [{(1, 2, 3, 4, 1), (2, 3, 4, 5, 2), (3, 4, 5, 6, 2), (4, 5, 6, 7, 3)}]$, and in numpy:
import numpy as np

A = np.array([[1, 2, 3, 4, 1],
              [2, 3, 4, 5, 2],
              [3, 4, 5, 6, 2],
              [4, 5, 6, 7, 3]])

A.shape
# (4, 5)

Am I right?

Based on my understanding, in common machine learning, we formulate our matrices, from vectors, as rows to be observations, columns to be predictors.

The rows (or, in general, the first dimension of your tensor) are typically the observations. For example, in TensorFlow, the first dimension of the input tensor typically refers to the batch size, i.e. the number of observations. If you are using Pandas (a Python library to manipulate data), the rows are typically the observations and the columns are the predictors.
However, in general, it does not really matter which convention you use, provided that you use one of the conventions consistently in your implementation (i.e. you choose one of the conventions and you stick with it throughout all your code, to avoid complexity), and make it clear in the documentation. So, you can have a matrix where either the rows or columns are observations and, consequently, the columns or, respectively, rows are the features (aka predictors or independent variables).
Anyway, it is probably a good idea to be consistent with existing literature and implementations/libraries, so you should probably use the rows for the observations.