text:
How can I predict the next number in a non-obvious sequence?
I've got an array of integers ranging from -3 to +3.
Example: [1, 3, -2, 0, 0, 1]
The array has no obvious pattern since it represents bipolar disorder mood swings.
What is the most suitable approach to predict the next number in the series? The length of the array is about 700 entries.
From where can I start the investigation? (provided that I've got some experience in Python and Node.js, but only a hello-worldish acquaintance with TensorFlow). Which training model might be suitable in this case? How can I chunk the data set properly?

Just to clarify, do you only have 700 observations or do you have n sets of 700 observations?

Thanks for the question! It's 700 observations only

Keep in mind that "no obvious pattern" may actually mean "no pattern at all", in which case the answer to "how can I predict the next number?" is "you can't". It's worth trying, but some sequences will be impossible to predict.

have you seen this recent video from mathloger? They address this question: https://www.youtube.com/watch?v=4AuV93LOPcE

As all you have is a series of numbers, you should try using a sequence model. I suggest you look into RNNs and in particular LSTMs. Of course this is assuming despite the lack of "obvious patterns", there are some kind of hidden patterns in your data. If not, what you have is not very different than random walk in 3 dimensions - which makes the case unpredictable in the first place.

This is a question of time series forecasting, since your numbers form a sequence.
You may want to take a look at the "forecasting" tag at CrossValidated.
If you have only 700 data points, ML/AI methods will likely not be very useful. Whatever you do, I would recommend you benchmark your chosen method against very simple approaches, like the overall mean, or the last observation (a "random walk forecast"), or a simple Exponential Smoothing method. These very simple benchmarks can often be surprisingly hard to beat, and they are trivially easy to set up.
You next step should be to include domain knowledge, as Sanyou recommends. This can be as simple as observing that bipolar mood swings follow a day/night cycle and modeling this seasonality, e.g., in a seasonal Exponential Smoothing method. (I'm not saying this disorder does exhibit this kind of seasonality, only that if it does, this can easily be modeled.) Or model any other kinds of drivers you know.
In my experience, understanding your data and your context always beats building more fancy models, or collecting more data.
As free time series forecasting textbooks, I very much recommend Forecasting: Principles and Practice (2nd ed.) by Athanasopoulos & Hyndman
and
Forecasting: Principles and Practice (3rd ed.) by Athanasopoulos & Hyndman.

I guess the most "suitable" approach is to look up research papers on ML/AI/Stats based methods on bipolar disorder mood swings prediction/regression etc. Focus on the abstract, intro/related works and conclusion. Find out why the method is proposed, what the well-known approaches are, what the intuition for the proposed methods are. Find out the fundamental resources cited on the intro/related works. From the intro and related works, look up the references and skim them.
As for the theoretical basis, the math and the proposed method, just skim them quick, next time you got time/the feels you can deepen them. Utilize sci-hub or lib-gen or similar webs if you/your institution is not subscribed to the publishers. Bonus points: some papers also include github/links to their implementation source code.
Quick search on google scholar with the query "bipolar mood swing prediction machine learning" resulted in cool (at least the titile) research papers. For example The impact of machine learning techniques in the study of bipolar disorder: a systematic review, and Review on Machine Learning Techniques to predict Bipolar Disorder.
Why do we go with this approach? Because your domain is specific, vast and complex it its own way. Most of the time, they already tried the "basics" prediction/regression/classification on your domain and published the methods as well as the results, so you can start from there and gain even more because of the additional knowledges/references from the papers.

Since you only have only 700 observations, I would not try a deep learning approach. I think it is very unlikely that any Deep Learning approach will learn a non-obvious relationship with that little data.
What you could try is create a set of features based on lags. Create a feature, that is lagged by 1, by 2, by 3, and so on. Also moving average of lagged variables could be useful, window of 2,3,5. Standard deviation could also be interesting, though at bit larger windows. And then train a regular ML model.
I would try this simple approach even if I had 10 million observations, and planned on using Deep Learning, so I could use it as a benchmark.

Rather than using hand-crafted features using lags, I'd start with a plain old auto-regressive (AR) model, perhaps then auto-regressive moving-average (ARMA).


text:
Why cannot linear activation functions be used to approximate any function?
In neural networks we use nonlinear activation functions such as sigmoid, ReLU, etc. Using a combination of these functions (with required scaling and shifting), we manage to estimate any nonlinear function.
I understand in theoretically that we cannot obtain a function such as ex1  + x22 ...(and other such nonlinear components of inputs x3, x4, etc.) using only linear combinations of xi's. However, when I am thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear (scaled and shifted) linear lines and I do not understand why this is incorrect.
In case, if it is possible to approximate any function using a ReLU activation function (which is linear in the first quadrant), why is it not possible to approximate with a function completely linear?

However I when thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear ( scaled and shifted) linear lines and I do not understand why this is incorrect.

It's not generally incorrect, but what you are describing there is not a single function but a piecewise function. For example:
$
f(x) = \left\{
        \begin{array}{ll}
            0.1 x & \quad 0 \leq x < 0.1 \\
            0.2 x & \quad 0.1 \leq x < 0.2\\
            0.3 x & \quad ...
        \end{array}
    \right.
$
This composition is non-linear, because the input space gets transformed differently depending on $x$ (because each subfunction in the example above has a different slope). There are methods to optimize such a function as well, e.g. piecewise linear regression.
An MLP without non-linearities is not a piecewise function, but a function composition:
$(f \circ g)(x) = f(g(x))$,
where $g$ and $f$ are two consecutive dense layers.
The difference is that if both functions $g$ and $f$ are linear, then $f \circ g$ is linear. This holds for any configuration of model parameters. Therefore, the model transforms any given input space $X$ into some output space $W$, where straight lines in the input space remain straight in the output space and can thus never model something non-linear.

In case, if it is possible to approximate any function using Relu activation function ( which is linear on first quadrant) , why it is not possible to approximate with a function completely linear ?

As soon as you introduce any non-linearity, the system becomes much more expressive, because each neuron now models a non-linear function. Combining the neurons (as done by a subsequent layer) can learn combinations of these non-linear functions.
And this is where the two worlds collide: If you take a ReLU activation, it allows the model to actually learn something like a piecewise linear function, because of the combinations of several linear functions with different slopes that are $< 0$ only for certain input ranges. Here is a simple example of that:
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

tf.random.set_seed(4)
np.random.seed(4)

# create a dataset
xs = np.linspace(-2, 2, 1000)
ys = xs**2 + 0.2

# create a model
m = tf.keras.Sequential([
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(1),
])

# train the model
m.compile(optimizer='adam', loss=tf.keras.losses.MAE)
m.fit(xs[:,np.newaxis], ys[:,np.newaxis], batch_size=8, epochs=20, verbose=None)

# plot the results
fig, ax = plt.subplots(1, 2, figsize=(15, 6))

ax[0].plot(xs, xs, label='model inputs')
ax[0].plot(xs, ys, label='ground truth')
ax[0].plot(xs, m(xs)[:,0], label='prediction')
ax[1].plot(xs, ys)

for n in range(m.layers[0].units):
    neuron_act = m.layers[0](xs[:,np.newaxis])[:,n]
    ax[1].plot(xs, neuron_act, color='g', label='piecewise function %d' % n)

ax[0].legend()
ax[1].legend()
ax[0].set_ylim((-1, 4))
ax[1].set_ylim((-1, 4))

The left image shows what the model has learned (green line), the right side shows the functions of the individual neurons. The more neurons you add to that first relu-layer, the better the approximation of the ground truth will be.

Thank you very much, great explanation and demonstration. The point is that using linear functions ( exactly, not piece-wise) and combining them in some way, we can never obtain the example function you gave right ?

Right! To achieve that, we would have to partition the function which itself is a non-linearity :)

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What loss function should I use if I only care about the accuracy of one class?
CrossEntropyLoss optimizes the overall classification accuracy as
$$ {n_{\text{correct}} \over N} $$
What loss function should I use if I only care about increasing the true positive rate of one class?
$$ {n_{\text{true A in predicted A}} \over N_{\text{predicted A}} } $$
For example, I predict 100 images to be in class A, and 90 out of this 100 are truly A. So the accuracy is 90%.
In the meantime, I predict another 900 images to be in class B, but 500 of them are actually A, and only 400 are B. So the overall accuracy is (90+400)/(100+900) = 49%.
In the meantime, I don't want $ N_{\text{predicted A}} $ to be too small, since one can see from above that a smaller $ N_{\text{predicted A}} $ can likely lead to larger true positive rate.

Cross entropy can also be weighted, standard practice when training on imbalanced datasets. Instead of the classic formulation
$$C = -\sum_{i=1}^{M}y_i\text{log}(\hat{y}_i)$$
you can rewrite it as
$$C = -\sum_{i=1}^{M}\sum_{c=1}^{N}w_cy_i\text{log}(\hat{y}_i)$$
were $w_c$ represent the weight of each training class. So if you care mostly about the accuracy of a specific class you could boost that specific class by using a large weight for it, let's say 0.8, and 0.2/(n-1) for all the remaining classes.

were → where

The formula here seems to show sum(w_c) as a separate constant factor that can be factorized out

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Using AI to extend an imagine pattern
I have created some nice patterns using the MidJourney tool. I'd like to find a way to extend these patterns, and I was thinking about an AI tool that takes one of these patterns and extends it in all directions surrounding the original pattern.
Just to give you an idea, this is one of those patterns:

Re "imagine pattern": Do you mean "image pattern"?

@PeterMortensen They may very well have meant "imagine", given that Midjourney has an "imagine" command, as described in the Midjourney page linked to in the question.

@NotThatGuy yes, you are right, it was a mistake due to the /imagine command :-)

As Edoardo says in their excellent answer, the task at hand can be approached as an outpainting problem and there's some great tools available to do this.
To throw an alternative into the ring, I'd point to an example in the field of texture synthesis -  Self-Organising Textures built with Neural Cellular Automata.
The theory revolves around teaching a very small neural network to generate an image using learned, local update rules. When given a loss function that compares the style of two images, the model can generate textures that seamlessly extend the original.
Within the Self-Organising Texture article, there's a a Google Colab which allows you to import a target image and train the model to reproduce it. I used your image as the target, and it was able to quickly (<20 minutes) make a model that captured the overall pattern of your image:

There are options for refining the resulting texture with different loss functions, and even exerting a degree of artistic control using relative noise levels in the generation process. One of the creators of the models, Alexander Mordvintsev, has an excellent YouTube channel where he walks through some of these techniques and I'd highly recommend checking it out if you want to pursue using this method. Have fun!

I've tested the link Colab you put in your reply, and it works great, but how do I change the resolution of the generated pattern ?

I'll start by saying that control of these systems is something of an open question! In this case, the biggest resolution bottleneck is when the image is loaded - it's then constrained to 128x128 pixels, meaning that much of the fine detail is never shown to the model. You can change this by upping the value in the imread() call but this will make the training time longer. Alternatively, you could train two separate models, one of the "branches" and one on a zoom in of the "detail" between and use the noise method I linked in my answer to make them work together. Good luck :)

The task you would like to accomplish is referred to as "outpainting". See example below.

Very recently, OpenAI released an outpainting feature that extends the possible operations to perform with their diffusion model DALL-E.
It is also worth to mention the Stability AI Stable Diffusion model infinity extension (from which I took the example GIF image above). The nice thing of stable diffusion is that, unlike DALL-E, it has been publicly released.


text:
Why do you sample twice from the generator during a GAN training step?
Most basic GAN algorithm definitions I found go like this:

Generate
Train discriminator
Generate
Train generator

Like this one: GAN pseudocode
If I'm not misunderstanding, we sample twice from the same generator in a single iteration due to the fakes being generated before the weights are updated for the generator. Can you tell me why we need to sample twice?

By resampling from the generator after each update to the discriminator, you ensure that the generator is adapting to the evolving discriminator and producing diverse and realistic fake samples in response to the current state of the discriminator. If you sampled only once in one iteration, the generator would be updated based on the same set of fake samples used in the previous discriminator update, leading to potential mode collapse where the generator's training in the same iteration after discriminator's training becomes overly focused on a subset of possible outputs and fails to produce diverse and realistic samples and losing its adversarial training nature, essentially just a slowly improved actor always reacting to an unfriendly critic.

Yeah, I arrived at a similar conclusion. The discriminator would have an 'unfair' advantage as it would have experienced the samples already when evaluating the generator loss. I do still wonder how much of an issue it is practically, as I do not have a good intuition about how much of an advantage a single exposure is, especially at low learning rates. But I guess that's a question that no one really tested given that it conceptually doesn't make sense to omit the second sampling anyways.

GAN's generator is always deterministic to directly output synthetic data capturing only their rough distribution without (conditional) probabilities like VAEs, so intuitively mode collapse is a much more serious problem for GAN's generator and a second random noise as exploration in the same adversarial iteration definitely helps.


text:
I don't understand how the KL Divergence works for the stated goal of Inception Score
Inception Score has two goals:

The entropy for the distribution prediced on individual samples should be small(the outputs are specific)
The entropy for the marginal distribution over all samples should be large (the output labels should be balanced)

I understand that this is one reasonable outcome of optimizing for the KL Divergence. But can't I get the opposite as well if I just optimize for that, e.g.

The entropy of the predicted distribution for individual samples is high (the individual samples looks like nothing in particular)
The entropy ot the marginal distribution is large (the generator mostly outputs the same class)

It seems to me that this second outcome is just as likely when just looking at the divergence between the two distributions. Are there any practical constraints that prevent this from happening?


text:
How does Microsoft use AI to make Windows 10 updates smoother
According to this news, Microsoft is using AI to make Windows 10 updates smoother. So I was curious and went further to search and came across this website, which describes:

Artificial Intelligence (AI) continues to be a key area of investment
  for Microsoft, and we’re pleased to announce that for the first time
  we’ve leveraged AI at scale to greatly improve the quality and
  reliability of the Windows 10 April 2018 Update rollout.  Our AI
  approach intelligently selects devices that our feedback data indicate
  would have a great update experience and offers the April 2018 Update
  to these devices first.  As our rollout progresses, we continuously
  collect update experience data and retrain our models to learn which
  devices will have a positive update experience, and where we may need
  to wait until we have higher confidence in a great experience.  Our
  overall rollout objective is for a safe and reliable update, which
  means we only go as fast as is safe.
Our AI/Machine Learning approach started with a pilot program during
  the Windows 10 Fall Creators Update rollout.  We studied
  characteristics of devices that data indicated had a great update
  experience and trained our model to spot and target those devices.  In
  our limited trial during the Fall Creators Update rollout, we
  consistently saw a higher rate of positive update experiences for
  devices identified using the AI model, with fewer rollbacks,
  uninstalls, reliability issues, and negative user feedback. For the
  April 2018 Update rollout, we substantially expanded the scale of AI
  by developing a robust AI machine learning model to teach the system
  how to identify the best target devices based on our extensive
  listening systems.

To me, it sounds like simple if-else statements would have implemented the whole thing without touching the AI; they mentioned that positive experiences include fewer rollbacks, uninstalls, and so on, so we may use these as a criterion of a positive experience.
I am just wondering if the word 'AI' is being misused, or can be misleading in this context? Could anyone point me out on this or give any insight on how AI can be used in this context? In my experience, I have only seen AI mostly being used in speech recognition, image recognition and other sort-of classifying problems, with a training and consequently a computer can "learn" from the data, not like an if-else statement. Today, AI seems to be everything that is considered "smart"?

Yes, "AI" has somewhat turned into a meaningless buzzword, meaning "involves computers"...

Microsoft may well have used a neural network classifier to scan through its telemetry to find "good" update candidates.  It's clearly a classification problem, with potentially lots of parameters and uncertain weighting.  That's what neural nets are good for.

Hint :  According to antlersoft's comment which says

".....Microsoft  may well have used a neural network classifier to
  scan through it's telemetry  to find good  update candidates, it's
  clearly a classification problem.... "

This could be right,however,here questions (information) concerning the company's internal research and development of their software products is classified.
Any machine learning engineer; is not professionally  allowed to discuss nor give out information pertaining the company's product i.e. source codes,plan/design or even the flow of the conceptual idea on how the system is implemented,and this is inline with what the question is all about.
Therefore,if you're passionate about how Microsoft implements artificial intelligence in previous operating system software source codes,then by all means you have to check out its;
BetaArchive Discussion and Collection of Betas and Abandonware;
If you're a serious machine learning engineer,this community can help you as well,it releases them for developers and those who are passionately about open source project contribution.
The archive holds terabytes of files,which you can download for free.
So select which windows files have some machine learning algorithms in!
I wanted to give you a little bit of a glimpse,if you're a machine learning engineer.
For your information; answers to this question, will tend to be almost entirely based on opinions,rather than facts concerning Microsoft's software products.


text:
Is Really "AI" Light Years Away from achieving Cognitive Ability of Human?
Oxford philosopher and leading AI thinker Nick Bostrom defines SuperIntelligence as

"An intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills."

Here is an interesting article, you may like Tech Crunch..
Artificial general intelligence : Wiki
Taking into account current limitations and the amount of progress that has been made in recent years, what is a realistic timeframe to expect an AI that has human levels of cognition?

If certain philosophers are correct, Artificial General Intelligence will be, like fusion power, "always twenty years away".  For the true believers, it is an inevitability, and opinions vary.
It may be most useful to look at the unreliability human predictions in this area.
There was an article in the MIT Tech Review in 2017 that contained this graphic, based on a survey of researchers in 2015:
 SOURCE: Experts Predict When Artificial Intelligence Will Exceed Human Performance (MIT Tech Review)
Most notable is that AlphaGo soundly bested the top human player in Go in March of 2016, years before even the most optimistic expert projections.

Everyone is just guessing, and we still don't if AGI is possible, or merely a myth.

Nevertheless, recent breakthroughs are promising!


text:
Matrix Dimension for Linear regression coefficients
While reading about least squares implementation for machine learning I came across this passage in the following two photos:

Perhaps I’m misinterpreting the meaning of beta  but if X^T has dimension 1 x p and beta has dimension p x K, then hat{Y}  would have dimension 1 x K and would be a row vector. According to the text, vectors are assumed column vectors unless otherwise noted.
Can someone provide clarification?
Edit: the matrix notation in this text confuses me. The pages preceding the above passage read:

Should the matrix referenced not have dimensions p x N, assuming a p-vector is a column vector with p elements?
Note: The passage is taken from “Elements of Statistical Learning” by Hastie, Tibshirani, & Friedman.

The text says Here we are modeling a single output, so K = 1. In that case, \hat{Y} would be a 1 x 1 vector. You could argue that this is not technically a scalar, but this is a minor point. If you are modelling more than one output, K would be greater than 1 and \hat{Y} would be a 1 x K row vector where K is the number of outputs.

@cantordust thank you for your reply. Is this a typical convention, to have Y as a row vector? I only ask because I would expect it to be written as Y^T. But I’m just beginning my reading and haven’t seen applications enough to verify. In a linear algebra context, one would expect Y to be a column vector, but perhaps convention is different here?

I suspect that the reason why they have omitted the ^T is that it doesn't matter for a scalar (a happens to be the same as a^T when a is a 1 x 1 vector). But in principle you are right that it should be Y^T. This is kind of sloppy writing. They have also missed the ^ in the last sentence - that should be \hat{\beta}.

@cantordust I noticed that as well and was left wondering if that beta was different from \hat{\beta} but assumed it was the same. There is another section on the preceding page regarding their matrix notation which I found confusing from a linear algebra context. I’ll perhaps update the question to include it.

Sadly, modern textbooks are rushed out the door without being proofread properly...

@cantordust I’ve added a passage on the matrix notation in the book that I also find confusing. But perhaps it actually is consistent and I just misunderstand the meaning. Maybe those passages can provide more insight?

This is a good example of what happens when you take text out of context.
The passage that was added in the edited question makes a difference, but it's not quite sufficient, and it doesn't help that the notation is all over the place. I found the textbook and the relevant passages (Section 2.3, p. 10-11). Here is a quick attempt at an explanation.
The authors call X a variable (either scalar or vector) with components Xj, but later in the paragraph they refer to Xj as a variable (which I think is the correct notation). Instead of "variable" with "components", think of X as a set of p variables (in the normal sense of a variable, such as temperature, the price of shares, etc.). You can put multiple variables xi (i = 1...p) in a vector X and call each instance of X an observation. In other words, an observation is a set of measured values for all variables xi.
Assume you have made N observations. Arrange your observations in a matrix with N rows and p columns, where each row represents a single observation (an instance of X).
Now also assume that you are trying to find the relation between your input variables xi and a different set of variables yk, called dependent or response variables. In general, the variables xi and yk are measured, and the model is simply trying to extract the relation between the input and dependent variables so you can then predict the latter from the former.
As a side note, observations are usually denoted with superscripts (x(n)) and variables with subscripts (xi) so there is no confusion about which is which. xi(n) is the nth observed (measured) value of variable xi.
In your case, you have a single dependent variable y and p input variables xi (i = 1...p). Assuming that their relation is linear (note: in many cases this assumption is not justified), we can assign weights ("importance") to each variable and try to find out those weights from measurements. In your case, the weights are denoted with betai (so the "importance" of variable xi is betai; note the same subscript).
If you have N observations of your dependent variable y, then you can arrange them in a column, just like the observations for your input variables. Note that we still have a single dependent variable, so essentially y is a scalar, but the N observations of that scalar form an N-dimensional column vector.
Now multiply the N x p input matrix by the p-dimensional column vector of weights beta. What you get is a column vector of N predicted values for y (\hat{y}). The difference of the N-dimensional column vector of predicted values \hat{y} and the N-dimensional column vector y of measured values is the error (which is minimised with the least squares method).
If you have q dependent variables yi (i = 1...q), each of them would generally have its own set of weights beta. In other words, the relation between the input variables and each yi will be different. The dependent variables can also be arranged in a matrix (just like the input matrix), and its dimensionality will be N x q, where q. In that case, the beta matrix will not be a single N-dimensional column vector but an N x q matrix. Each column in the beta matrix gives the relation between the input variables and the qth dependent variable yq. In that respect, a single observation of all variables yi will be a row vector in the y matrix.
I hope that this clarifies things up a bit. In summary, the explanation in the textbook is correct, but the notation is hard to follow and at times plain misleading (as in the case of variable and component at the beginning of the section). Honestly, you can get a much more intuitive explanation of linear regression from the Wikipedia article.


text:
How does Monte Carlo Tree Search UCT exploitation value change based on perspective?
In this blog toward the end, the author writes the following:

For the sake of my question, let’s assume that a terminal state gives a reward of +1 for a win and -1 for a loss.
When the author says “for any two consecutive nodes this perspective is opposite,” does that mean that if $Q_i$ is positive (for example, 4) for player A at a given node, the same node will have the negative of that value for player B (-4 in my hypothetical)?
Do I need to compute two statistics to store the node value (one for each player) or can I simply store statistics for one player and flip the sign at every consecutive node?

In UCT, the value of Q(vi) / N(vi) is bounded between 0 and 1. Normally when applying MCTS to 2-player games, what happens is the following:
N(vi) corresponds to the total number of games simulated in node vi.
Q(vi) corresponds to the total number of games simulated and won in node vi.
So in each simulation Q(vi) will add +1 to the winning player and +0 to the losing player.
In a tree representation of a 2-player game, each level will represent player A and player B alternately. So you will add 1 to every other node. I don't think it makes sense to add negative values to Qi.
So to answer your question:
You only need to store N and Q for each node, where N is the total number of times that node was in the path of a simulated node and Q is the number of times it won, out of those simulations.
I can give you the example of the way I implemented that part, in a board game I saved in the state class a variable which said who the next player was ( 1 or 2). So I would know that in that state (each node is a state) the player who made the last move was the other one (by the formula: player = 3 - next_player), and I would add 1 to the Qi of those nodes.


text:
Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?
In the original prioritized experience replay paper, the authors track $\gamma_t$ in every state transition tuple (see line 6 in algorithm below):

Why do the authors track this at every time step?  Also, many blog posts and implementations leave this out (including I believe the OpenAI implementation on github).
Can someone explain explicitly how $\gamma_t$ is used in this algorithm?
Note: I understand the typical use of $\gamma$ as a discount factor. But typically gamma remains fixed. Which is why I’m curious as to the need to track it.

In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do not usually do this, but it does happen sometimes.
I guess that, from a theoretical point of view, it was very easy of the authors to make their algorithm more flexible/general and also support this (somewhat rare) case of time-varying discount factor. If it had been very complicated for them to support this option, they may have chosen not to; but if it's trivial to do so, well, why not?
Practical implementations will often indeed ignore that possibility if they're not using it, and can avoid including $\gamma_t$ values in the replay buffer altogether if it is known to be a constant $\gamma_t = \gamma$ for all $t$. As far as I can see, in the experiments discussed in this paper they also only used a fixed, constant $\gamma$.

Thanks for your answer. I do recall in Sutton & Barto the generalization to a variable gamma. But I rarely ever see it used (maybe I’ve never seen a real life implementation that wasn’t for research). I guess I forgot!

@Hanzy You may also sometimes see similar things in papers for variable $\lambda$ (in Sarsa($\lambda$) and other TD($\lambda$) algorithms) and/or variable learning rates $\alpha$ (although I suppose having time-varying, in particular decreasing, learning rates is quite a bit more common than variable versions of the other things)


text:
How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?
In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator.
I have read it a number of times and understand it up through the end of the explanation of GARB. The author then goes on to explain how they use backprop to implement this procedure, which is the point at which I stop understanding.

Is there an open source implementation available to look at? I can’t figure out if $g_T$ is actually computed and used or not? And not certain how the internal reward gradient is calculated.
Any insight you can provide would be helpful.
EDIT: after reading it several times and several related papers, I think I have more understanding but not quite there. So my main questions are:
1) are we keeping a full eligibility trace vector with the dimensionality of the vector equal to the number of parameters in $\theta$ (all NN params)?
2) do we use the gradient calculation via backdrop at every step to calculate $g_t$?
3) do we have to maintain $3 * \theta$ parameters, one for theta, one for $e$ and one for $g_t$?
3) what then is the procedure at terminal $g_T$ to update the parameters? A simple element wise matrix operation?
4) How often to update the parameters of $\theta$?

Hi Hanzy! Please, ask one question per post. You're asking too many questions and thus you will make the life of possible answerers harder. Split this post into multiple ones: one post for each of your doubt, unless the questions are very related.

@nbro ok will do. My concern is I don’t know how widely read the paper is, so wasn’t sure if it would be helpful or not to break it up. I’ll edit later today.


text:
Strong and Weak Dominance Table
I have this table, 2 agents and I want to find for each agent if any action is strongly or weakly dominated. This is the table:

Now, i've found a solution but I'm not sure if it's correct. So for let's say agent 1(the one handling rows): 1<2<2, 1<3<4 and 0<3<4 so I don't have a strong dominance.
For agent 2: 1<=1<=1, 2<5<6 and 2<3<4 which means that I don't have strong dominance here too.
Is my logic correct?

Welcome to SE:AI!  We like game theory questions, but could you provide some background on dominance tables?  I assume you are referring to the formal concept of strategic dominance but I'm having trouble intuiting how you model works...

The image link appears to be broken.


text:
What algorithms does stackoverflow use for classifying duplicate questions?
Can I get details about the algorithms used for classifying questions in stackoverflow ("Questions that may already have your answer"). Most of the suggestions I get are nowhere related to the question I have intended to ask.

I'm reasonably sure it's very like a search-engine search: just a keyword search for questions that have the same keywords as your question, and questions with answers that have the same keywords.  Then sort by relevancy score, and take top 3.

I do not believe that StackExchange has published precisely what algorithm(s) they use for that, so we can't tell for sure.
However, in this meta.stackexchange question, you can follow some of the efforts that were undertaken in collecting training data for training such a classifier. The post also links to the "CQADupStack: Gold or Silver?" paper which describes the analysis of such a dataset that comes directly from StackOverflow. You might be able to find interesting literature by browsing google scholars for papers that cite this paper.
There is also another meta.stackoverflow discussion on this topic, where answers link to various community-developed projects / bots for this purpose. Again, not necessarily what is actually used by the StackExchange sites, but likely similar.
Finally, there is definitely a lot of research on performing such classifications (one example found by a quick google search is "Duplicate Question Detection in Stack Overflow: A Reproducibility Study", where many other relevant publications can be found in the list of References). This again does not necessarily lead to be precisely the algorithm that StackExchange happens to use, but many relevant ones, one or more of which they might be using.


text:
How to prevent overfitting in stacked models?
I understand the intuition behind stacking models in machine learning, but even after thorough cross-validation scheme models seem to overfit. Most of the models I have seen in kaggle forums are large ensembles, but seem to overfit very little.

Have you tried training each of the stacking models on separate training data?

yeah , separate models are perfect , but they overfit a lot when stacked ,

It seems weird. I trained a GBDT+LR model: I split training dataset into two subsets A and B. Then, I trained GBDT with A, and LR with B. The stacked model works well. Besides, what models are you training?

random forests  and knn

@thecomplexitytheorist, please may you describe the CV scheme that you are using. For example, are you using the scheme given on the Kaggle blog? In principle that can lead to overfitting (see the 'Stacked Model Hyper Parameter Tuning' section), though I've never experienced it

@Ben "But the meta features in {fold2, fold3, fold4, fold5} are dependent on the target values in fold1. So, the target values we’re trying to predict are themselves embedded into the features we’re using to fit our model"  what does that mean?

Did you try out any of the regularisation techniques available?

The effectiveness of dividing training data and piping divisions into networks for independent training, although possibly an effective workaround for specific cases, is not indicative of a robust solution to fitting excellence across a wide range of input data sets.
As suggested in the comment by varshaneya, over-fitting can be a result of unsatisfactory regularization meta-parameterization, as in a poor setting of the λ regularization parameter in a StackGAN.  All meta-parameters used to tune a stacked architectures should be scrutinized to determine whether its setting could lead to over-fitting.  A few can be eliminated up front.  For instance too high a learning rate at any level of any of the networks in the design can reduce convergence probability, but is not a likely cause for over-fitting.
H. Hutson, S. Geva, and P. Cimiano wrote, in their 2017 submittal to the 13th NTCIR Conference on Evaluation of Information Access Technologies, "Ensemble methods in machine learning involve the combination of multiple classifiers via a variety of methods such as bagging (averaging or voting), boosting, and stacking, to increase performance and reduce over-fitting."  Yet bagging has not produced robust responses to differing data sets in our experience, even when normalized, filtered to reduce noise levels, and redundancy is limited.
Zhi-Hua Zhou and Ji Feng [National Key Laboratory for Novel Software Technology, Nanjing University, China, indicated] wrote, "To reduce the risk of over-fitting, class vector produced by each forest is generated by k-fold cross validation."  Reading their paper, Deep Forest, may give you some causes to evaluate.
Over-fitting is usually the application of too sophisticated a model to which data is fit.  In the world of activated networks, excessive sophistication can be as simple as an excessive number of network layers in one or more of the stacked networks.
Feature extraction up front may be needed to remove complexity from the input which is not only unnecessary but counterproductive to generalization and thus the generation of useful output.


text:
Why studying machine learning is an opportunity in today's world?
I just wanted to gather some perspective on why this is a great opportunity to be able to study machine learning today?
With all the online resources (online courses like Andrew Ng's, availability of datasets such as Kaggle, etc), learning machine learning has become possible.
I understood that you can have high paid jobs; but you also need a lot of work dedication to be good at it, which makes your salary not so attractive! (in comparison to the number of hours you spend to keep up with this fast moving field)
Why it is so desirable to take this opportunity and start learning machine learning today? (community, ability to start a business, etc.)

What sort of opportunity it is depends on how much you want to focus on it.
If you want to be a regular programmer, you might take the time to learn a high level interface for some machine learning tools, such as Tensorflow or Keras. There will be plenty of things you don't know how to do (even within those tools), but you may be able to apply predesigned model architectures to problems. The models won't be as good as one designed specifically for the problem, but it's one more tool in your toolbox, and it's possible you'd be able to get some useful results occasionally without devoting a huge amount of time to mastering the techniques.
But if you want to really focus on machine learning, at the research level, you can potentially tackle problems that existing techniques haven't been able to solve. This is where most of the big projects that you've probably heard of will be happening: self-driving cars, AlphaGo, etc. What you can expect here is a lot of hard work. You will need to develop a fairly deep understanding of the mathematics involved so you can visualize (to some extent) what is happening in the potentially high dimensional spaces involved, identify potential failure modes, and identify models that won't fall into them. It involves a lot of trial and error, failed attempts and gradual improvement before you're able to develop a model that beats the stuff already out there.
It's very rewarding work if you enjoy it. There are well-paying positions in the field, but that's just a bonus if you already enjoy the work, and it isn't enough of a bonus if you don't. In my opinion, going into this for just the money would be a mistake. There are almost definitely other jobs that pay just as well but don't take anywhere near the time investment to become (and stay) competent at them. But if you really want to work in this field for its own sake, and also want to make sure you don't starve while you're doing it, it's absolutely worth it.


text:
Why does ChatGPT not give the answer text all at once?
When ChatGPT is generating an answer to my question, it generates it word by word.
So I actually have to wait until I get the final answer.
Is this just for show?
Or is it really real-time generating the answer word by word not knowing yet what the next word will be?
Why does it not give the complete answer text all at once?

It is just for show.

It also does in fact generate the output 1 token at a time, not "knowing" what the next word will be, because each token added to the sequence changes the probablities for further tokens, and this process is necessarily sequential. However, it is unlikely to be doing a whole round trip between the model running on a GPU and the web interface for each token, that would be very inefficient.

It can also be for inducing us to read it all. Should it dump it all at once, we would be more likely to skim over the answer or even quit. It helps engagement.

@Dr.Snoopy No, it is not just for show. It literally does not know the later tokens when it generates a token, the output is real time.

@VolkerSiegel How do you know that? any references?

@Dr.Snoopy If you use the API, you can specify stream=True if you to not want the server to wait until the whole response is ready, but return it as many small "events". A large completion can take quite a long time, so it makes a big difference wheter you get the first result after few seconds or a minute. On the api level, it would be somewhat absurd to have a built in delay on the server side. And the delay you see in a web interface behaves exactly like the API. https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb

@Dr.Snoopy Another strong argument to expect that the output is an incremental stream like we see is: That is hoe GPT-4 fundamentally works, the output is an individual token, repeatedly. Or even a list of tokens with probabilities what could be the next token. Creating incremental output, token by token is "just what a GPT does", fundamentally. A token is about 4 characters long on average, it can be a single character, or a word that occurs frequently.

ChatGPT is a conversational-agent based on GPT3.5, which is a causal language model. Under the hood, GPT works by predicting the next token when provided with an input sequence of words. So yes, at each step a single word is generated taking into consideration all the previous words.
See for instance this Hugging Face tutorial.
To further explain: while outputting entire sequences of words is in principle possible, it would require a huge amount of data, since the probability of each sequence in the space of all sequences is extremely small. Instead, building a probability distribution over half a million of english words is feasible (in reality just a tiny fraction of those words is often used).
On top of that, there may be some scenic effect, to simulate the AI "typing" the answer.

I think you may be right, given my knowledge of these models. However, ChatGPT could also generate all text (word-by-word or not) before showing it to the user, then show it to the user all at once. So, in a way, this is also for show.

Although the first three paragraphs of your answer are true, I'd be surprised if they were actually the direct cause of displaying the answer one word at a time. More likely, this is just for scenic effect, as per your last sentence, and perhaps also to slow the conversation down so that the server doesn't receive too many requests at all.

@nbro That would be just collecting output on the server side to return it as one document. The API actually does that, it is easier to use in programming when you do not care about the time and would otherwise need to collect the parts yourselves. You specify stream=True in the API for incremental results.

@VolkerSiegel Yes, I've worked with their quite unreliable (slow, often unavailable, etc.) API.

Why does ChatGPT not give the answer text all at once?

Because ChatGPT is autoregressive (=generates each new word by looking at previous words), as Rexcirus mentioned.

Is this just for show?

On https://beta.openai.com/playground, output words/tokens are displayed faster when using smaller models such as text-curie-001 than larger ones such as text-davinci-003. I.e., the inference time does seem to impact the display time.
https://twitter.com/ArtificialAva/status/1624411499375603715 compared the display speed of ChatGPT vs. ChatGPT Plus vs. ChatGPT Turbo Mode and showed that ChatGPT Turbo Mode is over twice faster to display the output, which further indicates that ChatGPT shows its response word by word due to its backend (computation time + autoregressive).

Or maybe the chatbot is just intentionally slowing the conversation down by displaying the words slowly, to avoid the server drowning under the huge number of users. And then if you register for turbo mode then the server gives you higher priority, and displays the words faster, regardless of how fast they are actually generated by the algorithm.

@Stef No, it is all real time output from different models, it is more clear when you use the API directly.

@VolkerSiegel Definitely, we've used the API quite a lot for https://arxiv.org/pdf/2304.05613.pdf and some other projects.

Only if your model is on your browser, so No. It would be taxing for your internet to communicate at word-level, it's partially for the show. Although if data animated at sentence or paragraph level is more believable.


text:
Does layer freezing offer other benefits other than to reduce computational time in gradient descent?
In Deep Learning and Transfer Learning, does layer freezing offer other benefits other than to reduce computational time in gradient descent?
Assuming I train a neural network on task A to derive weights $W_{A}$, set these as initial weights and train on another task B (without layer freezing), does it still count as transfer learning?
In summary, how essential is layer freezing in transfer learning?

Welcome to this SE :) Be sure to look around on the SE for all the questions that others have posted. The following describes some of the things you are asking for: https://ai.stackexchange.com/questions/22963/what-is-layer-freezing-in-transfer-learning

Both are transfer learning approaches, which this Pytorch tutorial explains very well:
https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

In practice, very few people train an entire Convolutional Network
from scratch (with random initialization), because it is relatively
rare to have a dataset of sufficient size. Instead, it is common to
pretrain a ConvNet on a very large dataset (e.g. ImageNet, which
contains 1.2 million images with 1000 categories), and then use the
ConvNet either as an initialization or a fixed feature extractor for
the task of interest.
These two major transfer learning scenarios look as follows:
Finetuning the convnet: Instead of random initialization, we
initialize the network with a pretrained network, like the one that is
trained on imagenet 1000 dataset. Rest of the training looks as usual.
ConvNet as fixed feature extractor: Here, we will freeze the weights
for all of the network except that of the final fully connected layer.
This last fully connected layer is replaced with a new one with random
weights and only this layer is trained.

In summary layer freezing is faster, but it is less accurate (after enough training) than finetuning.

from your post I assume you are having three sub-questions and I will answer them one by one.

For the 1st question, yes, layer freezing reduces the computational cost a lot and also helps the model to keep the many patterns it has learned to recognize things better, c negative to positive samples easier and avoid overfitting if it was trained on a bigger dataset. Said like having a model trained on COCO but now we only want a person detection model, without freezing weights, our model may not generalize as well as it was before due to backprops now just focusing on one class.
The second one, yes! Without freezing but still using pretrained model as initial weights is also counted as transfer learning
Finally, in summary, layer freezing is a very essential trick to have a good model, especially if you want it to work well in the open-world dataset and generalize for as many domains as possible, time-saving and easier for fine-tuning as we now just have to train the final downstream task layers.
Hope this helps :D


text:
Why do we need RL in RLHF?
In RLHF, the reward function is a neural network. This means we can compute its gradients cheaply and accurately through backpropagation. Now, we want to find a policy that maximizes reward (see https://arxiv.org/abs/2203.02155). Then, why do we need PPO to find that policy? Since the reward function is differentiable and cheap to evaluate, wouldn't some gradient descent algorithm such as SGD or Adam be enough?

I don't understand this question. It seems you don't understand the difference between a reward function and policy. A policy tells you how to act. A reward function does not. That should answer your question why we want to learn a policy and just don't stick with a reward function. I am even more surprised by this question given that you answered a question on RL here: https://ai.stackexchange.com/a/13123/2444 (that I asked a few years ago). Maybe I'm just misunderstanding what you're asking here. In that case, maybe your post should be clarified.

@nbro of course I never asked "why we need to learn a policy". Where did you get that from? I asked why we use PPO to learn the policy instead of SGD, given that the reward is differentiable and thus we could easily compute/estimate its gradient. Enigman below got the meaning of my question right.

Let's assume the reward function is differentiable. Then you would use SGD for what? How? You don't explain how you would do that. So, your question is unclear, yes. It's unclear if you have any knowledge of PPO or RL at all from your post. Maybe they got your "question" right, but they had to actually write a essay and answer apparently 4 questions, not just one. So, how come? The answer to your question "why do we need PPO to find that policy?" is simple - we don't strictly need it, but we use it because that's an RL algorithm that works.

Let's define the differentiable reward function as $r(s, a)$. Then you would differentiate it with respect to what? $a$? $s$? To obtain which minimum/maximum? Do we really want to find a minimum/maximum of the reward function? It seems that's your understanding - we just want to optimize the reward function. But that's not the case. You want to find a policy, so find the parameters of a policy. But what's the objective function here? It's not the reward function, actually. That's why I think this unclear. It seems you're mixing the concepts of objective function, reward function and policy.

But there are so many things that I'm trying to assume or hypothesize here. So, that should be a clear sign that your post is unclear, but ok. If they guessed your doubts and satisfactorily answer your questions, great.

Recall that RLHF essentially has three stages:

Step 0: Pretrain a text-generating LLM.
Step 1: Collect human ratings on text, and from that, learn a reward model $r_{\theta}$ which can provide ratings on any unseen text. This model may use parts of the LLM. (This is where you backpropagate through the reward model!)
Step 2: Finetune the parameters $\Phi$ of your text-generating model, $LLM_{\Phi}$ with respect to a frozen reward function $r_{\theta}$ to maximize the rewards.

Q1: In Step 2, why don't we apply gradients through the (frozen) reward function?
It seems there is some confusion around what it means to maximize the rewards. In RL, you think of the the reward function $r_\theta$ as something dictated by the environment --- you can't change it, nor does it make sense to change the reward itself however differentiable it may be. What does make sense is to take actions with our agent's policy, and see how the environment rewards them, and change the agent's policy to "maximize the rewards".
Analogously, the idea in RLHF is to use a (fixed) reward function $r_{\theta}(y)$ which can provide useful ratings of text $y$ generated from an $LLM_{\Phi}$. This can be used to improve the parameters $\phi$ of $LLM_{\Phi}$ into assigning more likelihood to text that are of higher quality.  Concretely Step 2 involves:

(a) making your agent take various independent action(s) according to your current policy i.e., make $LLM_{\Phi}$ generate various text completions, $y_1, y_2, ...$ given a prompt $p$
(b) then looking at the reward(s), $r_\theta(y_i)$
(c) then adjusting $\Phi$ (NOT $\theta$) to increase/decrease the probability of that (series of) action(s) according to the reward.

Hope you see why it does not make sense to differentiate through the reward model itself. You want to use the useful signal from the reward model to improve your generator.
Q2: Why do we need a neural network for the reward function in the first place?
Ideally, a human should provide feedback on all the generated text. That would be infeasible. So we resort to training a reward model that learns from human feedback, and generalizes its ratings to unseen text. Indeed, at this stage of RLHF, we differentiate through $r_{\theta}$. But after that $\theta$ is frozen.
You may still ask:
Q3: Given that we have $r_{\theta}(x)$ where $x$ is the output of $LLM_{\Phi}$, why not pass gradients through $r_{\theta}(x)$ to update $\Phi$?
This is difficult to do because $x$ is a discrete variable. Instead what we do is to change the distribution over $x$ so as to maximize $r_{\theta}(x)$. This does not require "backpropagating through" $r_{\theta}(x)$. Instead, this only requires updating the likelihood of $x$ under $LLM_{\Phi}$, proportional to $r_{\theta}(x)$.

Note: The rest is my older answer, which may not have addressed your original question.
You may still ask, within Step (c) above:
Q4. Why PPO? Why not simple SGD to update the likelihood?
(but I realize this may not be your question?)
PPO is in essence doing a simple SGD, but with some fancy penalties for the sake of stability. The "simple SGD" part of PPO takes care of updating your "policy" so as to maximize your "reward". This update on $\Phi$ (NOT on $\theta$), improves the likelihood of $LLM_{\Phi}$ on text that $r_{\theta}$ assigns higher rating to.
Meanwhile, the "fancy penalty" makes sure that in doing your simple update, your policy remains in sufficient proximity to your "original policy". Hence the name, Proximal Policy Optimization. Without this penalty, the agent may diverge out of instability and start producing garbage as explained in the next question. See https://huggingface.co/blog/deep-rl-ppo for a nice technical discussion of the algorithm.
Q5. What exactly are these instabilities?
Recall that what you have is a reward model trained on a specific kind of data (e.g., meaningful text from a pre-trained LLM). Thus, it is possible that your reward model has no clue how to score text from other distributions: as an extreme case, imagine complete gibberish text. In fact, it's even possible that the global maximum with respect to the learned reward model, lurks in some gibberish/strange space of language where the reward model wasn't trained on. If your LLMs by happenstance produces a gibberish that the reward model thinks is great, the LLM may quickly devolve into fooling your reward model by producing only said gibberish.
Q6. Can't you simulate this with a small learning rate with simple SGD?
I do not have a good answer to this as I'm not an RL expert. My guess would be either that (i) this may require limiting to very small learning rate that makes learning too slow or impossible (perhaps precluding any type of feature-learning) or (ii) it may still be possible to get caught in bad local maxima, since this corresponds to purely maximizing only the reward objective.
Now, going by the title of your question, perhaps you may still ask (or maybe you don't; I did wonder about this when I first came across RLHF for LLMs):
Q7. Isn't this just SGD with some supervision and a fancy penalty? Why is this called RL?
Remember that in supervised learning, you're given an immutable set of labeled input-output pairs, while in RL, you're interacting with your environment by producing your own inputs and collect output values in response to that. Analogously, in standard LLM training, you are given an immutable set of human-generated text for supervision on which you maximize likelihood. In RLHF however, the LLM has the freedom to generate whatever text it may please as in step (a), and gather supervision on that text, as in step (b). This is exactly why it can generate "whatever it may please" and devolve into a gibberish state (which in turn motivates a need to fetter your policy close to the initial policy).

Thank you for the answer. Unfortunately I won't be able to go through it before the weekend, so please bear with me.

No problem. I had to update your answer because I realize you probably had confusion with something else. I left my original answer as it is for the sake of posterity.


text:
Solve the AI alignment problem using (meta-level) AI itself?
If the AI alignment problem is one of the most pressing issues of our time, could AI itself augment our (i.e., human) quest to solve the alignment problem? Or would AI itself actually be counter-productive for such a meta-level goal?

Yes, this is one the most popular approaches today. For instance a narrow AI model is built to capture the preference of the user and then this model is used to dispense reward to a reinforcement learning agent.
The narrow AI model is often called Reward Model or Preference Model.
The recent RLHF training technique used for chatGPT uses this approach.
This can be automated further: https://arxiv.org/abs/2212.08073


text:
How is ChatGPT trained?
According to OpenAI, ChatGPT is trained in a 3-step process.

Are the steps where human AI trainers are involved, i.e. training the initial policy and providing the A>B>C>D grading as training sets for the reward model, the ONLY place where actual knowledge is entered? Are there no other learning steps where human trainers are NOT involved and the model learns from high-quality sources like authoritative texts? The sampled prompts in step 2 are supposed to cover EVERYTHING on the internet, so what happens when the trainers do not know anything about the topic? Did OpenAI invite domain experts to grade specific prompts-answer pairs?

This is impossible to answer as there is no more public information on how ChatGPT was trained.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

I think their prompt dataset contains half the Internet.

Step 2 is (mainly) useful to tune the style, alignement and general feeling of the ChatGPT answers, not the specifics of what is answered.
The actual "knowledge" comes simply from ingesting a large corpora of text as training dataset and using unsupervised learning technques on it. This knowledge has been encapsulated in the GPT3.5 on which ChatGPT is built on. To be super clear: ChatGPT is a specialised version of GPT3.5.


text:
Does Apple DepthPro model reach AGI level
I have played around with StyleGAN and now recently familiarized myself with Apple DepthPro. Now, DepthPro models a 2D frame input to 3rd dimension for the input image i.e generates the depth information map.
But what is this depth information AI based generation called; is it modeling, generating or even perhaps "figuring out"? Also it surely does it better than I could never achieve.
For reference this YouTube video I made https://youtu.be/3TdwTyOnRv4 and am still struggling with the idea what my part, making this all visible the way I want, is.
But truthfully I've yet to decide is the video mentioned above AI generated or not too.

What you're describing falls under the concept of depth map as a subset of computer vision.

In 3D computer graphics and computer vision, a depth map is an image or image channel that contains information relating to the distance of the surfaces of scene objects from a viewpoint. The term is related (and may be analogous) to depth buffer, Z-buffer, Z-buffering, and Z-depth.

The AI task here is called Monocular depth estimation. Here's an example Keras code you can run.

Depth estimation is a crucial step towards inferring scene geometry from 2D images. The goal in monocular depth estimation is to predict the depth value of each pixel or inferring depth information, given only a single RGB image as input. This example will show an approach to build a depth estimation model with a convnet and simple loss functions.

Since it's apparently a specialized application, it's not general in the sense of AGI.

Artificial general intelligence (AGI) is a type of artificial intelligence (AI) that matches or surpasses human cognitive capabilities across a wide range of cognitive tasks. This contrasts with narrow AI, which is limited to specific tasks... The timeline for achieving AGI remains a subject of ongoing debate among researchers and experts. As of 2023, some argue that it may be possible in years or decades; others maintain it might take a century or longer; and a minority believe it may never be achieved.

I like the tone of this answer - you are hired.


text:
Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?
Soft question here.
I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million parameters and the training data could be from 1GB - 1TB, and that the training could take from about a day to a week. Also, there is an open source GPT here.
My question is, if you consider that ChatGPT is trained on a very large and diverse amount of data, you may think a solo project could never compete with it. However, what if you chose a specialized set of training data that was smaller but a much richer, more reliable knowledge base, like only academic science textbooks, or only English literature, or only Python libraries documentation, and so on?
Could it actually be much more useful because it's open-source, you have freedom of use (unlike ChatGPT's heavy behavioral conditioning from OpenAI), and you can choose what kind of knowledge the transformer contains? If the data is smaller but way, way higher quality, could you just make a library of niche GPTs for any topic you are studying?

Yes, with caveats.
Yes: If the data covers a niche and is very rare, you can indeed fine-tune a large model to your needs.
Caveats: Fine tuning a model still require significant compute. Moreover the largest models to start the fine-tuning with (e.g. GPT-3) are not open-source, even though this may change, so you would settle on a older open-source model. And without the expensive RLHF process to train the agent (paying the human labellers), your agent will be very knowledgeable in your niche area, but overall poorly aligned and potentially toxic.

Thank you very much. I think it would be interesting to have total control over the transformer - oversight or deliberate selection of all data it "knows" - not pre-training on say, the web in general, and then fine-tuning with specialized knowledge. I guess I thought the toxicity of a model was (often) just a reflection of toxic material / debris that fell into its training material, trawled in from the internet at large. I still don't have a great intuition about how much data a transformer needs to perform well... they say quality over quantity. What if trained exclusively on like, OED?

If a transformer knew nothing except the entirety of the Oxford English Dictionary, I would hope it must find at least some patterns, and it could be interesting to see at minimum what it is capable of, if it has any interesting use at all.

If you are proposing training from scratch, that would give you the highest quality, but it is going to be expensive (millions). Indeed the network must be large to be good (at least as of today architectures), which means a lot of parameters to train, therefore a lot of compute required.


text:
Is the "Chinese room" an explanation of how ChatGPT works?
Sorry if this question makes no sense. I'm a software developer but know very little about AI.
Quite a while ago, I read about the Chinese room, and the person inside who has had a lot of training/instructions how to combine symbols, and, as a result, is very good at combining symbols in a "correct" way, for whatever definition of correct. I said "training/instructions" because, for the purpose of this question, it doesn't really make a difference if the "knowledge" was acquired by parsing many many examples and getting a "feeling" for what's right and what's wrong (AI/learning), or by a very detailed set of instructions (algorithmic).
So, the person responds with perfectly reasonable sentences, without ever understanding Chinese, or the content of its input.
Now, as far as I understand ChatGPT (and I might be completely wrong here), that's exactly what ChatGPT does. It has been trained on a huge corpus of text, and thus has a very good feeling which words go together well and which don't, and, given a sentence, what's the most likely continuation of this sentence. But that doesn't really mean it understands the content of the sentence, it only knows how to chose words based on what it has seen. And because it doesn't really understand any content, it mostly gives answers that are correct, but sometimes it's completely off because it "doesn't really understand Chinese" and doesn't know what it's talking about.
So, my question: is this "juggling of Chinese symbols without understanding their meaning" an adequate explanation of how ChatGPT works, and if not, where's the difference? And if yes, how far is AI from models that can actually understand (for some definition of "understand") textual content?

You are already anthropomorphizing when you asay it has a "good feeling"!

Considering how poor ChatGPT is at counting (e.g. stating "Here are 5 examples why:" and listing anywhere between 3-7 examples) and graphical AI in general at drawing hands (counting fingers is hard), there's definitely some degree of "without understanding their meaning" indeed.

I'm surprised that most discussions of the Chinese Room thought experiment do not mention the Turing test. If it barks like a dog, smells like a dog and licks my hand like a dog, it's a dog. The beauty of that is that we don't need to get into the nitty gritty details of what a dog "really" is. If it's silicon based, so be it. Saying "but I opened it, it's not a real dog!" entirely misses Turing's elegant point.

We don’t have a concrete model of intelligence. We don’t have a mathematical theory of intelligence. All we have are some thought experiments and ill-defined philosophical concepts. Historically, when humans try to build models of the real world based on their amazing deductive powers without evidence, they come up with stuff like the world being made of someone’s armpit or, at best, Lamarckian evolution.

By the way, 'combining symbols in a correct way' describes what, but learning the language?
Did you follow the Turing Test?
How could you show that one who responds with 'perfectly reasonable sentences' has no understanding of Chinese, or - whatever that might mean in Chinese or English - 'the content of its input'?
I suggest it's not for the Question, but for any Answer that it doesn't matter whether the knowledge was acquired by parsing many examples to get a 'feeling' for what's right (AI/learning), or by a very detailed set of instructions (algorithmic).

The purpose and function of intelligence is to solve problems. There are various mental capacities which enable one to do so more or less effectively, which are measured when attempting to measure intelligence. People are conflating intelligence and self-awareness.

Yes, the Chinese Room argument by John Searle essentially demonstrates that at the very least it is hard to locate intelligence in a system based on its inputs and outputs. And the ChatGPT system is built very much as a machine for manipulating symbols according to opaque rules, without any grounding provided for what those symbols mean.
The large language models are trained without ever getting to see, touch, or get any experience reference for any of their language components, other than yet more written language. It is much like trying to learn the meaning of a word by looking up its dictionary definition and finding that composed of other words that you don't know the meaning of, recursively without any way of resolving it. If you possessed such a dictionary and no knowledge of the words defined, you would still be able to repeat those definitions, and if they were received by someone who did understand some of the words, the result would look like reasoning and "understanding". But this understanding is not yours, you are simply able to retrieve it on demand from where someone else stored it.
This is also related to the symbol grounding problem in cognitive science.
It is possible to argue that pragmatically the "intelligence" shown by the overall system is still real and resides somehow in the rules of how to manipulate the symbols. This argument and other similar ones try to side-step or dismiss some proposed hard problems in AI - for instance, by focusing on behaviour of the whole system and not trying to address the currently impossible task of asking whether any system has subjective experience. This is beyond the scope of this answer (and not really what the question is about), but it is worth noting that The Chinese Room argument has some criticism, and is not the only way to think about issues with AI systems based on language and symbols.
I would agree with you that the latest language models, and ChatGPT are good example models of the The Chinese Room made real. The room part that is, there is no pretend human in the middle, but actually that's not hugely important - the role of the human in the Chinese room is to demonstrate that from the perspective of an entity inside the room processing a database of rules, nothing need to possess any understanding or subjective experience that is relevant to the text. Now that next-symbol predictors (which all Large Language Models are to date) are demonstrating quite sophisticated, even surprising behaviour, it may lead to some better insights into the role that symbol-to-symbol references can take in more generally intelligent systems.

Comments have been moved to chat; please do not continue the discussion here. Before posting a comment below this one, please review the purposes of comments. Comments that do not request clarification or suggest improvements usually belong as an answer, on [meta], or in [chat]. Comments continuing discussion may be removed.

Yes it is a good analogy, as explained nicely by Neil.
Regarding your second question:

how far is AI from models that can actually understand (for some
definition of "understand") textual content?

Here's the catch: how do we know that we (humans) are not simply very sophisticated chinese rooms?
For instance suppose that current AI models are improved so much that their performance is on par to human performance, without the current catastrophic failures, yet they retain the current model architectures. Now you have an apparent paradox: they are indistinguishible from humans and yet you know that they are not "understanding".
Personal guess: It's chinese rooms all the way down.

Isn't this also what Turing was trying to demonstrate with his Imitation Game?

That seems like a pretty big "if". It will probably never be possible for purely language-based models to, for example, produce a correct original proof of an unproven math theorem.

I would not bet on it being impossible, think about AlphaGo's move 37. Also, the average human is probably not able to prove a new math theorem.

@Barmar Not sure about Turing intent, but my understanding is that he considered the Turing test as a definition of intelligence. If you pass it, you are intelligent. Today it is widely considered unsatisfactory, due to many techniques to fool the judges. For instance the AI may play ignorant and forgetful, which is very "human".

@BlueRaja-DannyPflughoeft Coming up with a correct original proof of an unproven math theorem requires (a) creativity (read: randomness), (b) knowledge of what's correct and (c) an ability to chain things together "intelligently". I don't see which of those language-based models would be fundamentally incapable of. Prior to ChatGPT, I probably would've agreed it's extremely unlikely that a language-based model would theoretically be capable of coming up with such a proof. But now I'm not so sure any more. ChatGPT is very flawed, but it does bend and redefine what we thought the limits were.

Disproving anthropic general intelligence does not, alas, bring us any closer to a successful implementation of artificial general intelligence!

"For instance suppose that current humans  are improved so much that their performance is on par to machine performance, without the current catastrophic failures..."; for example in computations, chess, Go, memorization ...

Regarding your final personal guess, it cannot be an infinite regress like turtles all the way down since human's neural system is physical and thus limited and finite in terms of levels of such rule-based symbol manipulations which is supposedly to account for the distinct "hard qualitative" understanding of human beings.

@Rexcirus Note that AlphaGo and its chess equivalents are not a language-based models, but traditional tree search engine augmented by neural networks to help determine where to search. A similar approach could possibly be applied to prove theorems.

@Rexcirus Turing never said he ever considered passing the test to mean the machine is intelligent (whatever that means). The only purpose is to determine whether a machine can accurately imitate a human.

@user253751 I watched a video from facebook's research group on applying AI to solve math problems. The mathematician in the video mentioned how some people have tried using AI to find proofs in a very nonhuman way using what I believe is a mathematical axiomatic coding language called lean. I suppose it is some sort of pattern matching in the space of all connections between mathematical statements but the mathematician did not develop much on this aspect. There was also a paper by the Facebook research team on computing integrals using neural networks but I saw some critiques one that paper

@user253751 seems like some sort of tree search but I am not sure, The paper: https://arxiv.org/abs/2205.11491 and a blog post: https://ai.facebook.com/blog/ai-math-theorem-proving/?utm_source=twitter&utm_medium=organic_social&utm_campaign=blog

Searle's Chinese room is not intended as a functional description of any real-world machine. Searle was a philosopher who created the Chinese room as a thought experiment to show what he considered an absurd conclusion of the computational theory of mind. The intended absurdity is that the person inside the room doesn't understand anything about the inputs or outputs, but that when just looking at the room from the outside, the room (i.e. the system person+dictionary) appears to understand Chinese.
To Searle, it was clear that there was no "understanding" located anywhere here, and so this system was clearly not equivalent to a human consciousness that actually understands Chinese. But "strong AI" computationalists believe that all that matters for consciousness is the inputs and the outputs. Since Searle considers the conclusion that the room is conscious absurd, this thought experiment is supposed to be a refutation of this computationalist viewpoint of consciousness.
ChatGPT and other large language models are not a realization of Chinese rooms. There isn't a human in there who doesn't understand English and instead uses a dictionary or set of rules to translate inputs to outputs.
The point of the Chinese room is that it is clear that a) the human doesn't understand Chinese and b) the dictionary/rules are just a book that isn't conscious in itself either, otherwise it doesn't work as a reductio ad absurdum. The point of the thought experiment is that it eliminates anything to which we could attribute understanding - but indeed one of the replies to Searle was that it was just the room itself that had understanding/consciousness, and the interplay between the human and the dictionary is just analogous to the way different regions of the human brain might interact to produce the overall "understanding".
Instead, large language models consist of a big neural network that transforms the inputs to outputs. It's not two distinct entities like in the Chinese room - "rules storage", i.e. the dictionary, and "rules implementor", i.e. the human - it's one big algorithmic structure whose exact inner workings are often hard to explain for specific use cases. You may or may not assign the ability to "understand" to this network, but there are no identifiable substructures here as there are in Searle's room.
These models, of course, raise much the same questions about consciousness and understanding that Searle's Chinese room does, but the room with its clear two-component structure bears no actual resemblance to how the transformer networks underlying large language models work. You might argue that these models are not conscious in the same way that Searle's room is not conscious, but your argument for why they aren't conscious (or why they don't "understand" the language they're using) needs to be very different from Searle's argument.

If the person can respond with perfectly reasonable sentences, they clearly do have a fairly good understanding of the relation between Chinese symbols. They just don't have an understanding of the relation between those symbols and the real world. It seems far from trivial to conclude that one of those involves understanding and the other one doesn't. If someone spends enough time in a Chinese room, and you then tell them what a few of those symbols mean (if they don't figure it out themselves), that may very well make everything fall into place to have them perfectly understand the language.

@NotThatGuy Note that I am careful to say that Searle says it is clear that there's no "understanding" here, not that this is an indisputable fact. The "Replies" section of the Wiki article I link in the first sentence has plenty of viewpoints that don't accept the Chinese room as an argument against computationalism.

ChatGPT et al. are precisely implementations of Chinese rooms. Symbols go in, rules are applied, symbols go out, where is the understanding? The fact the rules are applied by a high-speed machine and not a human seems irrelevant.

@user253751 That's not a fact, that's your judgement: You are already certain that ChatGPT has no understanding, so you say it's just an implementation of a Chinese room. But a computationalist would say a human is also just a Chinese room: Words go in, a neural network (the brain) processes them, words come out. But e.g. Searle would disagree with this statement. That is, "X is an implementation of a Chinese room" is not a factual description of X, it's based in certain philosophical positions on what it means to be conscious and which things are or are not conscious.

@ACuriousMind Notice that I did not say anything about humans but only about ChatGPT.

Searle did respond to one criticism by saying nothing would change if he memorized all the cards and the rules and just wrote out the responses himself from memory; that's basically your explanation of a large language model as one algorithmic structure.

The chinese room argument is useless because it can be applied to the brain as well. Replace the slit with sensory input, the handbook with the wiring of the brain and the activity of the agent inside the room with neuron activity. In the same way the argument demonstrates that the room has no understanding it demonstrates that the brain has no understanding.
My personal assessment is that LLMs like chatGPT have a true understanding of the domain they were trained on. My reasoning is that the training forced the model to squeeze all the information it can utilize to make its predictions into the limited amount of its network parameters. In this regard the incorporation of understanding is a far more efficient usage of the available space than any other kind of data compression.

As many have stated, the Chinese room analogy is intended to show that any hardware + software instance that relies on rules alone (logical operators on input symbols) cannot be said to have understanding.  It is not a good analogy, and the argument does not apply well to trained neural networks. Neural networks (ChatGPT is actually two trained NNs -- an input-trained NN and output-trained NN) are produced as a result of extensive training (for ChatGPT, unsupervised training to generate a language model, and a couple of stages of supervised training on its output sentences). This training creates many billions of weights that are instantiated in the NN, and these weights are applied across the NN nodes as it 'processes' an input prompt.  From a macro view, the NN code is written in the logic of computer language, so one might conflate this with the logical operations on input signals described in the Chinese Room analogy, concluding that a NN is nothing but logical operations.  This is a mistake, in my view.  A trained NN is different in kind from the purely operational program that Searle described.  By evolving weightings through training, NNs encode an incredibly complex object that Searle could not have imagined when he created the analogy.  Does a neural network then have some kind of understanding? I think that it's clearly not anything like human understanding, but I also believe that it is at least some kind of understanding. There are many aspects to this discussion, to say the least. One main objection to Searle's argument is that if you look at individual neurons in the brain, you will not find understanding there either, but the physical brain does give rise to consciousness in some way (unless you believe in some extra 'magic' that overlays the physical brain -- something that Searle, a physicalist, would not allow).

I suggest it makes all the difference in the world whether 'the knowledge' was acquired by parsing many examples to get 'a feeling for what's right' (AI/learning), or by detailed instructions (algorithmic).
Can you say how the exposition here relates to the Question title?
What point is there but that the AI system should and the algorithmic should not be able to expand its programmed capabilities?
How could anyone's understanding Chinese matter?
Do you know - or know of - anyone who believes ChatGPT understands anything? More importantly, anyone who can explain where most people's understanding comes from?
Do you know of people who know how to choose words based on anything but what they've seen?
Can you look again at '…it mostly gives answers that are correct?
Is that less or more useful/worth-while than the answers most people give?
How could understanding Chinese matter, unless you were specifically asking for translations? Are you?
I suggest that juggling Chinese symbols without understanding their meaning is irrelevant to ChatGPT.
If what you're really Asking is how far AI might be from 'understanding' anything, why not first define 'understanding'?

Do you know - or know of - anyone who believes ChatGPT understands anything? Yes - people who don't understand how current AI works, i.e. most of the planet.

@IanKemp Ho, ho, ho. Was that pure cynicism or was there a useful point hidden in there? If you meant 'most of the planet' might believe that, had they any understanding or had ever heard of ChatGPT or anything similar, please Post some details.

@IanKemp People who know a whole lot about AI, and some about neuroscience, might also wonder about the difference between those things, and whether AI can really be said to not understand things.

Interesting discussion, I stumbled on it reflecting on a ChatGPT experiment I did. I asked it to decrypt the word Hello, using the vignère cypher. When I asked it to provide only the answer, it simply guessed. When I let it explain all the steps it gets to the answer easily. So to link that to the chinese room, it seems that in the case of chatGPT, the man in the middle starts clueless but eventually knows a bit of chinese if there is a strong enough pattern within its answer. Maybe I'm knocking on the wrong door here but I find this result very interesting, how much more powerful could these models be if we could first get them to write out the reasoning we want from them as opposed to simply asking them to reason.
TLDR: ChatGPT is similar to a chinese room but it would seem that the man in the middle has the capacity to "learn chinese" if provided with instructions on how to do so within the string to translate, I find that fascinating!

Here is the link to the full conversation: https://sharegpt.com/c/henBmRG


text:
How was ChatGPT trained?
I know that large language models like GPT-3 are trained simply to continue pieces of text that have been scraped from the web. But how was ChatGPT trained, which, while also having a good understanding of language, is not directly a language model, but a chatbot? Do we know anything about that? I presume that a lot of conversations was needed in order to train it. Did they simply scrape those conversations from the web, and where did they find such conversations in that case?

Can you please focus on one model at a time (either LaMDA or ChatGPT), and ask about the other model in a different post? The answers may change from one model to the other, so it's a good idea to focus, otherwise, you may get different answers, for different ways of training the chatbots, which may be what you're looking for, though.

@nbro While I would really be interested in knowing about all promising approaches, I chose to focus on ChatGPT since that's the only chatbot that I've personally tried (in recent times at least).

This article describes it well: "ChatGPT: How Does It Work Internally?"

The key ingredient is called Reinforcement Learning from Human Feedback (RLHF), that is having humans rate the model answers and use the feedback to guide the model training.

The official blog explains this fairly well.


text:
Is there any AI model that predicts short-term stock price well?
One simple way of predicating short-term stock price is to simply take the average price of the previous few days.
I wonder if there is some AI model/methods which can do better than this using only previous prices as input?

If it existed, it would probably be so widely used that any prediction, say a price rise in 5 minutes, would push up the price so that price increase disappears. See the efficient markets hypothesis. So any such AI model will be subjective and carry risk.

Check the follow up! https://ai.stackexchange.com/questions/39594/are-there-public-examples-of-ai-models-that-predicted-short-term-price-well?noredirect=1&lq=1

Attempt-wise, yes. All big players (e.g. J.P. Morgan) in the stock market who do automatic trading utilize reinforcement learning systems, which may contain components for short-term prediction.
But as @miguelmorin commented, if a simple algorithm ever exists, it would be so widely adopted until any gain from it disappear - that is, totally wrong predictions.
AFAIK, in long run, the only winners in short-term trading are those who can hire dozens of experts to build reinforcement systems, which are hosted on multi-million hardware, listening to thousands of market signals coming in from dark fibers 24/7, plus algorithms which can digest and place orders in less than 1 second (perhaps faster than our eyes seeing a price and our brain perceiving it). So, good luck!


text:
Why is the status of artificial life software so under-developed?
I'm interested in self replicating artificial life (with many agents), so after reviewing the literature with the excellent Kinematic Self-Replicating Machines I started looking for software implementations. I understand that the field is still in the early stages and mainly academic, but the status of artificial life software looks rather poor in 2019.
On wikipedia there is this list of software simulators.
Going trough the list only ApeSDK, Avida, DigiHive, DOSE, Polyword have been updated in 2019. I did not find a public repo for Biogenesis. ApeSDK, DigiHive and DOSE are single author programs.
All in all I don't see a single very active project with a large community around (I would be happy to have missed something). And this is more surprising considering the big momentum of AI and the proliferation of many ready to use AI tools and libraries.
Why is the status artificial life software so under-developed, when this field looks promising both from a commercial (see manufacturing, mining or space exploration applications) and academic (ecology, biology, human brain and more) perspective? Did the field underdelivered on expectations in past years and got less funding? Did the field hit a theoretical or computational roadblock?

Why (or how) does artificial like look promising from a commercial point of view?

It is one of the few ways in which one can sustain a consistent space mining supply chain. Doing the same with individually built instruments would be too expensive and time consuming, while with artificial life you could drop a single probe on a planet to start a mining facility.

You say "It is one of the few ways in which one can sustain a consistent space mining supply chain.", according to whom? Also, what do you mean by "consistent space mining supply chain"?

Here you can find many references: https://en.wikipedia.org/wiki/Self-replicating_machine Plenty of studies have been funded by NASA. By supply chain I mean mining, manufacturing, transportation, refueling and other activities needed to support industrial processes.

By consistent I mean "of relevant size", so not just few probes going around, but a full industrial scale process, in order to serve space tourism and colonization.

You might find this https://www.karlsims.com/evolved-virtual-creatures.html by Karl Sims amusing.  It shows the power of evolution in an artificial life environment.

A new interesting project: https://alien-project.org/

I don’t know for certain, but I can make a guess. This is just my opinion, some others may disagree.
The field of ALife has four branches that I’m aware of:
Self-Organizing/self assembly behavior. This is the application you refer to, another context it’s useful is swarm control (for drone swarms, for example). While this is technically ALife, as far as I’m aware it’s not really where most of the emphasis is. Swarm control and self assembly are seen as “different” problems, as machines that can work together and also build more of themselves is interesting (and potentially dangerous), but is missing out on the diversity, the open-endedness that life on earth has. Much of ALife research is focused on trying to formally define this open-endedness and coming up with systems that achieve that. Self assembly and swarm control are interesting and difficult problems, just different. This leads to the other three sides of ALife research:
Coming up with environments, and running tests on them. This is a constant game of coming up with a definition that seems to capture open-endedness, then coming up with ALife sims that meet that criteria but fall short of our expectations. So new definitions are made and we repeat. Geb is a classic example: Geb has passed pretty much every test so far, but it’s fairly uninspiring to watch. Most of those programs you reference chose a particular ALife paradigm, but that paradigm may not be the right one, and is often disappointing. Because we still haven’t found something that really “looks like life”, new paradigms and programs are constantly being created and abandoned when they fail to work (Or perhaps some would have already worked, but the computing time is too much). That’s what you’re seeing. Without any unifying theory or sim that is really convincing, I suspect it’ll stay this way for a while. And because:

we still haven’t made much progress since Karl Sims in the 90s, or since Geb (this point is debatable)
these sorts of sims don’t really have much commercial use aside from games

the direction of making new simulators seems to be lacking funding and research interest, as far as I can tell. Commercial sim games seem to push the boundary these days.
Fortunately there’s a sub field of cellular automata life that’s pretty interesting, its software is slightly more developed due to the overlap with cellular automata and ease of implementation, and research seems to be progressing there at an okay rate.
Realistically, there seem to be two things people want: novel behavior, and novel bodies. My two cents is that these are separate problems, and achieving both is more expensive than just achieving one. But most of these sims end up not balancing development happening in both of these factors (doing this is very difficult), so one factor develops much further than the other, and this disconnect is disappointing to the sim creator. For example, Geb does behavioural diversity really well, while Karl Sims does body diversity well. Sensitivity to small details like mutation rate or genetic encoding also can be quite frustrating. Fortunately, eventually we’ll sorta get behavioural diversity for free in any sim once RL/AI is really understood well.
The third piece of ALife research I’m aware of is the theoretical side, which right now mostly isn’t really far enough along to warrant practical implementation. One big branch of this is the learning theory side, represented by Valiant’s Evolvability theory and followups. Essentially this talks about what functions are possible to evolve, and using stuff like PAC Learning theory they are able to prove some things. Some of these models are more natural than others, but it’s an interesting perpendicular approach to coming up with sims and seeing if they do what we want. Maybe eventually these two approaches will meet in the middle at some point, but they haven’t yet.
The fourth piece is Artificial Chemistry. I recommend this paper as a somewhat dated overview. While this is technically a field of ALife, and is centered around understanding a chemical system that has the necessary emergent properties, it has broken off into applications that may have industrial relevance. For example, robust self repairing and self assembling electronic systems, DNA computing (DNA is capable of simulating arbitrary chemical reaction networks which are capable of arbitrary computing), and artificial hormone systems for automatic task assignment. This has some software developed, but much of that software isn’t really considered ALife anymore since it has branched off into its own domain.