text:
How come machines have become intelligent?
I was studying AI when a question came to my mind.
Basically, I still can't figure out in what AI differs from traditional programming. I don't mean to doubt the cleverness of its solutions, methods and applications, but they just seem to me the natural evolutions of machines performing more and more complex tasks, not a disruptive breakthrough with respect to the past. The improvements and progress are crystal clear, but I don't find equally clear how, when and where the 'intelligence' begins.
Algorithms and more in general IT have always tried to internalise into machines human intuitions. The technological progress, indeed, has made it possible to deal with formerly intractable amounts of data, and to do it with more power and efficiency, allowing the implementation of more and more human-specific abilities, to but this also seems to me more a progress then a change of paradigm, nothing I would call 'intelligence' unless having to use that name for the whole stories.
Basing on my perpexities, I ask: what (in your opinion or according to authoritative experts) justifies the use of the word 'intelligence' paired with 'artificial'? how to recognise, at least roughly, the borders between this technologies and the former ones? I know it's a general question, so everyone feel free to decline it according to his/her sensitivity to (and expertise of) the topic.

Indeed this is a hotly debated question in the field of AI/ML and basically your current view is based on the more traditional intelligence amplification (enhancement) paradigm where AI is a process for creating smart tools and the ultimate goal is not to simulate or exceed human but instead to help humans get stuff done more easily, and indeed under such view it's not that different from all other technologies.

Intelligence amplification (IA) (also referred to as cognitive augmentation, machine augmented intelligence and enhanced intelligence) refers to the effective use of information technology in augmenting human intelligence. The idea was first proposed in the 1950s and 1960s by cybernetics and early computer pioneers... AI has encountered many fundamental obstacles, practical as well as theoretical, which for IA seem moot, as it needs technology merely as an extra support for an autonomous intelligence that has already proven to function. Moreover, IA has a long history of success, since all forms of information technology, from the abacus to writing to the Internet, have been developed basically to extend the information processing capabilities of the human mind.

However, with the advent of big data and tremendous hardware/software improvement of deep learning some multi-modal 'intelligent' systems are believed to be able to make decisions on their own without a human in the loop essentially becoming a dominant or even dangerous role warned by many experts, and this is believed by many to be the path towards artificial general intelligence (AGI). For example, a robot equipped with advanced NLP, computer vision and physical movement continuous learning components can learn its own reward from its environment by observing human behavior known as inverse reinforcement learning which could be a step towards AGI. These systems' learning rules are not explicitly coded from top down functionally specified business logic like most other software only addressing problems in some narrow scope, they're expected to solve many (maybe simple) general problems under many quite different environments.

There are many different ways which make AI different from traditional programming.
Traditional programming is mostly of hardcore fixed logic based mostly on conditions or arithmetic operations. If-else, loops, conditional operators, ternary operators, exponential operations etc are all based on conditional logic or arithmetic operations involving mathematics. But in real world there are many instances in which you need to take decisions which don't have pre-defined use cases. Let's go through the examples below -

An image is basically 3 dimensional array of pixels. Now if you are given a task of identifying car in the image, there are thousands of unpredictable use cases. A car can be of any color, any size (small or big depends on how far car is from the camera), any shape (SUV, sedan, hatchback etc.). Now let's try to solve this using traditional programming tools. How many conditions you can code? The car can be of any color (multiple range of values of pixels in the array), any size (multiple small sub-array sizes), any shape (you can't fix the shape of car array - rectangular, triangular, quadrilateral etc.). On what conditions you will tell your computer to find the car in image? Its impossible using conditions. This is where artificial intelligence steps in. Convolutional networks CNN's, multi layer perceptrons etc layers in neural networks learn the weights and biases which enables the neural network to learn these unpredictable conditions and identify car in your image and draw a beautiful bounding box around it. Artificial intelligence has mathematical calculations but they are nowhere similar to conditional and traditional arithmetic.

Similar concept can be applied to Natural Language processing as well. For example if I tell you to complete the sentence "Plastic is _____ for ____". There are multiple options to fill the blanks - Plastic is harmful for environment, Plastic is used for manufacturing, Plastic is bad for health etc. There are many such combinations that can go to fill the blanks. Lets try to solve it using traditional programming. Which all combinations of words from vocabulary can be used to solve this question? How can you create some logic to handle all such questions to fill the blanks for such problems like - "Programming is ___". Which all contexts you need to consider? Its basically impossible to solve for universal use cases using traditional programming tools of conditional operators and arithmetic. Models like GPT can understand the context using embeddings, attention layers etc. and use it to fill those words for you.

Traditional programming and AI can be said to be analogous to linear algebra and calculus. Linear algebra is used for fixed logics and numbers and calculus can be used to solve problems involving ambiguity and dependance.
This justifies the word "intelligence" (it can handle all possible universal use cases once is is sufficiently trained). Artificial, because it was made by humans after all and the above examples I guess are sufficient enough to explain the boundary between traditional programming and AI.
Cheers!! Thank you!


text:
How to choose the right size LLM for inference to balance latency and accuracy, among several LLMs from 3B through 140B?
I have multiple instances of a same LLM architecture in different sizes from 3B through 72B. I want to build a system that simple questions will be sent to smaller models, and hard questions to bigger models.
How to design such a system? Thanks.
It is ok if it requires to send the question to the 3B model at first sometimes to accumulate experiences.

The most straightforward solution would be to define "difficulty" as the accuracy of a model on a dataset. If you evaluate all models on a dataset, then you can create subsets of data where e.g., all models can answer accurately, only the k-largest models can answer accurately, only the largest model can answer accurately, etc.
This then gives you a supervised classification task where the goal is to map inputs to the difficulty of the sample. You could train a model to do this or try to manually come up with some heuristics.
Another option is to look at current research on LLM calibration/uncertainty. There's a lot of work on getting "confidence" estimates from LLMs (and deep learning models, more broadly). You can then start with the smallest model, then move onto larger models only if the current prediction is deemed unconfident. This kind of stuff is definitely in the realm of on-going research rather than a technique that'll be reliable out of the box, but it could be something to look into.

Thanks @alexander-wan, your suggestions sound reasonable.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
collaborative filtering using linear regression
Currently doing andrew ng's unsupervised learning specialization, I came across this algorithm for collaborative filtering:

here the Xi refers to feature vector of objects(ex: action in movies, romance in movies), Wi is the weight vector defined for each user, so their dot product gives us Yi (rating given user and movie)
this particular descent formlae is used when features of movies/objects is NOT GIVEN
the idea, as I understand is to iteratively update weight and input after randomly initializing both.
I am struggling to understand how this would even work practically, in linear regression we have:

WX + b = Y

which defines a relation between given independent and dependent algorithm. In the algorithm, aren't we estimating the independent feature using weights ? (in the case where we only have rating given to us)
The output of weights to my understand is just random, depending on how we initialize X
this is the descent algorithm provided:

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
My loss is increasing instead of decreasing when i use a regularizer, but if i don't use regularizer then it stays at 00000e+00 or something
This is my model architecture:
activation = LeakyReLU(negative_slope=0.1)  # You can adjust the slope (alpha) as

needed

# Text input
text_input_layer = Input(shape=(500,))
embedding_layer = Embedding(max_words, 50)(text_input_layer)
normalized_embedding = BatchNormalization()(embedding_layer)  # Add BatchNormalization layer
replace_zeros = Lambda(lambda x: tf.where(tf.equal(x, 0), 0.001 * tf.ones_like(x), x))(normalized_embedding)
replace_nan = Lambda(lambda x: tf.where(tf.math.is_nan(x), 0.001 * tf.ones_like(x), x))(replace_zeros)
text_layer = Conv1D(256, 3, activation=activation)(replace_nan)
text_layer = MaxPooling1D(3)(text_layer)
text_layer = Conv1D(256, 3, activation=activation)(text_layer)
text_layer = MaxPooling1D(3)(text_layer)
text_layer = Conv1D(256, 3, activation=activation)(text_layer)
text_layer = MaxPooling1D(3)(text_layer)
text_layer = Flatten()(text_layer)
text_layer = Dense(512, activation=activation)(text_layer)

# Image input
image_input_layer = Input(shape=(256, 256, 1))
replace_zeros_image = Lambda(lambda x: tf.where(tf.equal(x, 0), 0.001 * tf.ones_like(x), x))(image_input_layer)
replace_nan_image = Lambda(lambda x: tf.where(tf.math.is_nan(x), 0.001 * tf.ones_like(x), x))(replace_zeros_image)
image_layer = Conv2D(256, (3, 3), activation=activation)(replace_nan_image)
image_layer = MaxPooling2D((2, 2))(image_layer)
image_layer = Conv2D(256, (3, 3), activation=activation)(image_layer)
image_layer = MaxPooling2D((2, 2))(image_layer)
image_layer = Conv2D(256, (3, 3), activation=activation)(image_layer)
image_layer = MaxPooling2D((2, 2))(image_layer)
image_layer = Flatten()(image_layer)
image_layer = Dense(512, activation=activation)(image_layer)

# Concatenate text and image branches
concatenated_layer = Concatenate()([text_layer, image_layer])
#kernel_regularizer=l1(0.001)
# Dense layers
dense_layer = Dense(512, activation=activation)(concatenated_layer)
dense_layer = Dense(1024, activation=activation, kernel_regularizer=l1(0.0000001))(dense_layer)
dense_layer = Dense(2048, activation=activation)(dense_layer)
output_layer = Dense(1, activation='sigmoid')(dense_layer)

# Define the model with two inputs
model = Model(inputs=[text_input_layer, image_input_layer], outputs=output_layer)
model.summary()

Here is my compiling section and then running it:
optimizer = RMSprop(learning_rate=0.1, clipvalue=1.0)
loss_function = CategoricalCrossentropy()
model.compile(optimizer=optimizer,
              loss=loss_function,
              metrics=[CategoricalAccuracy()])

Am still new to two-input layer-type architecture
Also here is the shape of my data:
print("Text Data Shapes:")
print("Training:", x_train_text.shape, y_train_text.shape)
print("Validation:", x_val_text.shape, y_val_text.shape)
print("Test:", x_test_text.shape, y_test_text.shape)

print("\nImage Data Shapes:")
print("Training:", x_train_image.shape, y_train_image.shape)
print("Validation:", x_val_image.shape, y_val_image.shape)
print("Test:", x_test_image.shape, y_test_image.shape)
Text Data Shapes:
Training: (5000, 500) (5000,)
Validation: (2000, 500) (2000,)
Test: (1000, 500) (1000,)

Image Data Shapes:
Training: (5000, 256, 256) (5000,)
Validation: (2000, 256, 256) (2000,)
Test: (1000, 256, 256) (1000,)

Here the text data was tokenized, as such the shape is like that.


text:
i have a combined pandas dataframe X_train with 22200 samples and 3 features. how can i model this
more info on how data is generated: A signal is passed to a concrete specimen while increasing the frequency of the signal
conductance,susceptance of concrete is measured.The experiment is performed on concrete upto 60 days.The individual day’s concrete strength was obtained by destructive experiment.
how can i model a ml model with this kind of data for predicting a random day concrete strength based on sensor data on that day. sensor data generated for each day is in shape (3700,3), again the strength value for random day is only available we donot have every day concrete strength value.And strength value increases over days
here's the glimpse of dataframe


text:
Shape mismatch error in a multi-input model with Keras API
My intention is to have a classification model where different types of input could have individual hidden layers before being concatenated downstream. I get the error (below) stating that my input shapes are different from expected when I add an extra hidden layers with number of nodes larger than the input dimension of the corresponding input branch:
ValueError: Input 0 of layer "model_21" is incompatible with the layer: expected shape=(None, 3), found shape=(None, 2).
My understanding is that after I add an extra layer with more nodes the dimension of the input dimensions for that branch should not change (?). The extra layer for each branch is commented in the code snippet I provided; in this case, the model runs/trains as expected. Please, point me to what I am missing including. My semi-pseudo-code:
# dim_meta & dim_vec - are 1D dimensions of data
def create_multiinput_model():
    dr = 0.2
    data_inputs = []

    # branch meta
    input_m = keras.layers.Input(shape=(dim_meta,))
    input_m = keras.layers.Dense(int(dim_meta*1.5), activation='relu', kernel_constraint=MaxNorm(3))(input_m)
    input_m = keras.layers.Dropout(dr)(input_m)
    # n_nodes = int(dim_meta*1.5)
    # input_m = keras.layers.Dense(n_nodes, input_dim=dim_meta, activation='relu', kernel_constraint=MaxNorm(3))(input_m)
    # input_m = keras.layers.Dropout(dr)(input_m)
    data_inputs.append( input_m )

    # data branches
    for i in range(3):
        input_d = keras.layers.Input(shape=(dim_vec,))
        input_d = keras.layers.Dense(dim_vec, activation='relu', kernel_constraint=MaxNorm(3))(input_d)
        input_d = keras.layers.Dropout(dr)(input_d)
        # n_nodes = int(dim_vec*1.5)
        # input_d = keras.layers.Dense(n_nodes, activation='relu', kernel_constraint=MaxNorm(3))(input_d)
        # input_d = keras.layers.Dropout(dr)(input_d)
        data_inputs.append( input_d )

    merged = keras.layers.Concatenate()(data_inputs)
    merged = keras.layers.Dense(int(dim*1.5), activation='relu', kernel_constraint=MaxNorm(3))(merged)
    merged = keras.layers.Dropout(dr)(merged)
    output = keras.layers.Dense(1, activation='sigmoid', )(merged)

    model = keras.models.Model(inputs=data_inputs, outputs=output)
    model.compile(
        optimizer=keras.optimizers.Adam(learning_rate=1e-3),
        loss="binary_crossentropy",
        metrics=["accuracy"],
    )
    return model

train_data = [x_train_m, x_train_d1, x_train_d2, x_train_d3,]
test_data = [x_test_m, x_test_d1, x_test_d2, x_test_d3,]

bs = 256
model = create_multiinput_model()
model.fit(x=train_data, y=y_train, validation_data=(test_data, y_test), epochs=300, batch_size=bs, callbacks=[early_stopping])
```


text:
Preparation of multivariate time series data
I am doing a university project on index/stock price prediction.
I plan to use a combined cnn-lstm model, and I have several different types of data: Open High Low Close Volume, values, fundamental data such as unemployment and various rates, technical indicators like RSI, MACD and others, and moving averages like SMA, EMA, WMA and etc. At this moment I am using the following transformations

for these OHLCs - simple differentiation
for fundamental data - logarithmization
for moving averages - subtract the candle opening value from the value of this moving average
indicator values unchanged

Then I use StandardizeNormalizer for all dataset.
I also tried normalizing (robust scaling, standardization, minmax scaling too) each sequence separately, and differentiating all the data, but it was not effective
And i think my current approach is not effective either. What is the best way to prepare this type of data for the network?

Just as a thought, wouldn't things like the moving averages be redundant data? These are derived from the OHLCs which I'm assuming you are also supplying to your model. I believe including these just adds noise to the data. Do you agree?


text:
Is a single hour enough for Google AutoML training 1000 images with single objects in it?
I am a newbie AI practitioner. Please don't be too harsh.
I'm currently looking at the pricing for AutoML for Google Vision Vertex AI. The pricing is in per hour. Let's say I have 1000 image files. Each image file contains only 1 object. I want to use these images for training a model in AutoML.
Is a single hour enough for these 1000 images for the training?

You're missing a lot of detail here. For instance, how many object categories do you have? What exactly is the task you're trying to do anyway? Also, what's stopping you from just trying and see how far you come?


text:
Why are neural networks optimized instead of just optimizing a high dimensional function?
I know that neural networks are universal approximators when given a sufficient number of neurons, but there are other things that can be universal approximators, such as a Taylor series with a high enough order.
So, my question is that, why can we not just optimize some high parameter count (or high dimensional) function instead? I am using a Taylor series just as an example, it can be any type of high dimensional function, and they all can be tuned with Backprop/gradient descent. I know there is lots of empirical evidence out their proving neural networks to win out over other types of functions, But I just cannot seem to understand why this is. Why does something that vaguely resembles real neurons work so well over other functions? What is the logic?
PS - Maybe a dumb question, I am just a beginner that currently only sees machine learning as a calculus optimization problem :)

You're asking many questions here. What is your main question?

Main question is why prefer neural networks over any old function

You might like this arXiv paper by Chang (though Matloff seems to be the author most associated with it): Polynomial Regression As an Alternative to Neural Nets.

In theory there're many universal approximators, but in practice polynomial or Fourier series are too rigid globally while too smooth locally, you'd better need many more units than a MLP for natural problems.

First of all, a neural network can easily be described as some high dimensional function:
$$A_2\sigma(A_1 \sigma(A_0 h_0 + b_0) + b_1) + b_2$$ where theta describes our parameters $A_0 \cdot \cdot \cdot A_k$ and $\sigma$ is some activation function like a relu or a sigmoid.
For the answer to your second question, as to why we use neural networks instead of other methods, is that we don't really know why neural networks work, they just seem to work well empirically better than other methods despite the theoretical evidence. In fact if you ask top figures in this field, you'll pretty much get the same the answer

$p(x \mid \theta) = A_2\sigma(A_1 \sigma(A_0 h_0 + b_0) + b_1) + b_2$, this is very confusing, it seems like the NN is the PDF of a density, which is definitely not


text:
No matter how I change a loss function I get it equal to infinity
I am a bioinformatician, and at the moment I am working with a dataset containing ~12.3 million mutations for ~5500 individuals. The goal is to perform binary classification. I use this framework to create the model, it's called GenNet.
The problem is that my dataset is severely imbalanced, it has a ~9:1 ratio. Controls and cases are annotated by 0 and 1 respectively.
The current iteration of the loss function looks like this, added assertions and print statements to monitor what is happening. I even used the unweighted Keras BCE function in the hope that it would help me to deal with inf values and added weights to it:
def check_for_nan_or_inf(tensor, name):
    tf.print(name, tensor)
    tf.debugging.assert_all_finite(tensor, f"{name} contains NaN or Inf values")

def weighted_binary_crossentropy(y_true, y_pred):
    y_true = tf.cast(y_true, tf.float32)
    y_pred = tf.cast(y_pred, tf.float32)
    y_pred = tf.clip_by_value(y_pred, tf.keras.backend.epsilon(), 1 - tf.keras.backend.epsilon())

    # Use tf.print for printing tensor values in graph mode
    tf.print('*' * 100)
    tf.print('Y_TRUE:', y_true)
    tf.print('*' * 100)
    tf.print('Y_PRED:', y_pred)

    # Create weights based on the class (y_true values)
    weights = (y_true * (weight_positive_class - weight_negative_class)) + weight_negative_class
    weights = tf.cast(weights, tf.float32)  # Ensure weights are float32
    tf.print('*' * 100)
    tf.print('WEIGHTS:', weights)
    tf.print('*' * 100)
    tf.print("WEIGHT FOR POSITIVE CLASS:", weight_positive_class)
    tf.print('*' * 100)
    tf.print("WEIGHT FOR NEGATIVE CLASS:", weight_negative_class)

    # Calculate the binary crossentropy
    bce = tf.keras.losses.binary_crossentropy(y_true, y_pred)
    bce = tf.cast(bce, tf.float32)  # Ensure bce is float32
    tf.print('*' * 100)
    tf.print('BCE:', bce)

    # Check for NaN or Inf values
    check_for_nan_or_inf(y_true, 'Y_TRUE')
    check_for_nan_or_inf(y_pred, 'Y_PRED')
    check_for_nan_or_inf(weights, 'WEIGHTS')
    check_for_nan_or_inf(bce, 'BCE')

    # Apply the weights
    weighted_bce = tf.reduce_mean(bce * weights)
    tf.print('*' * 100)
    tf.print('WEIGHTED_BCE:', weighted_bce)

    # Check for NaN or Inf values in the final loss
    check_for_nan_or_inf(weighted_bce, 'WEIGHTED_BCE')

    return weighted_bce

Here is an example of weighted_binary_crossentropy prints:

Y_TRUE: [[0]
         [0]
         [0]
         ...
         [0]
         [0]
         [0]]
            *****************************************************************************************

Y_PRED: [[0.510742188]
         [0.506835938]
         [0.512207031]
         ...
         [0.514648438]
         [0.513183594]
         [0.512207031]]
            *****************************************************************************************

WEIGHTS: [[0.566014647]
          [0.566014647]
          [0.566014647]
          ...
          [0.566014647]
          [0.566014647]
          [0.566014647]]
            *****************************************************************************************

WEIGHT FOR POSITIVE CLASS: 4.2870369
            *****************************************************************************************

WEIGHT FOR NEGATIVE CLASS: 0.566014647
  *****************************************************************************************

BCE: [0.714865446 0.706913114 0.717864 ... 0.722881556 0.719867945 0.717864]

*****************************************************************************************

WEIGHTED_BCE: 0.569198668

Judging by the print statements and assertions everything is relatively normal, at least returned weighted_BCE is not infinity. But somewhere downstream the loss function becomes infinity and I cannot figure out where and why. I contacted the author but he remains silent, unfortunately. The loss function works fine on a test dataset that came with the framework.
Also, just in case, I am copy-pasting the command I run from jupyter notebook to launch the framework.
%%timeit

weight_positive_class = (1 / 648) * (4908 + 648) / 2.0
weight_negative_class = (1 / 4908) * (4908 + 648) / 2.0

%run GenNet/GenNet.py train -path /home/aydar/DataspellProjects/GenNet/GenNet/processed_data/01_data_prepared_for_analysis_final/ -out /home/aydar/DataspellProjects/GenNet/results/ -bs 16 -lr 0.002 -ID 2 -epochs 50 -wpc $weight_positive_class -wnc $weight_negative_class -mixed_precision -workers 12

As I figured out the main model training module is this one. The generators that provide data for training_network.py are defined here. Also, Create_network.py module seem important, but I can't figure out the exact way it is used. Please help, as I am debugging code line by line at the moment and it is just painful. I would appreciate it. Thanks.

did you check if any of y_pred is extremely close to 0 or 1 and is a wrong prediction?


text:
Structure and parameters of modern AI-baed chess engines
I understand some recent chess engines (like alphazero or muzero) are based on neural networks.  This question is not specific to chess, any other game (e.g. go) would do, but I keep chess for concreteness.
I am not interested how these engines are trained but how they "look like" after training.  (For concreteness, it suffices for me that they give a "score" for a given board position.)  Somewhat similar questions have been asked here but I can't find what I need.

Is it correct to view them as a neural network?  Or, is there some other major component like search trees or other type of algorithm.  I'm assuming that answer here is "yes" in an approriate sense.  Note: The alpha zero paper talks about various forms of tree search.  It is not clear to me how they are used to play, and from the appendix apparently some other engines use very limited search, like depth 2?  That would be already interesting to me.

What is the depth of the network?  Any reference to precise parameters?

What is the input and output format of these networks?  Is the input just an array with pieces positions (excluding three-fold repetition rules and similar secondary issues).  What is the output?
Some info is here but I can't quite understand how it works specifically e.g. for chess.

What activation functions are used?

This is far too broad.

Please, vote to close it as "needs more focus" @Chenmunka

Hello Manu. You're indeed asking too many questions in the same post. It's perfectly fine to have multiple questions and you questions seem interesting, but you should split this post into multiple ones, one for each question, unless they are really very related. As a rule of thumb, if you can't write your title as a specific question, then it means you need to split your post.

OK.  I put another question here https://ai.stackexchange.com/questions/45750/how-do-neural-network-chess-engines-select-moves


text:
How a MLP learns from images and boundaries of its capabilities
(Sorry, I'm not very experienced with ML and I apologize if these questions are vague, naive, or not written well. I also moved this from CV to here.)
I think I understand that a MLP learns (via forward pass and backpropagation) by pulling each parameter in the opposite direction as its gradient of the loss for each training image. However, given that MLPs don't capture spatial relationships, I think I find it puzzling, without considering the existing empirical evidence, to understand why MLPs are able to learn image tasks like classification. I'm wondering, is there possibly a solid statistical/analytic reason we expect MLPs can learn tasks without primarily basing intuition on existing empirical evidence? Is there a tangible way to understand how iteratively updating the parameters to minimize the loss can translate to learning patterns such as learning digit "templates" with positive/negative regions to classify MNIST digits?
I'm also curious about what MLPs can and cannot do. Although not capturing spatial relationships seems to be a big limitation of MLPs, the universal approximation theorem and them being fully connected seem to give me some (subjective) sense that training large MLPs for advanced tasks might also be possible. I'm curious if there exists a database, kind of like this list of NN performances on MNIST, for the most advanced tasks people have gotten MLPs to learn? For example, has anyone ever successfully trained a cat or dog recognizer with a MLP? Or have MLPs really only worked for tasks that linear models have worked equally well for? Thank you
(For anyone interested I found this)


text:
Segment a spectrogram into a series of images by (constant) beats per minute to train a Deep Neural Network
I have a .csv file with information about a soundtrack and it is divided into beats (per minute), which are ordered by row. As in: the index corresponds to each beat, and the columns have info about what is going on in each beat. The idea is to train a model to be able to infer similar data from any given song following the example of the .csv. [EDIT] Take into account that, in this specific case, the beats per minute do not vary in length, but rather, have a constant, fixed time in each song. The BPM may vary between songs but not within them.
I already have tried some different ways to make a spectrogram. Now, I need to divide it into beat-sized chunks to train a deep neural network.
I used the librosa.core.spectrum to create a logarithmic spectrogram out of a .wav file. I have tried looking for tutorials on how to use spectrograms to train DNNs but they already have separated sound files which they then convert into a spectrogram. Or else they cite papers with a close solution with no code given (take into account that I am a Python beginner).
I need to take a spectrogram and divide it into chunks the size and duration of a beat (total number of beats = beats per minute multiplied by the length of the song), but I don't know where to begin looking for guidance on that process.
Unless there's a different way to relate each beat in the .csv to each segment of the spectrogram? What would you recommend?

How complex is the relationship between time and musical beat in your training data in general? If it's all modern dance, rock, pop, and without any timing-affecting expression like ritardo, accelerando, pauses or free expression, then your task can maybe be handled with relatively simple tools. But a lot of real music won't fit that

@NeilSlater Not complex at all. It is almost abstract. Specific colors and off-on commands for a lamp, fog machine activation, and so on. It is less about the music itself and more about what happens at certain points of the song at the night club.

That sounds like your data per beat is simple, but not quite what I was asking. Does each piece of music have a fixed and precise timing, with little flexibility? E.g. electronic dance music written on a sequencer with a fixed bpm per song would have simple and precise timing, whilst a recording of an orchestral piece performed by humans probably would not. Extracting beat timings will be a lot easier and simpler in the former case.

@Neil Slater Ah, sorry about the confusion. No, no tempo rubato, no change in tempo, no pauses, no variation in speed and no variation in beat length. In this specific case, BPM is less about musical tempo and more of a technician's hack to slice time in regular intervals for the sake of data entry.


text:
Best way to classify chess pieces on a chessboard (on a square) [more details in the post]?
Ok, so I am working on a project which classifies chess pieces. The input is just a chess piece from a specific chess set on a white / black square on the chessboard. So it's just an image of a chess square on the board, with a chess piece on it (or without).
Assume the dataset on which the model is trained is balanced (has the same amount of photos of every piece on a square of a specific color, etc.).
What's the best way of labeling these? I will explain what are my ideas and thoughts:

13 classes, every combination of PIECE-COLOR and EMPTY square, 13 in total. The problem with this approach though is that I think the color of the square that the piece is on will mess up with the prediction (not sure if this would be the case), even though the dataset is balanced.

25 classes, every combination of SQUARECOLOR-PIECE-COLOR and EMPTY square. I am thinking maybe this would have better results, but I don't know if that's the case, maybe it's a bit too complex?

Classify the type of piece first / empty (ignoring the color at first), so 7 classes, and after the type of piece is identified then just the classification of the color (white / black). Maybe this is better than the two above, but I kinda run into the first problem, when trying to classify the color of the piece maybe the color of the square will mess up the results.

What would be best? Are there any other ideas? I want to use a classifier, not object detection, for experimental purposes. I know each one has pros and cons, but I want to hear some opinions.
Edit: After thinking a little bit I thought of a fourth method which is similar to the third one. Classify the type of piece / empty square, and then classify the color by having 4 labels which indicate square color and piece color (so 4 combinations in total).

Is your task to classify pieces from photos of in real life, or from screencaps of electronic chess games, or both, or something else?

From photos from real life, but basically the input of the problem that I described here is just a part of that photo, just think it as small photo of a chess square on a chessboard, with a piece on it (or empty)

I suggest that you just use 13 classes, and let the network learn to ignore the square's background color. I used this approach when I did a "chess position search engine" of Youtube videos, and didn't notice any issues. Neural networks are very good at learning on which features are important, and what should be ignored (or internally "normalized" away).
Granted, real-life photos have a lot more variability than digital ones. But my data had its own issues, due to overlay graphics, and pieces being in the process of being moved. Here are samples of white and black pieces.
I actually ended up having two extra categories: squares where multiple pieces overlap, and squares where the content wasn't from chess board at all (the source video was showing something else).


text:
How To Resolve This Error
I have a cell here in google colab:

And here is the error message:

I don't really understand how to resolve this error. Could someone help me understand how to fix this? Any help is greatly appreciated.


text:
Research on Direct Preference Optimization in subsequence of the response
I wanted to start research on Direct Preference Optimization (DPO) but in the specific case where we're interested in part of the generated sentence, as if we could highlight where the model gave a bad response, and train specifically on it.
More rigorously, given a generated sequence of tokens $\{y_1, y_2, ..., y_t\}$, I wanted to fine-tune the model on a selected range $\{y_i, ..., y_{i+k}\}$.
Could I use DPO directly by doing $\pi_\theta(y_i, ..., y_{i+k}\vert x, y_1, ..., y_{i-1})$? Or could similar works be pointed out?
Thanks!


text:
What’s more efficient in multihead attention: multiply QKV by $W_i$ then split or linearly project QKV $h$ times into dimensions $d_k$?
I’m looking to bridge two implementations of multihead attention.
Approach 1: Multiply and Split
Each of the queries, keys, and values is multiplied by a separate square weight matrix of size (embedding size, embedding size). The transformed embeddings are then split into $h$ number of subsets (heads), where each subset has a dimension of (embedding size / h).
Approach 2: Direct Projection
Each of the queries, keys, and values is multiplied by a weight matrix of size (embedding size, $d_k$), where
$d_k$ is the dimension of each head, and typically $d_k = \text{embedding size} / h$. The projected embeddings are then reshaped into $h$ heads, each of dimension $d_k$.
Question
Are these equivalent mathematical implementations with differing efficiencies?
I’d like to grasp both the math and the PyTorch implementations here. There are similar answers here, but no one goes into explaining the differences in efficiency or the mathematical similarities between both.

Hello. Our site supports mathjax, so you can format the math symbols by wrapping them with $. Example: $d_k$ = $d_k$.


text:
Proof of decomposition of policy performance
this is a proof of a lemma 1 from the TRPO paper by Schulman et al about decomposition of policy performance. I have rewritten it down in a notation that I prefer:
\begin{equation*}
\begin{aligned}
\mathbb{E}_{S_t\sim d^{\tilde{\pi}},A_t\sim \tilde{\pi}(\cdot|S_t)}\left[\sum_{t=0}^{\infty} \gamma^t A_\pi(S_t, A_t)\right] & =\mathbb{E}_{S_t\sim d^{\tilde{\pi}},A_t\sim \tilde{\pi}(\cdot|S_t)}\left[\sum_{t=0}^{\infty}\gamma^t (r(S_t) + \gamma V_{\pi}(S_{t+1}) - V_{\pi}(S_t))\right] \\ & = \mathbb{E}_{S_t\sim d^{\tilde{\pi}},A_t\sim \tilde{\pi}(\cdot|S_t)}[-V_{\pi}(S_0) + \sum_{t=0}^{\infty}\gamma^t r(S_t)] \\ & = -\mathbb{E}_{S_t\sim d^{\tilde{\pi}},A_t\sim \tilde{\pi}(\cdot|S_t)}[V_{\pi}(S_0)] + \mathbb{E}_{S_t\sim d^{\tilde{\pi}},A_t\sim \tilde{\pi}(\cdot|S_t)}[\sum_{t=0}^{\infty}\gamma^t r(S_t)] \\ & = -\eta(\pi) + \eta(\tilde{\pi})
\end{aligned}
\end{equation*}
I don't see here how $\eta(\pi)=\mathbb{E}_{S_t\sim d^{\tilde{\pi}},A_t\sim \tilde{\pi}(\cdot|S_t)}[V_{\pi}(S_0)]$ since the state $S_0$ is sampled from the stationary distribution $d^\tilde{\pi}$ rather then $d^\pi$. Does anyone have an idea what I am missing here?


text:
Few questions surrounding CPI, TRPO and PPO
I was studying the papers in quite some detail in the past few weeks and ther are some questions that I couldn't find an answer for yet.

In the TRPO paper is $\nabla_{\theta}\eta(\pi_\theta)$ really just the policy gradient, so esentially $\eta(\pi_\theta) = J(\theta)$ in most literature?
Does the final TRPO algorithm (which is NPG + Trust Region) give a guarantee on monotonic improvement? The theoretical result of the paper gives a gurantee, but since we use a lot of approximation for the surrogate objective I wonder if the guarantee is still there, although I understand that the trust region method wants to preserve as much as possible of the guarantee.
In the PPO paper it is stated the $\mathrm{L}^\mathrm{CPI}$ is porposed in the paper about Conservative Policy Iteration, but I don't see any signs of using importance sampling in the paper. Does anyone know what the authours refer to here?
In the PPO paper it says $\mathrm{L}^\mathrm{CPI} = \mathrm{L}^\mathrm{CLIP}$ to first order around $\theta_\mathrm{old}$ and gets different as $\theta$ moves away from $\theta_\mathrm{old}$. Can anyone provide a proof that this is the case?
Since NPG is proposed as a special case of TRPO, why doesn't it use importance sampling? Also to show that NPG is a special case of TRPO we would need to show that the linear approximation of $\sum_s \rho_{\text{old}}(s) \sum_a \pi_\theta(a|s) A_{\text{old}}(s,a)$ corresponds to the same objective as NPG right (which is the gradient of the objective)? Is there a proof of that?
Also in the TPRO paper the objective given by expanding $L_{\theta_\mathrm{old}}$ in (13) is missing the $\eta(\pi)$ term somehow.


text:
How exactly do you backpropagate the gradient penalty in WGAN-GP?
I am trying to implement WGANs from scratch.
The loss function for the critic is given by :

which i implement in my code as L = average(real output) - average(fake output) + lambda*GP.
For calculating GP, i just backpropagated ones in the critic to get gradient w.r.t interpolated images, and did (l2_norm(gradient)-1)^2.
I understand that pytorch and tensorflow have some sort of automatic differentiation to calculate the gradients of the loss function. However, I am trying to manually backpropagate the gradient of the loss function.
For vanilla WGAN, the gradient that i backpropagate is -1/n for real images and +1/n for fake images. But I am having trouble understanding how to calculate the gradient of the loss function taking into account gradient penalty.
**How exactly do you apply gradient penalty after calculating it? What exactly should you backpropagate? **
Is there a different way to implement this by hand without using automatic differentiation? Or am i misunderstanding something? Any help is appreciated.

Edit:
according to this repo,
the training of the critic happens like this :
                # Train with real images
            d_loss_real = self.D(images)
            d_loss_real = d_loss_real.mean()
            d_loss_real.backward(mone)

            # Train with fake images
            z = self.get_torch_variable(torch.randn(self.batch_size, 100, 1, 1))

            fake_images = self.G(z)
            d_loss_fake = self.D(fake_images)
            d_loss_fake = d_loss_fake.mean()
            d_loss_fake.backward(one)

            # Train with gradient penalty
            gradient_penalty = self.calculate_gradient_penalty(images.data, fake_images.data)
            gradient_penalty.backward()

But, what exactly is happening in gradient_penalty.backward() ?


text:
Apply weight to one feature based on another one for training a regression model
I have 1000 items that have a numerical feature y, the ground truth that I want to predict.
Each item has another feature c that is in some way related to y.
For each item, I can take 11 samples over a period of time.  A sample is a numeric value that fluctuates around the real value y.
I have added another column weight to my data frame where the value is the number of samples for the particular item minus one divided by 11, in other words a range from 0 to 10/11.
Finally, I have added a column 'mean' that is the average of the samples taken before the current sample. This is initially 0 for each item.  and it gradually converges into the direction of y.
How should I train this model? My initial idea was this:

Write the average of all samples into another column total.
Drop the column sample.
Configure the model in such a way that it takes the weight into account when considering the mean.

Is step 3 possible at all? And does it make sense or is the model able to learn these relations itself? In that case, does it make sense to average the previous samples at all?
A more realistic example could be a professional sports league. y would be the ranking of a team at the end of the season.  c would be some team feature like the net value of the team in money. The samples would then be the performances of each team during the season.
The following Python code emulates such a data set. I have also added scatter diagrams for that data.
import numpy as np
import pandas as pd

nitems = 100
nsamples = 11

np.random.seed(42)

y = np.random.normal(0, 1, nitems)
# Some random relation.
c = (y - 1) ** 2 + 2 + np.random.normal(0, 1.0, nitems)

# Repeat each row nsamples times.
df = pd.DataFrame({ 'y': y, 'c': c })
indices = np.repeat(df.index, nsamples)
df = df.loc[indices].reset_index(drop=True)

df['weight'] = (df.index % nsamples) / nsamples

# Generate samples with more noise than the feature c.
df['sample'] = df['y'] + np.random.normal(scale=0.6, size = nitems * nsamples)

# Calculate the cumulative mean of 'sample' with weights less than the current weight
groups = df.groupby(np.arange(nitems * nsamples) // nsamples)
def calculate_mean(group):
    samples = group['sample'].values
    mean = np.zeros_like(samples, dtype=float)
    for i in range(1, len(samples)):
        mean[i] = np.mean(samples[:i])
    return mean
df['mean'] = np.concatenate(groups.apply(calculate_mean).values)

pd.set_option('display.max_rows', None)
print(df)


text:
Is sparsity required for behaviour exhibited by modular neural networks such as compositional generalisation, resiliency to catastrophic learning?
Having recurrency is intuitevely linked with modularity for me (you would need a network topology with an infinite number of layers to account for every combinaisons of situtations possible), but it's not clear for me wether recurrent sparse networks are more modular than recurrent ones with continuous activations.
Are brains very sparse and modular only out of energy consideration or is it required to their composition/blending of concepts, etc.?
Edit: Modified my question according to my comment

this is mere speculation imho

Wikipedia says that "networks with high modularity have dense connections between the nodes within modules but sparse connections between nodes in different modules". If correct, I should reformulate it as exhibit the same behaviour as modular network (e.g. combinaison to form new knowledge, resilience to catastrophic forgeting).

According to research I found, they show that it helps the network generalise and avoid catastrophic forgeting, but they're not sure wether it is due to its modularity (which they define as "many clusters (modules) of highly connected neurons that are only sparsely connected to neurons in other modules"), or its sparsity (or probably a combination of both):

However, it cannot be ruled out that sparsity itself mitigates catastrophic forgetting 1, or that the general learning abilities of the network have been improved due to the separation into a skill module and a learning module. Either way, the data support our hypothesis that a connection cost promotes the evolution of sparsity, modularity, and increased performance on learning tasks.


text:
How to evaluate the relevance of a search made with an AI Agent?
I’m building an AI agent designed to find the best results for with a search tool. The agent is essentially a Large Language Model (LLM) connected to a Search API via function calling. The goal is for this agent to use the Search API to identify the most relevant results.
Let's imagine that I have an evaluation function f(x) that allow to score a specific result.
How can I evaluate the performance of my agent ?
For instance, how can I tell between two versions of my agent (v1 and v2), which one is performing better ?
Here is a specific constraint due to the fact that in a real world search, I don’t have access to the entire database of results due to the sheer volume (millions of results) and the cost involved. Since I can’t purchase all these results, it’s difficult to know if a specific record should be included in the search results. So I can't use recall like metrics.
Given this constraint, how can I effectively evaluate the performance of my AI agent?

How do you assess leads found by humans doing the same task?

Looks like an Unsupervised Learning task to me. So you could cluster your leads by some criterion. There are many criteria, and many methods of cluster analysis.

@NeilSlater that’s a very good question. Typically, this process is done through qualitative human evaluation (in other words, it’s not very scientific or rationalized). But that’s precisely my question! How can I turn this manually handled problem into one that can be solved with AI?

That would be useful to know, and helps frame the question as automation of something existing (as opposed to asking for magic, which is close to how the question currently phrases it). Please put some information about how humans performing the same task are assessed into the question. It doesn't have to be detailed, just a variation of your comment. As a spoiler, of course someone answering will be suggesting you assess the AI's outputs in the same way, so you may want to elaborate on the resources you have at your disposal, and what opportunities you have for online feedback.

Another obvious follow-up is do you have data from a history of good and bad leads found by humans to use for training of any assessment function?

@NeilSlater, sorry, my question wasn't clear. I don't need help in finding a business metric to specifically evaluate the quality of a lead. It's more of a generic problem that is "how to evaluate the accuracy of a search done by an AI agent". The lead use case was just to give an example more concrete to the problem. I edited the post to remove mention of the lead use case.

OK, so now we're back to "magic" unless you can explain what you want . . . ignoring the AI for a moment, do you have any way to assess search results from your system? Any AI is just going to be a system that tries to maxmise the results from that metric in a more sophisticated manner than a non-AI-based solution. The metric does not need to be direct. It could be based on results such as saving the searchers time, making them happy, or some financially-measured benefit. However the more direct and individually targetted metrics are easier to use for training.

Also the metric does not need to be something assessed on all results forever. You could collect and score data about good and bad search results, and use that to train an agent that you may reasonably expect to perform at a certain level. So, do you have any way at all to tell whether one search result is better than another, or to give it a rating, assign a financial benefit etc?

OK @NeilSlater, there nothing related to magic here. I'm asking for a scientific method to evaluate search results. Evaluating search results is a problem like many other in LLM evaluation. Let's say that we have a function f(x) that evaluate the score of a specific result. It's still far for trivial to tell wether one version or another version of the search agent is performing better than the other.

An output cannot be evaluated in a vaccum. You need to state some kind of measurement, goal or outcome. Often search results are scored arbitrarily on custom expert ranking (e.g. a "relevance" score that might be constructed from number of matches to input text). If yo uare trying to replicate that, but using an AI-powered search, then you can say that. But so far the question literally says "There is no objective metric to determine if one result is better than another." and gives no other clue to what you are doing. This is impossible to answer

The problem was indeed composed of multiple sub problems so I reduced the scope to focus on only one problem. Let's say that there is an evaluation function to score a specific result. If I have this function, how can I assess the performance of my search ?

Let's imagine that I have an evaluation function f(x) that allow to score a specific result.

It's not clear from this, whether f(x) is a relevence score for a single found item, or is a rating for a complete search result. If it is the former, then you can construct the latter by aggregating results in some semi-arbitrary fashion such as taking an average of f(x) for the top ten results returned by a search. This aggregation is arbitrary, but you can base it on intuitive concepts such as the number of results presented in a page. Ideally you have this grounded in some sense relating to the higher level goals of the system (i.e. although the system is a search system, no-one has "searching" as a stand-alone goal in practice, there is always some context).
To further aggregate this value per attempt into a value per search agent, you will need to construct a test across multiple requests, covering a gamut of different search terms representative of the kinds of search you expect the system will be used for. Take the mean result or f(x) score over all test searches in the test set. This will be necessary because it is very unlikely that different searches will be equally effective, so you will want some kind of averaged score that represents the system's performance against how it will be used in practice.

For instance, how can I tell between two versions of my agent (v1 and v2), which one is performing better ?

It is the one with the better aggregated f(x) score, measured over a standardised test set. You can reduce cost of measuring this, at the expense of accuracy, by using a smaller fair sample from the test set. Or you could achieve similar by having small/fast test set to get rough indication and then a larger test set for refining things. You will want to understand the standard error in your measures of aggregate f(x) so you can tell when a difference is large enough to be significant.

I don’t have access to the entire database of results due to the sheer volume (millions of results) and the cost involved.

Whilst this might rule out certain kinds of system validation, such as demonstrating the current f(x) is the best possible, it does not prevent you testing the system statistically. The important detail is to make the results on repeated test runs with different agents equivalent in as many ways as possible - changing only the internal and controllable aspects of the agent between tests, not how or what you measure.


text:
Is deep learning suitable/preferable for string similarity detection and application automation? If so, which type?
newbie here. I have developed an app that basically does:

Perform OCR, check if words are contained in the resulting text and then perform an action.
If no words are detected from the given list, then use string similarity algorithms such as Levenshtein to check if there might be similar words.

In reality, I am already using DL since the OCR engine uses LSTM.
Since I would rather avoid developer supervision (checking for words and and specify the action based on the detected words), I was wondering if DL would be suitable in this case scenario.
Let's say we want to automate the battle in a videogame.
The dataset would be basically either images/videos of basically the user playing.
So we would have STATES and ACTIONS.
More specifically I would like to ask you if DL could help in:

Learning in which STATE the app is.
Learn what ACTION to perform without the developer creating an association of STATE -> ACTION.

But I think the last one might be a stretch, meaning I would still need to specify what action follows a certain STATE.
If what I've described is suitable, what amount of data would be required?
Because I've been told with DL the dataset would need to be huge, so If I would need milions of screenshots/videos I guess I'll just keep the current architecture.

Funny, DL could stand for Damareu-Levenshtein method or deep learning (I’m assuming you meant the second). In short, yes, both are technically possible using deep learning. Now, how you execute this depends on the specific game you are trying to automate. Since you mention a battle game, the execution will be a little bit complicated in terms of how much you can do without access to lots of compute resources. However, it is doable using reinforcement learning.
I’ve seen work on similar ideas for popular PvP battle games, using something called NEAT. This is a good intro to the principle: https://macwha.medium.com/evolving-ais-using-a-neat-algorithm-2d154c623828
The way NEAT works for your use case is that given the game controls and some other information from the player angle, NEAT will generate several unique models and make them randomly interact with the given information. How you control it to do what you want is set rewards for doing something right in the game and add punishments for doing something bad. For every iteration NEAT works, models that earn the most rewards improve and continue, while the other “species” die. This way, the initially naive and random models slowly get more and more skilled through “natural selection”. Note: This is a very simplified explanation.
However, here is the issue: training. If you look to train it with videos of gameplay, it will take hundreds or thousands of videos at a very optimistic minimum, much more if the game is very complicated. Additionally, to train with NEAT with this much data to achieve a successful species, it will most likely take lots of time and a lot of your CPU / GPU’s resources. Again, it’s hard to give specific estimates. Note that you’ll have to encode the “rules” of the game well and figure out a way to break down what is going on in the video into specific components that the model can see and control. That falls more into how to apply NEAT for the task, but in theory, game automation can be done using it. The model can pick up the state that a game is in and (after a lot of training and trial and error) will ultimately be able to react to states with actions.
Why I suggest NEAT is that I’ve seen a project where, given the models an input of game controls (up-down and left-right) and vision of walls and opponents with ray tracing, a model was made that could play the online platformer PvP game “Tag”, where players jump on platforms to try and catch each other.
You were mentioning OCR a lot, so that makes me wonder: is the game easily playable based on text shown on the screen? Because if so, there is a much easier way to achieve what you are trying to do based solely on text. If you were to use text, you wouldn’t even need to train a model on game footage. You could hard-encode certain keywords to actions, or if the keywords differ each time, you could use a language model to get the idea of the text and convert it into an action. Hope this helps!
Edit: If I have understood correctly, ideally, you would have a highly accurate OCR reading of the text from the screen, with correction methods to retrieve the intended text with high accuracy. After reliably extracting the keywords, you would associate them to unique "states", which are associated with "actions". At this point, you could either use machine learning for classification, given input keyword and output state. Or you could hard-code keywords to states and states to actions using a dictionary.
I would recommend you hard-code them, because either way you will have to yourself decide what keywords mean what state and then tell the computer program this. It's a common misconception that results will be "better" or "smarter" in any way when using machine learning for tasks like this. Think of it this way: if you hard-code keywords to states, you're directly telling the computer what the correct thing to do is, which is always going to be faster. If you choose to use some form of machine learning or reinforcement learning, you're still having to hard-code keywords to states in the form of a dataset, but even after that effort, you also have to wait for your model to learn the patterns and associations that you want it to. Remember, for simple tasks such as connecting a keyword to an output, AI is simply a glorified search algorithm for its "learned" database. Both methods work, but it's up to you whether you choose the faster method or the more time-consuming one. Training an AI model takes a decent amount of data, and while it's nothing impossible to do, you could also spend that time just directly hard-coding instead of trying to teach an algorithm the pattern that you already understand exists. In short, from what I have understood, using AI for this task would be redundant and unnecessary.
Now, if you say that the OCR model itself is struggling to pick up the keywords reliably, and that is the step you want to implement AI, then you could potentially have a valid use case. Otherwise, hard-coding is the way to go (at least in my mind).

Thanks for the reply. Yes, the game is easily playable based on text shown on the screen. As a matter of fact, the approach of keywords to actions is exactly what I was doing but the downside is that keywords for each state of the game were chosen by me. Maybe what I could do is trying to automate keyword identification? If so, any suggestion?

Are the keywords always the same for each state of the game? Because if so, even though you could achieve your goal using deep learning, it’d be a bit redundant, and it’d be much easier/faster to use hard-encoding. If you can’t hard-encode them, you can fine tune a large language model using a custom dataset. Create a small dataset with the keywords of your game as the input feature and  their corresponding actions as the output feature. Fine tuning language models is much more doable using HuggingFace. I’m a little confused as to your use case. Can you edit your question with more specifics?

Yes, the keywords are the same for each state. If I create a small dataset of basically keywords as input feature and actions as output feature it means that I can also hard-encode them since keywords are already known, right? What am I missing? Maybe a LLM would be beneficial for better OCR recognition.
Yes, I will update my question.

I edited my above response with 3 more paragraphs (sorry!). Take a look at it and see if it helps. I'm open to any further queries!


text:
Would the DDPG algorithm still function effectively if some transitions stored in its replay buffer are generated by a completely unrelated policy?
Let's hypothesize a scenario where some of the records (si, ai, ri, si+1) in the replay buffer are generated by another completely unrelated random policy. If the DDPG algorithm still samples random minibatches from this buffer for learning as usual, would the learning process proceed successfully?
Actually, there's a pre-processing stage before training DDPG in my online learning application, where another module learns the safe action range. I wonder if the transition records obtained during this stage can be used to pre-train the DDPG agent.

TLDR: it might work, but it will probably struggle, or not work at all, depending on how optimal your "other" policy is

So, strictly theoretically speaking, as long as the behavioral policy explores all the state-actions involved in the optimal policy, DDPG will indeed work (in other words, if the optimal policy involves an action that your safety module excludes, don't expect DDPG to learn it)
Now, we don't live in a ideal world, and we have to face the reality of the approximations made along the way; and practically, it probably won't, or at least, it will struggle a lot
Indeed, DDPG is part of the Q-learning algorithms, which is, by definition, off-policy, which means that you can learn from experience coming from another policy... however, the state distribution of the behavioral will bias the Q estimate, and thus increating instabilities for the learning

Thanks for your answer. The objective of this "other" policy in the safety module is fundamentally different from the objective of the DDPG policy. In other words, it does not aim to maximize the value defined in DDPG. It might simply be some random control signals, with nothing to do with reinforcement learning. Under this circumstance, would it be more beneficial for DDPG to completely disregard the transition records generated by this "other" policy and instead explore and learn an optimal policy on its own within the safety boundaries provided by the safety module?

@JJbow the behavioral policy is not forced to be following the same reward function, so should not make any difference... and the on policy scenario (DDPG is both the learning policy and the exploring policy) is always the best one


text:
Analyzing a deep learning model by constructing a Matrix from input and output data
I am currently completing my bachelor’s thesis, and recently my supervisor suggested that I could strengthen my arguments by explaining why deep learning models perform so well in my case.
To give you some context, I am essentially attempting to learn, in a supervised manner, an output vector given an input vector.
Specifically, what I aim to achieve with deep learning is to learn the inverse operator of a matrix transformation, as illustrated by the following equation:
$$
\vec{x} = A^{-1}\cdot \vec{y}
$$
So, what the deep learning is trying to do is the following:
$$
\vec{x} = DL(\vec{y}, \theta),
$$
where $\theta$ are the weights of the model.
I thought of in trying to conduct a spectral analysis of the deep learning model to compare its spectral analysis (distribution of eigenvectors) to the spectral analysis of the inverse of matrix

A. However, since the deep learning model is non-linear, I cannot perform a traditional linear spectral analysis. Given this limitation, I have come up with an alternative idea:
Let's assume that there exists a matrix $C$, such that it can map the input vectors and the output vectors yielded by the deep learning model. My idea is now to conduct the spectral analysis on this constructed matrix (e.g. by the method of the least-squares or SVD) and then state that the spectral analysis of this matrix $C$ is approximate to the spectran analysis of the matrix $A{^-1}$.
I have tried this empirical validation and it seems to confirm my assumption. Somehow, the deep learning model, even though its non linear, it has transformed the input y into a input x, so that there exists an approximate linearity between these vectors and which can be captured with the matrix $C$.
Since the deep learning model performs very well and the spectral analysis between matrix $C$ and $A^{-1}$ are similar and the opposite with a bad deep learning model, could i affirm that the deep learning model seems to perform well, because the linearity that its produce seems to be in the spectral realm of the matrix $A^{-1}$ ?
The thing about this is that its a big assumption and i haven't found literature on this...
I would be very glad to here some insights from you
Best regards

Your neural network is overfitting to the distribution of matrix you are feed it: matrix inversion is $O(n^3)$, your neural network is a finite set of $O(n^2)$ operations, so you definitely cannot teach a plain neural network to invert "any" matrix


text:
What about the loss and custom metric with per-pair weights in multi-class classification?
Let's suppose that we have a multi-class classification problem with 5 classes: 0, 1, 2, 3, 4. The order is not random, they are neighbors. For example, imagine that a labelling is 1. If the prediction for it is 1, it is the best. If the prediction is 0 or 2, it is also good, but not the best. If the prediction is 3 is worse, while if the prediction is 4, it is the worst.
I want neither the classical cross entropy which simply penalizes any misclassification equally, nor the weighted one which simply penalizes more the classes with less samples in an imbalanced dataset. I want to specifically and per-pair penalize any of the possible (25 in my case) combinations of prediction-label.
So I have 4 specific questions:

How is this so much customized loss called? What do I have to google and what do you suggest me?

What about the custom metric? I am talking about a custom metric for monitoring during the training, not in meta-training. I suppose it is something more advanced than the F1 score or the Fβ score?

Can these weights be somehow trainable? For example, let's say, by using Gradient Reversal layers?

Is this problem really multi-class classification or should I switch to something else? Generally speaking, is it good that I focus on improving the loss and the custom metric for monitoring, because I feel my model (Transformer) is doing very well but needs full human help and customization, or I should totally focus on its architecture and the classical - default accuracy - cross-entropy are more than enough?


text:
How do I make an artificial intelligence for a game played on a continuous board?
There is a lot of introductory literature about artificial intelligence for time and space discrete games, like chess. A few fundamental methods are offered:

Minimax to depth $n$ with an evaluation function will pick a move that reliably leads to the outcome best according to that evaluation function after $n$ further moves. There is a whole industry of building evaluation functions, but the principle is simple.

A monolithic policy function simply weighs all possible moves and picks the heaviest. It will commonly be a fancy neural network. There is a whole industry of building policy functions, but the principle is simple.

Weighted random search learns by playing randomly many times and giving more weight to those moves which lead to victory more often. There is a whole industry of building the distribution of random moves, but the principle is simple.

None of these methods apply to games played in continuous space (or a finely enough discrete space) or continuous time (or a discrete time where interesting events are sparse). An example of such a game is StarCraft.
There are two methods a computer plays StarCraft:

A bespoke handcrafted algorithm with many moving parts, such as build order, path finding, micro control. This is expensive because you need a lot of human effort and expertise.

A neural network that takes screen pictures to mouse clicks. This is expensive because you need a lot of hardware and electricity.

So, anyone can write an entertaining, if not dominating, artificial intelligence for a discrete game, but only very few can afford to write an artificial intelligence for a continuous game that is any better than inept. This is a sad conclusion.
Is there anything I am missing? Are there any helpful methods out there that I have not mentioned? Is there a way to adapt discrete methods to continuous games?
It seems that the difference in principle is that, in continuous games, we need to create a new move from scratch every time instead of picking out of a small set. If we simply create random moves, many of them will be inadmissible — trying to build stuff that cannot be afforded, sending units to unreachable places. Of those which are admissible, the majority will be meaningless random walking of units. The subset of moves that have any measurable meaning is tiny, and even the best single move has only tiny effect on the outcome. If we could make the computer generate a small amount of plausible moves, then we can apply discrete methods. But how do we teach the computer to generate plausible moves?
A case in point is the navigation mesh technique that sidesteps the problem of smooth path finding by finding paths in a discrete approximation of the given smooth space. Is this the best we can do?

RL works fine in continuous state space, what you are referring to are tree based/search based methods, which are obv hard to apply in continuous cases

@Alberto   I know one can teach a snake to gather fruits with reinforcement learning, or something such. I am not sure if this would work for games where all the reward is concentrated in the end. Would it?

RL is an algorithm that does not make any assumption on the reward function, except that they are in $R$ (thus no $+-\infty$), so yes it can work

@Alberto If you write an answer explaining the application of reinforcement learning to a realistically complicated continuous problem (like StarCraft), I shall give you a vote at once.

I mean... https://deepmind.google/discover/blog/alphastar-mastering-the-real-time-strategy-game-starcraft-ii/ ... they probably know better than me

Ah, that is what I had in mind when I mentioned expensive neural networks that take screen pictures to mouse clicks. I looked at their research article a while ago and I remember it was an impressively complicated neural network that took days if not weeks to train. What I am looking for is something as «naturally efficient» as the minimax algorithm for discrete games.


text:
Can Diffusion Models denoise an unseen probability distribution during inference?
I am trying to understand if it is possible to condition the reverse condition process with a weight tensor. Normally this weight tensor is more restrictive (binary) and could be used for downstream tasks like inpainting. In regions where weight=1 then we add the output of the model and in regions where weight=0 we add the initial image corrupted with noise.
In my case, I want to relax the binary condition to values in $[0,1]$. Let's denote the initial image as $i_0$ and the denoising model as $f$. At timestep $t$, during the reverse process, we blend the corrupted input $i_{t-1}$ and the model output $f(x_t)$ as:
$$x_{t-1} = (1-w) \cdot i_{t-1} + w \cdot f(x_t)$$
For binary weight masks this is OK, because the model has seen noise samples from both the distribution of $i_{t-1}$ and the distribution of $f(x_t)$. However, when the mask is non-binary then, the linear combination of two gaussians end up in a new gaussian:
$$G = \mathcal{N}((1-w)\cdot\mu_i + w\cdot\mu_f, (1-w)^2\cdot\sigma_i^2+w^2\cdot\sigma_f^2)$$
where  $\mu_i, \sigma_i$ and $\mu_f, \sigma_f$ are the parameters of the distributions of the corrupted initial image and the reverse diffusion process respectively.
Unfortunately, I am not having good results if I sample $x_{t-1} \sim G$. Is this because $G$ is not a known distribution for $f$ to denoise? Or is my intuition on this problem wrong?
Thanks!


text:
Why generative models produce mesh structure at the beginning?
I don't really understand the reason of this. I have listed the outputs of different models below.
Gan (source: self made simple GAN for CIFAR10)

Vq-vae (source: link)

What i mean is illustrated here.
The image that we want to encode then decode (reconstruction)

The input of the model

To provide an answer, it would be helpful to get the sources of your images as well as a bit of elaboration on what i am looking at. Is this the output of the model? Is this somewhere in the middle? What is actually visualized here?

@RobinvanHoorn , edited question


text:
How to Handle Masked Values in Neural Networks for Geospatial Data?
I am working on neural networks for oceanographic data and face challenges in dealing with masked values, which I set to NaN.. I can train a neural network model with 1D vertical profiles (e.g. temperature and salinity at different depths in the ocean) and 2D ocean surface data (e.g. sea surface temperature), where

In 1D data, values near the seafloor are masked by the ocean floor.
by the ocean floor.
In 2D data, masked values occur due to land masses.

I would like to understand the best practices for handling NaN values in these contexts. Specifically:

How can NaN values be handled within neural networks during training
without altering the original data by filling in arbitrary values?
Which neural network architectures or strategies are best suited to deal with
for dealing with masked values, where the masked values represent important
important information about the system?
If filling in masked values is unavoidable, is there any evidence or proof that adding
evidence that filling in masked values does not adversely affect the training of the
the training of the neural network?

Any technical advice or references would be greatly appreciated.


text:
CNN: Accuracy gap of 5-7 % between accuracy computed on-the-fly and separate model evaluation on the training set
I am training a CNN for some basic classification task. During training, I compute the training accuracy after every epoch. After the training has finished, I evaluate the model again on the entire training set (and compute additional metrics). I noticed a gap in the accuracy scores between the accuracy as I compute it after the epoch and the accuracy when evaluating the model on the entire training set in one go.Now of course, during training, the model can improve across the batches. So it is to be expected that the training accuracy of the model AFTER finishing the epoch will be greater then the "average accuracy" during the epoch. BUT, this behavior persists even in later epochs, where training accuracy increases only rather slowly. Even then I see a difference of between 5-7 %. This seems weird to me and I'd like to figure out why this occurs and if I perhaps do something wrong!
Here is the code I am using:
def TrainCNN(model,  train_set,  valid_set, training_params, outpath):
    loss_function = torch.nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=training_params['lr0'], weight_decay=training_params['l2'])

    train_loader = DataLoader(train_set, batch_size=training_params['batch_size'], shuffle=True)
    early_stopping = EarlyStopping(outpath, patience=5, verbose=True)

    metrics = training_params['metrics'] #{'balanced_accuracy':balanced_accuracy_score, 'accuracy':accuracy_score}
    scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.25, patience=3)

    for epoch in range(0, training_params['epochs']):
        train_accuracy, avg_loss = TrainOneEpoch(model, train_loader, loss_function, optimizer, l1_lambda=training_params['l1'])
        valid_metrics_scores = EvaluateModel(model, valid_set, metrics)
        early_stopping(-valid_metrics_scores["balanced_accuracy"], model)
        scheduler.step(-valid_metrics_scores["balanced_accuracy"])

        if early_stopping.early_stop:
            print("Stopping training due to no increase in validation accuracy...")
            break

        print(f"Epoch {epoch}: Train {train_accuracy * 100} % Accuracy")
        print(f"Epoch {epoch}: Valid {valid_metrics_scores['balanced_accuracy'] * 100} % Balanced Accuracy")

    final_train_metrics = EvaluateModel(model, train_set, metrics)

    return model, final_train_metrics, valid_metrics_scores

def TrainOneEpoch(model, train_loader, criterion, optimizer, l1_lambda):
    model.train()
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    model.to(device)

    correct_train = 0
    total_loss = 0.0
    size = len(train_loader.dataset)
    for batch, (inputs, labels) in enumerate(train_loader):
        inputs, labels = inputs.to(device), labels.to(device)
        optimizer.zero_grad()
        pred = model(inputs)
        loss = criterion(pred, labels)
        l1_norm = sum(p.abs().sum() for fc_block in model.fc_blocks[:-1] for p in fc_block.parameters())
        loss = loss + l1_lambda * l1_norm
        loss.backward()
        optimizer.step()

        correct_train += (pred.argmax(1) == labels).type(torch.float).sum().item()
        total_loss += loss.item()

        if batch % train_loader.batch_size == 0:
            loss, current = loss.item(), batch * len(inputs)
            print(f'loss: {loss:>7f}  [{current:>5d}/{size:>5d}]')

    train_accuracy = correct_train / size
    avg_loss = total_loss / size  # average loss for the entire dataset

    return train_accuracy, avg_loss

def GetPredictions(model, dataset, batch_size=128):
    data_loader = DataLoader(dataset, batch_size=batch_size)
    device = 'cuda' if torch.cuda.is_available() else 'cpu'

    all_predictions =  []
    all_labels      = []
    with torch.no_grad():
        for inputs, labels in data_loader:
            inputs, labels = inputs.to(device), labels.to(device)
            outputs = model(inputs)
            _, predicted = torch.max(outputs,1)

            all_predictions.append(predicted.cpu())
            all_labels.append(labels.cpu())

    all_predictions = torch.cat(all_predictions)
    all_labels = torch.cat(all_labels)

    return  all_predictions, all_labels

def EvaluateModel(model, dataset, metrics, batch_size=128):
    model.eval()
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    model.to(device)
    predictions, labels = GetPredictions(model, dataset, batch_size)

    metric_scores = {}
    for metric_name, metric in metrics.items():
        score = metric(labels, predictions)
        metric_scores[metric_name] = score

    return metric_scores

The difference I am talking about is the diffence between the accuracy computed here
train_accuracy = correct_train / size

and the accuracy computed here (among other metrics):
predictions, labels = GetPredictions(model, dataset, batch_size)

metric_scores = {}
for metric_name, metric in metrics.items():
    score = metric(labels, predictions)
    metric_scores[metric_name] = score

The metrics I am using are these (from sklearn):
def BuildMetrics():
    f1_score_weighted = lambda y_true, y_pred: f1_score(y_true, y_pred, average='weighted')
    precision_score_weighted = lambda y_true, y_pred: precision_score(y_true, y_pred, average='weighted')
    recall_score_weighted = lambda y_true, y_pred: recall_score(y_true, y_pred, average='weighted')
    metrics = {'balanced_accuracy': balanced_accuracy_score, 'accuracy': accuracy_score, 'F1': f1_score_weighted,
               'recall': recall_score_weighted, 'precision': precision_score_weighted, 'confusion_matrix':confusion_matrix}
    return metrics

My CNN uses dropout in one of the fully connected layers. However, deactivated dropout cannot (!) explain this issue, as I get the same  accuracy using the accuracy_score function using the model in train and eval mode. I just tested exactly this one after the other while debugging.


text:
Can local learning rules minimize a global loss?
It is widely believed that synaptic plasticity is the way biological brains learn. Artificial implementations of this mechanism are for instance local weight-update rules in Spiking Neural Networks. Locality here means that a weight update only depends on things that the post-synaptic neuron 'can see' at the time of the update.
This is in strong contrast to updating all the weights simultaneously by stepping along a gradient of some global loss function.
Question: Have you seen cases where local update rules are proven to minimize a global loss function? Given a loss function, can we formulate update-rules that minimize this loss?
I do not see any reason why local update rules would result in stepping the weights along some gradient field, in general. But maybe in some special cases?
Edit: Hopfield networks seem to do this! The local Hebbian rule does minimze the (global) energy function.

I once tried to optimize a neural network by gradient descending 1 input-output pair at a time ("local in time"), but it did not seem to work for reducing the total global errors even over very many iterations. My theory was that if the optimizer can only see / check 1 output-vs-target result at a time and have to make a decision to adjust a parameter based on just this 1 item error it can see, then in most likelihood this adjustment will in hindsight be a bad decision for the global error overall.


text:
Modern methods for denoising and deblurring images with different image sizes
What modern methods exist for denoising and deblurring images?
Yes, I haven’t forgotten how to Google and spent a lot of time searching for the code of a neural network that would not have been created 5 years ago, but I didn’t come up with much.
The idea is to create a modern neural network to remove noise and blur from astronomical images. There are already a couple of options on the market, but they are paid and not available to the average user, or not trained on astronomical data.

There was a Helsinki Deblur Challenge 2021, where the goal was to de-blur images of ASCII text. Maybe the submitted networks are suitable also for astronomical pictures. At least the winner used U-net architecture, as you might guess. Note that here the problem was gray scale, but I guess you can transform the image into (for example) YCbCr color space, de-blur the Y channel and just apply noise filtering to the color channels.

Very interesting information, thanks! I assume that it is enough to divide the RGB into three mono layers and get an average PSF, which can already be applied to each layer and end up with a good result.


text:
NN training by optimizing hidden layer content with information bottleneck "inf_T I(X;T) - beta I(T;Y)"?
There was a lot of hype about Naftali Tishby's information bottleneck method a few years ago, but it is nearly silent now, especially that sadly the author has died in 2021.
In theory it allows to directly optimize the content of hidden/intermediate layer T - that it should not only maximize mutual information about the output I(T;Y), but simultaneously also minimize information about the input I(X;T) - remove noise, (lossy) compress/extract the most valuable information.
In Tishby's talks like https://www.youtube.com/watch?v=utvIaZ6wYuw he uses it for evaluation, but it seems tempting to try to directly use it for training - optimize content of intermediate layers. Personally I see it quite promising, currently working on new ANNs which can be cheaply trained this way ( https://arxiv.org/abs/2405.05097 ), so wanted to ask if it makes sense.
Was this e.g. "inf_T I(X;T) - beta I(T;Y)" directly used for NN training? Should it be?
Maybe some more sophisticated minimization-maximization would be better?

Update: Turned out there is direct training with information bottleneck - HSIC in 1 and 2, using similar central formula Tr(Kx Ky) as I have found, but on local basis (KDE, width dependent) while I use global (polynomial on variables normalized to U[0,1]). Their diagram claiming it is more biologically plausible training:


text:
YOLOv1 - Why do we predict multiple bounding boxes?
When you look at the YOLOv1 paper and corresponding implementations it is always mentioned that

for every grid cell, we predict B bounding boxes (usually two).

Then we use IoU to choose the  bounding box out of the B predicted ones, which fits best the real one. All the other B-1 boxes are neglected.
My question is:
Why do we even predict B bounding boxes? Isn’t this a waste of resources and may even decrease performance compared to only predicting one box? It is also not like „achor-boxes“ which were introduced in the YOLOv2 model.


text:
For copyright purposes, how can AI-generated content (eg.lyrics) be proven as such rather than human-generated?
Most answers I have seen to the question of whether AI-generated content is copyrightable cite law in the US and other countries limiting copyright to human-created content. However all the answers I have seen fail to address the question of how a government copyright office and then potentially later a rights holder, copyright enforcement company, or court would be able to prove it was AI-generated.
AI platforms are often vague about retention and/or use of user input and AI output, and it is not clear to me how accessible this data would be to government copyright offices, courts, and copyright enforcement companies for purposes of determining whether a particular lyric were AI generated.
In some cases, already copyrighted content may appear verbatim in AI output, and to the extent this can be easy recognized as an existing work this would make the AI authorship question moot. However most of the time this will not be the case, especially where AI only enhances a user's original work rather than composing something from scratch or with minimal prompts.

In long-run, I would say it is a lost battle. Sooner or later AI can generate content that is indistinguishable from real human's work, just like a handsewn sweater versus a factory-produced one.
Even if all AI platforms fully comply in every way they can, that does not stop small work groups or even individuals to create their own AI tools. And it only gets easier and easier as technology advances.
I have seen some artists being asked for proofs that they did not use any AI tools, to which they complied by showcasing how they created the work (draft, diary, elaboration of thoughts and motivations). However, given sufficient demand, there will be AI which can generate 'proofs' (or simply fake some paperwork). To some point these proofs will become so real that fools everyone, again.
So I am on the pessimistic side. The most feasible solution I can think of is to fight magic with magic: train an AI detective. Or we can do the old-fashion way: have a couple witnesses to monitor every moment an artwork is created. But alas, human lies a lot too.

Although often today there is a stylistic "tell" to AI generated content, whuch might raise suspicion (for example it's still relatively easy to spot cut&paste from ChatGPT in Stack Overflow answers, and the mangled fingers and weird not-quite-sure-what-that-is parts of generated artworks), there is no reason to expect that to always be the case. It is likely to be a technical impossibility to detect high quality AI generations, as distinct from human creative works, purely from their content, in the not too distant future.
In humans, the creative process takes time and effort, and will leave evidence. For instance a writer will have notes, early drafts and also have provenance for other works they have made. They will be able to explain how they got the idea for a creative work, and at least some of the decisions they made in order to arrive at a final published version.
In contrast, an AI generated work without human input is generated complete without any other artifacts - the producer of it will often not comprehend it fully, and won't have any evidence of the work they did. They may also not have a history of producing similar works, and be unable to demonstrate any skill with the type of content they claim to have made, when challenged under controlled conditions.
This does leave more subtle interactions, where a skilled artist uses AI to augment their work, where such challenges would not be able to tell. But this is not very different to difficult human vs human "art theft" cases, where a lot of the decision is not technical, but based on statements made by plaintiffs and defendents in court. There will be some success rate due to the fact that most people will stick to a variation of the truth as opposed to blatantly lie about things they did. There may also be witnesses to behaviour that corroborate one side or another in a dispute. This will not be part of any technical test, but then there are plenty of other legal issues where there isn't a simple technical test, so that doesn't break anything.


text:
Please help me understand constant-α Monte Carlo method
This is my understanding thus far about Monte Carlo method for approximating value function:
Instead of using a recursive Bellman equations and knowledge of environment dynamics, Monte Carlo methods use statistics to evaluate a value function. For a given policy π and starting state S_t, multiple episodes are generated. The return for each of these episodes is calculated and averaged out. If the number of samples is large enough, the calculated value converges to the expected return.

But according to constant-α Monte Carlo, the value function is evaluated using following update rule:

where V(S_t) is current estimate of value function for state S_t and G_t is the return after time step t. I don't understand how this is in-line with the averaging method.

Can you please write the equations using mathjax and ask your specific question in the title? "Understanding x" is not really a question and very specific. Thanks.

Say you have an estimate of $V(s_t)$ and you visited $s_t$ $\#s_t$ times, using the mean the update would be:
$$
V'(s_t) = \frac{\#s_t \cdot V(s_t) + G_t}{\#s_t+1}
$$
which can also be seen as:
$$
V'(s_t) = \frac{\#s_t}{\#s_t+1} \cdot V(s_t) + \frac{1}{\#s_t+1}G_t
$$
Where the sum of the two coefficients sum to 1, and they are strictly positive... in other words, is a linear combination of your current estimate, and your new estimate
Now, check the constant step size formula:
$$
V'(s_t) = V(s_t) + \alpha[G_t - V(s_t)]\\
V'(s_t) = V(s_t) + \alpha G_t - \alpha V(s_t)\\
V'(s_t) = (1 - \alpha)V(s_t) + \alpha G_t
$$
And again, you can see that you are taking a linear combination of your old estimate and the new estimate
Hence the similarity/being in-line


text:
Missing types not shown as errors in VAL
Consider the following set of types for a PDDL domain:
(:types aType1 aType2 aType3 - object)

where predicates and actions have parameters of all 3 types above. I have removed one of the types (say aType1) from above. When running VAL, I do not seem to get any errors displayed. Are there other tools that may discover this error?

I don't know why people downvoted this question. It might be off-topic, but it's related to AI, and the action that you should take, voter, is to Close, then choose the closing reason, not downvote. Downvoting doesn't help the OP to understand that the question is off-topic. See https://ai.stackexchange.com/help/on-topic

Downvote is indeed weird...question is on-topic. Trying something out to see if there's a decent answer.

@haz Asking for tools is generally off-topic here, as we focus on the theory, but this question is still related to AI. I was just pointing out that, if a question is off-topic, people should vote to close it, not downvote it, unless there's also a reason to downvote it, which doesn't seem to be the case.

Yes, the pddl Python library can help you out here! It's pip installable (pip install pddl), and trying to parse the domain file will fail:
from pddl import parse_domain
domain = parse_domain('domain.pddl')


text:
Do we use RL to train ANNs, or use ANNs as par of a RL solution?
I am learning about deep learning. Currently just at a superficial level, I think I am misunderstanding how reinforcement learning and artificial neural networks are used together.
For what I first understood, artificial neural networks can be used as part of a broader RL solution, enhancing the so-called Q-learning algorithms that are normally utilised there. Yet after prompting ChatGPT, it seems that RL can also be used as a framework to train ANNs, but then it gets confusing (maybe it is hallucinating ?).

RL has nothing to do with ANN, though if you want to make it scalable, you use it with ANN (chatgpt is a ginormous compressor of a ngram model, thus you can replace with a huge table ideally)

But what about policy gradient methods like PPO that was introduced by OpenAI to fine tune their models ? (Just found that)

you can use PPO with tabular RL as well, so not sure what you mean

Your first answer suggests that we can use ANN to scale RL (for what I understand, when environment have large state spaces). So that's for "ANN used in RL".
However, I also found after further research that we can apply the RL paradigm (find the best policy that maximise the cumulative reward if I am correct) to train ANN, whether by treating ANN as a policy (policy gradient method) or as an actor (actor-critic methods).
It suggests that there is now two different relationships between ANN and RL to consider, but is it correct ?

RL is a learning algorithm, just like evolutionary strategies or SGD. ANNs can be used as the value function in RL, which is learned. The classic example is deep RL with AlphaGo. However, there are many RL methods which don’t use NNs, like Q learning as you mentioned.
A clarification by Sutton and Barto which may clear up the area of confusion: RL is both a general field of models/techniques as well as a set of learning algorithms. It depends on the context, but it seems like what you are referring to is RL as a method.

RL is not a learning algorithm. It's a field/area of study. But there are RL algorithms. I think it's important to use the right terminology to avoid further confusion.

So now that I inquired further about the question, here are some answers.
Generally speaking, when we use reinforcement learning (RL) and artificial neural networks (ANNs) in the same sentence, it's about using ANNs to approximate RL algorithms. In this case, we train the ANNs as we would normally do (Stochastic gradient descent...).
Here, the ANNs serve as the RL algorithms agents.
However, we can also use RL for :

Hyperparameter optimization and Neural architecture search (NAS), where ANNs are treated as the environment. We use RL algorithms to find the best architecture (number for layers...) or hyperparameters (activation function...). See Zoph, B., & Le, Q.V. (2017) "Neural Architecture Search with Reinforcement Learning" as an example.
Weights optimization (i.e. RL algorithms replacing the classical training algorithms), which is a very rare use case as our traditional optimizers prove to be already pretty efficient. See Bello, I., Zoph, B., Vasudevan, V., & Le, Q.V. (2017) "Neural Optimizer Search with Reinforcement Learning" as an example.

In those cases, the RL algorithms are the agent, and the ANN is the subject being optimised (environment).

I would say that this is just be the semantics of the sentence you read. I would understand the sentence 'RL can be used as a framework to train an ANN' to mean that we are using an RL algorithm to learn the weights of an ANN -- an example would be to learn the parameters of a deep Q-network, which is used to approximate Q-values.
What's important to remember is that, as @Alberto points out in the comments, RL is generally separate to ANNs. RL is a branch of machine learning that can use ANNs as function approximators for the Q-function/policy (this is the context in which one might say that RL is 'training' the ANN), but is in general not tied to them (obviously you could argue that right now, they are tied to ANNs in the sense that to get them to work in real world settings, you need a deep network for the function approximations, but this might not always be true).

I fully understand the second part, the confusion remain in the first part, I'm trying to clarify : while you say it is possible to use an RL algorithm to learn-optimise the weights of an ANN, you talk about DQN, which I thought were precisely an example of ANN being used inside a RL solution, and that DQN are trained using the classic optimisers we know (like gradient descent etc...). I am rather expecting that if RL can be use to train ANN, it's by incorporating novel optimisers (same way genetic algorithm can be use to optimise ANN weights), but that might not be the case.

okay, yes, I can see based on the answer provided that this is the angle you were trying to understand. indeed, RL has been used in neural architecture search though it is quite limited currently.