text:
How do you name your deep learning training outputs?
After some time starting the deep learning project, training output files (model weights,training configuration files) will be piled up. Naming all outputs and training files can become complicated if the clean naming convention is not used. There are some example naming styles below. I wonder that how do you manage your training outputs and training files?
'Outputs/run1/'
'Outputs/run2/'
'Outputs/run3'
    .
    .
    .

'Outputs/20230222/'
'Outputs/20230223/'
'Outputs/20230224/'
    .
    .
    .

'Outputs/WithDropout/'
'Outputs/WithDropout_RotationTransform/'
'Outputs/WithDropout_RotationTransform_AdamOptimizer/'
    .
    .
    .

Depending on how many instances of models you train you can do one of the following:

For when the amount of models is still somewhat manageable: Generate a settings file together with the model file in which you store all the hyperparameters of the model.
For when it really gets out of hand with the amount of models. Generate a random unique number which you can use to name the model, and store the settings and the file name of all models in a csv document. You can then use the csv document to retrieve the correct model name corresponding to a set of settings and a result.

It's a bit of a hassle to implement but its worth it in the end ;) You can of course also mix and match the options if that suits your needs better. Unfortunately, I do not know a simple 'hack' which allows you to do this very easily.
You can also try to add the model parameters in the name of the model itself, but in my experience this usually gets messy real fast once you realise 'oh i have to add this parameter as well', and 'oh this model does not have this parameter, but the other one does'.
If you do something like bayesian optimization, a service such as WeightsAndBiases can keep track of all this stuff for you. The applicability of such a method is of course heavily depend

I create folders for each training and store training parameters in yaml files in those folders. I add Tensorboard event files in those folders too. When I visualize the event files in the Tensorboard it becomes difficult to interpret graphs as there are many training graphs. So I wondered how people manage and interpret many training graphs easily. It could be easy if I could associate the training names with the graphs clearly. Thanks for your answer :)

You could use the most important distinctions to build a folder structure. For example I experiment with multiple model architectures (resnet, mobilenet, ...) and different types of classification (binary, multiclass and multilabel). Later in the development process it is possible new classes are added. So these parameters I include into the folder structure as well. Eventually I use the date like you showed in your example.
For example: models/resnet/multiclass_4/20230302/...
This would be a resnet multiclass model with 4 classes saved 2 march 2023.
After I save the model I gather all the relevant training outputs to save them in a separate json file in the model directory. Finally I add a custom field called "description" where I give a short explanation what I tried compared to the previous training cycle.
Disadvantages:

The description field will not help if you compare models which were not trained in succession.
You can't get a lot of information based on the folder structure.

Advantage:

No endless series of subdirectories or chaotic directory names.
Minimal information in description can help you remember which model it was.


text:
When retraining a model are you adding a layer or changing existing node values?
I am a little confused, lets take Stable Diffusion for example. I download the SD model and I then retrain for a specific art style. Am I adding a layer of nodes on top of SD or am I changing all the weights and biases on the original model?

You can do either one.

By default, you are changing the weights and biases of the original model.
But it depends on the training method you choose. If you do LoRA training, they add extra weights to the model.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?
I'm new to LSTMs, and I'm trying to do a basic timeseries prediction using stock prices. However, I'm a bit confused as to how the LSTM is supposed to remember outputs from previous timesteps when it has a many to one shape.
For example, let's say we're at timestep n, and the following timeseries is part of my input:
[[100, 10], [300, 30], [200, 20]]

And it maps to some output, let's say 1
Great. But let's say at timestep n - 1, when the input was just [[100, 10], [300, 30]], the output was 0. How will the LSTM know this?
Should I include the same data at different timesteps (using something like zero padding) with the corresponding output? Or am I totally misunderstanding something about how LSTMs work?

The hidden and carry states of an LSTM contain the current 'embedding' of the past data that has been passed through the cell. The hidden state is also taken as output at each timestep. If, as you are writing, make the hidden state of size 1, then indeed it will not remember much of the past.
However, what is more common, is to have a larger number of hidden neurons in the LSTM and then connect a fully connected dense layer to the output, which takes in this hidden state (embedding) and transforms it to a single output node. That way you can remember a lot of data, but still have a 1 neuron output.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What should I do, reinforcement learning agent gives different result on every train?
I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and  tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results on testing data (portfolio change at the end of test period). I think, its happening due random initialisation of NN parameters and solution reaching different local maxima. So, how am I to evaluate the agent if it gives anywhere from negative to positive change on every train?

Is the trading bot trained only on price data? If so, there is typically no usable pattern in that, and you should expect the bot to do about as well as throwing darts at a board with "buy" and "sell" written on it, even if your training was successful. Although there are other possibilities, including that something is wrong with your implementation or training approach or hyperparameters.

I suggest you should try fixing the seeds. Here's the code how to do it, if you are using torch and the environment is inherited from gym
import torch
import random
import numpy as np

env = import gymnasium as gym
env = gym.make("LunarLander-v2")

def set_global_seed(env, seed=42):
    random.seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = True
    env.seed(seed)
    env.action_space.seed(seed)

set_global_seed(env)

In that case should I experiment with various seeds? In some train session I got good results. Or should the whole process be agnostic of seed i.e. set the seed once and not revisit seed setting ever again. I'm asking this because what if the seed leads to a poor testing result.

The seed will always affect the result. This paper experiments with 10^4 seeds and finds it true. However, I recommend you to select about 5 seeds, e.g., 0, 1, 2, 3, 4, then tune the hyperparameters based on those seeds and never touch again. You can do 10 if you are unsure about it

What if I want to evaluate my architecture? I was thinking of training the agent multiple times and calculate average change which will give me a good idea of expected value of the change. What do you think of this approach?

That would be the same thing as I mentioned earlier. Fix the seed, do anything you want

The problem is not that the algorithm is getting inconsistent results and you need to nail it down to one of the "good" strategies.  It is instead that the algorithm is getting inconsistent results and you need to figure out how to get consistently good results.  Depending on how you've tested the good strategy the model could just be over or under-fit and will perform poorly in production.
If there is a good solution, on a long enough timescale value estimation should converge, no matter the initialization.  In supervised learning we turn to cross validation to give some indication that a model will do well on unseen data.  It is not enough to pick a well behaving initialization there and the principle holds in RL as well.  DQN was not impressive because the authors found a single initialization that played pong, they found an algorithm and data representation that could play pong.


text:
References for the theory of pretraining and unsupervised learning to improve subsequent supervised learning
I am not sure if the title of this post uses the correct terminology, so suggestions are welcome.
I have been following a lot of the ideas of using Pre-training methods on neural networks, to improve the accurancy of those networks on subsequent tasks. For example, the Word2Vec paper and others in that same line developed the idea of pretraining word embeddings, as a way to improve subsequent tasks like text translation. Transformer networks themselves being with converting raw data through content and positional embedding layers.
There are other examples of this idea as well. Chris Re at Stanford has explored the idea of "weak supervision" where a neural network is trained on a set of weak or noisy labels--that are cheap to obtain. After training on the weak labels, the network is trained on some higher quality labeled data to gain greater accuracy.
In images, this approach has had a little less success, but still attempted. There has been work on trying to pretrain a neural network on in-painting sections of an image. Once the network is pre-trained, then ML engineers will use that model to try and generate new images, or to improve performance on image segmentation tasks.
So in each of these cases, the model is trained on one task as a way to embed domain knowledge into the network for subsequent tasks. However, I have never really read an
explicit explanation or elaboration on this theory. I was wondering if anyone knew of
a good reference--paper, article, book, etc.--that discusses this theory and its approaches, limitation, etc.

we call this transfer learning. Your search may start here.

@lpounng I can see the logic of your answer. I am not sure if transfer learning is really the right word though. I always think of transfer learning as training a supervised model to classify a set of labels on one dataset; then you use that same model to perhaps learn a different set of labels on a different dataset. So it is a supervised -> supervised kind of training. I was thinking more of the relation between unsupervised -> supervised training. But I can include references to transfer learning in my search.

If the pretrain phase has no objective, we call it unsupervised learning or dimension reduction. See, isn't that like taking the PCA of a dataset, then use the transformed features to do different predictions?


text:
How to get ZINC 500k dataset?
I have been using the ZINC graph regression dataset available through pytorch geometric datasets for a while now in two of its modes (12k examples and 250k examples). However, in the PapersWithCode comparison there is always reference to a dataset with 500k examples that I am unable to find. does anyone know where a download link or something like that might be?

It might be an error by paperswithcode. I've checked some publications that are listed there and 1) The paper that published the ZINC dataset says that the full ZINC data consists of 250k samples and their smaller version contains only 12k (s. p.17). 2) The top listed papers all use either the 120k or the 12k-version and the scores they achieve for i.e. 250k are listed in the 500k leaderboard (cf. this paper p.9). I'm pretty sure this is an error and the dataset should actually be listed as 250k.


text:
What method to use for Monte-Carlo Tree Search to prefer depth search
The basic Monte-Carlo Tree Search algorithm uses the tree policy:
while v is nonterminal:
  if v is not fully expanded:
    expand v
  else:
    v = v.best_child

By always expanding a node as long as it is expandable a lot of time is consumed to explore the direct next move. When there are a large number of possible moves it can be time (and space) consuming to create nodes for all move candidates and if many of them may have similar outcome it would be preferable to descend in existing subtrees even when the node is not fully expanded yet.
The closest related method that I found is first-play urgency, but as far as I see it also creates the nodes for the actions and also only modifies the algorithm which children are selected to use more exploitation than exploration, but doesn't reduce the amount of move candidates that are created when inserting a node.
I thought about removing the check if the node is expandable and treating it like its children, but I think the UCT of a parent node doesn't get larger than the UCT of its best child because the values of the children are propagated to the parent and the visit count of the parent is always larger than the visit counts of the children, so the parent node would not be selected again later.
Should I try to find a modification for the UCT or are there other methods to allow for exploring subtrees of not fully expanded nodes while still allowing to further expand the parent node from time to time?
Another alternative would be to stochastically only generate a subset of all possible moves at the risk of missing out on a good move, so I would prefer to find a modification of UCT or the descend / expansion algorithms to revisit inner nodes that are still expandable.

When you need to expand a node, you just need to add all its children (with 0 visits). Then you select one of the children for rollout.
Is this already a problem (to add all the children as nodes)?

You can't do it with the standard UCT formula because $n_i$ is in the denominator. I also think it is there for a reason and the standard approach basically says that evaluating if an unexplored node leads to a good solution has infinite priority. The equivalent to what you're thinking of would probably be not adding a node and then you again need to make sure that the algorithm doesn't always descend in the tree to add the missing node.

I would add all children (with 0 visits) altogether. Then before testing for UCT I would check if any has 0 visits and choose from those. None with 0 visits? Then fall back to UCT.


text:
How much do we know about the architectures of the Codex (prototype) models?
The transformer model Codex by OpenAI was introduced in a 2021 paper. The paper does not give complete information about the architecture. Below I've quoted all the passages in the paper that give hints as to the architecture:

...we hypothesized that a specialized GPT model, called Codex, could excel at a variety of coding tasks. This paper describes several early Codex models, whose descendants power GitHub Copilot and the Codex models in the OpenAI API.

We fine-tune GPT models containing up to 12B parameters on code to produce Codex.

...we hypothesized that it would be beneficial to fine-tune from the GPT-3 (Brown et al., 2020) model family...

A table in the paper (Table 1) lists various Codex models studied in the paper, giving only the number of parameters for each one: 12M, 25M, 42M, 85M, 300M, 679M, 2.5B, and 12B.
So, what we can glean from this is: the models discussed in the paper are not the production Codex model that powers Copilot, but prototypes (so the production model could in theory be completely different - that's fine, I'm not asking about that one). A number of different size versions of the model are studied in this paper, and the only description of their architecture is that they are "GPT models".
I'm not sure if this is underspecified, or if I just don't know the field well enough. Saying a model is "a GPT model" does not seem to specify the architecture uniquely, to me. I know I can go read the GPT-3 paper for more information, but at minimum if you say something is say a 300M parameter "GPT-3 model", it seems to me I still don't know how many layers there are, how many attention heads, etc.
Can we deduce more about the shapes of these models? At least the number of layers and parameters per layer?


text:
Tool for detecting AI Generated code?
Can you recommend any tools for detecting AI generated code? I am aware of AI-generated text detectors such as ZeroGPT and OpenAI-Classifier, but I am specifically interested in detecting AI-generated code. As of now, I have not been able to find any publicly available tools for this purpose.
I would greatly appreciate any guidance or recommendations on available tools or methods for detecting AI-generated code. Thank you in advance for your help.

Hi @Haseeb Mir, and welcome to AI Stack Exchange! This question may not be considered on-topic for this site since it is asking about specific software recommendations. It might be better suited for Software Recommendations Stack Exchange. If you have the time, please look at the on-topic page for AI Stack Exchange to determine if your question is on-topic or not. Thank you for posting, and we hope to see more of your questions soon!

https://copyleaks.com/ai-content-detector
This one i found to be interesting enough for coding


text:
Representing non-fixed state-spaces in RL algorithms
How I can define the state of an environment where I don’t have a fixed representation for states? E.g. the state at one step is a list of size 5 and in another state is a list of size 6. What is the general RL technique for representing states of such environments where we don't have a fixed-size representation of the environment?


text:
How does Proximal Policy Optimization deal with sparse reward
In the original paper, the objective of PPO is as follows:. My question is, how does this objective behave in a sparse reward setting (i.e., reward is only given after a sequence of actions were taken)? In this case we don't have $\hat{A}_{t}$ defined for every $t$.

Why won't we A_{hat}_{t} for every t? There is usually a tuple of (state, reward, next_state) associated with every time step t. For sparse reward settings, the reward will be 0 for non-reward states.

@desert_ranger yes you can think of reward as 0 for those states, but in some situations it might be undefined

It is the user who designs the reward for each step. Therefore, as long as the environment is formulated correctly, this shouldn't happen.

@desert_ranger think of Go. By default, not every move has a reward assigned. Are you suggesting going down the reward-shaping route to introduce artificial rewards?

A reward must exist for every $t$. As we are operating in an MDP, each $t$ corresponds to a transition from a state $s$ to another state $s'$ given an action $a$. By the definition of the MDP, a reward must be associated with this transition, otherwise you are not working in a proper MDP. This is true for the game of Go. I believe the work by Deepmind just assigned a score of +1 for winning, -1 for losing, and every intermediate step was assigned a reward of 0.

@DavidIreland yeah so to fit into MDP framework, can assign dummy reward 0 to the intermediate states.


text:
Using AI to Play a Coin Flipping Game?
Imagine I have the following coins:
   coin_name success_prob failure_prob positive_reward negative_reward expected_reward
1     coin_1    0.5229156   0.47708442       0.7901860      -1.0909964     -0.10729679
2     coin_2    0.7211000   0.27889996       1.9919323      -1.6501755      0.97614858
3     coin_3    0.8994624   0.10053758       1.0815446      -1.4799171      0.82402145
4     coin_4    0.5609496   0.43905037       2.2192418      -2.5168569      0.13985590
5     coin_5    0.7804740   0.21952601       2.3571215      -2.9998136      1.18113491
6     coin_6    0.6032657   0.39673431       0.2093356      -0.2257176      0.03673507
7     coin_7    0.5637658   0.43623417       1.3674532      -0.4321832      0.58239026
8     coin_8    0.8766539   0.12334607       2.2418057      -1.1516021      1.82324220
9     coin_9    0.9475227   0.05247732       1.4234440      -0.7954108      1.30700448
10   coin_10    0.6872314   0.31276861       1.1958754      -3.4163135     -0.24667254
11   coin_11    0.8325576   0.16744240       2.3964000      -1.8754937      1.68110388
12   coin_12    0.5474203   0.45257967       1.1880020      -0.9212376      0.23340304
13   coin_13    0.6919848   0.30801518       1.7261695      -2.7642130      0.34306353
14   coin_14    0.6371918   0.36280818       1.4743202      -2.4181846      0.06208761
15   coin_15    0.9073200   0.09267998       0.3470192      -2.7124723      0.06346562
16   coin_16    0.7242582   0.27574183       2.2595799      -3.0144799      0.80530102
17   coin_17    0.9050322   0.09496782       0.6906106      -2.0905425      0.42649051
18   coin_18    0.9061948   0.09380525       0.2009429      -2.0933373     -0.01427263
19   coin_19    0.8971712   0.10282884       0.8870097      -2.2459255      0.56485364
20   coin_20    0.7199158   0.28008416       2.3908088      -2.9816879      0.88605758
21   coin_21    0.8772376   0.12276242       2.2348944      -3.0280594      1.58880142
22   coin_22    0.8146106   0.18538943       1.7627282      -2.7076841      0.93396098
23   coin_23    0.8550912   0.14490880       1.6372164      -1.9157277      1.12236350
24   coin_24    0.5003124   0.49968761       2.4862475      -2.5956930     -0.05313525
25   coin_25    0.7376583   0.26234171       1.6736939      -0.5833858      1.08156777

Now, imagine a game where two players (player 1 always starts the game and both players start with reward = 0) take turns choosing and flipping coins while tallying their score (player 1 always starts) : the first player to reach a reward = + 10 wins and the first player to reach a reward = -10 loses.
However now, imagine the following modification: At the start of the game - the success probabilities and rewards are unknown to both players. The players follow an "exploration-exploitation paradigm" where they try to estimate the probabilities and rewards of different coins based on the data they observe and the data of the other player - and then make decisions regarding what coin to pick based on the current state of the game.
My Question: I would like to learn how AI algorithms like Reinforcement Learning (e.g. Q-Learning, Monte Carlo Tree Search, etc.) can be used to develop strategies for optimally playing this game as more information about probabilities/rewards becomes known while the state of the game evolves (e.g. if a player is about to win, the other player might take a risk-averse decision and flip coins he knows have higher payoffs but lesser probabilities of success).
Conceptually, I think it might be possible to design a random simulation in which both players play this game randomly again and again - and then try to incorporate different strategies based on Reinforcement Learning. I thought of two scenarios:

Scenario 1: The probabilities and rewards stay the same in each game - in this scenario, perhaps the AI algorithm will eventually play enough in which success probabilities and rewards can be reliably estimated and an optimal policy can then be estimated
Scenario 2: The probabilities and rewards are reset after each game - in this scenario, perhaps an AI algorithm can learn at what points is it better to explore vs stick to low risk strategies

I have never worked with Reinforcement Learning before - can someone please show me how Python libraries can be used to paly this game using Reinforcement Learning algorithms?
Thanks!

If I read correctly, your question is about how can I implement this simulation rather than how AI algorithms learn. If it is the case, you might want to implement a custom Gym environment that suits your scenario.

Are players allowed to observe other player choices? Are they allowed to observe other player results?

One other thing: You will be generating the coin probabilities at the start, and the players will not know them. However, wil the players know the distribution of probabilties (e.g. will they know that each coin has success_prob chosen using equally probably from range $[0.5,0.95]$?

@ Neil Slater: Thank you for your reply! Players are allowed to observe other players choices and results. Yes, lets say that all coins have success_probabilities that are equally probable between some range (e.g. 0-5- 0.95). Thank you so much for everyting


text:
Question about "unsupervised learning objective" in the GPT-2 paper
I find it difficult to understand the following from the GPT-2 paper.

Language modeling is also able to, in principle, learn the tasks of McCann et al. (2018) without the need for explicit supervision of which symbols are the outputs to be predicted. Since the supervised objective is the the same as the unsupervised objective but only evaluated on a subset of the sequence, the global minimum of the unsupervised objective is also the global minimum of the supervised objective.

I assume the supervised objective is some loss function if we are to train on labeled data sets that look like (task, input) -> output.  But what is the unsupervised objective here?
And why does it say that the supervised objective is only evaluated on a subset of the sequence? My understanding is that GPT-2 is never specifically trained on some of these tasks. So how can it  say that it is evaluated on a subset of the sequence?
I would appreciate if someone helps me understand.


text:
Reference request: AI panel discussion 2012
I have seen (probably on youtube) the recording of a panel discussion at an AI conference around 2012 (year could be wrong, but close) where the audience was polled for the question of when will AI crack Go. Most of the participants predicted it would take at least decades (some thought 100+ years). Then 4 years later AlphaGo arrived. I've searched for hours, but I can't find that video. If anyone has seen it could they link it?


text:
Can I use zero-padded input and output layers in a 1D convnet to predict an element of interest from a variable-length input sequence?
I have developed a small encoding algorithm that accepts a time series of n = 750 samples and m = 1 feature from a scientific instrument, and encodes/transforms it into a new ordered sequence with an n that can vary between 1 and 12, and m = 3.
I want to build a neural network that can read the sequence of elements (likely via 1D convolution and subsequent dense layers?), and make a prediction as to which element is the 'element of interest', which corresponds to a particular position in the original time series.
I am not sure if I am going about structure this the right way, so I wanted to ask if this looks roughly correct. The network should ideally read this input tensor, consider each element's features in the context of adjacent elements' features and then associate a fractional probability that a particular element (say, the green one, in the image) is the one I care about.

Step 1 takes a time series, finds point of interests, and associated features with those points. This step may find anywhere between 1 and 12 points of interest.
Step 2, should ideally look at each point of interest, and figure out which one is the "true" one, based on its local and global environment in the encoded input tensor.
I hope this makes sense, I am not a deep learning expert by any means, but I have done some work in the past with CNNs, so please let me know if any additional information would be helpful.

For the output layer size, is it fixed? or is it varied based on the previous output?

That is a great question, and something I neglected to include. The output is varied. The output corresponds to an input element. So, it's not a 'classification' problem in the sense that there are 12 possible categories, but rather a variable number of categories depending on the input. I want to select the index of the input.


text:
Second Language for Machine Learning
I want to pursue Machine Learning and Data Science as a future career and wanted to get your advice on a good second language that will be useful in the ML field. Currently, I am learning Python and using libraries such as Pandas and Tensorflow to create some practical projects. However, for a job, I feel like I would need a second language to supplement Python. What recommendation do you guys have for a second language? Additionally, if you know of any resources or materials that could help me accelerate my learning in this area, I would appreciate any recommendations you may have.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

Sorry about the general question. I just wanted to know if there is a second language other than Python that is highly used in the Machine Learning/ Data Science industry to build Machine Learning models.

Python has been quite important, I personally use it for my job, mostly to write APIs and custom actions/classifiers, otherwise, probably you can learn R and SQL.

I recommend that you learn SQL as your second language. Most data scientists need to pull data from SQL databases. That doesn't mean you won't have to deal with NoSQL databases, but SQL is more common. If you don't know SQL, you will always need to ask someone who knows it and will be at their mercy to give you what you want. A data scientist should understand their data's provenance and pedigree, the data model, schema, data dictionaries, and how to query the database.
Here is a course by Coursera for furthering your career: Databases and SQL for Data Science with Python

Would you recommend learning a second language to perform Machine Learning, or is knowing Python enough to build the machine learning models? I've read articles talking about how companies use Python to build the model but use faster languages such as C++ to deploy the model, and I don't know if I have to know C++ as well to get hired as a Machine Learning Engineer.

Python is enough for most jobs. C++ is a great language to know, but most ML people don't know it and don't need it.


text:
Why is the variational lower bound is easier to compute than the original marginal distribution?
Why is the ELBO of $p(x)=\int p(x|z)p(z)\mathrm{d}z$ easier to compute/estimate than the expression itself? Can we compute this quantity itself through sampling in the same way? I understanding that aggregating over data means taking log before summing over, but does it create any complication?

To calculate $p(x) = \int p(x|z) p(z) dz$, you have to calculate it with all configurations of $p(x|z)$ and $p(z)$, which scales exponentially with time. Thus, it is easier to estimate $p(x)$ than to calculate it correctly.

I don't think that makes sense. x is a fixed value given by data. I'm basically asking why p(x) can't be estimated through sampling the same way as the ELBO can. Also, I'm not sure what time means in your comment. I'll taking about VAEs in general.

Time here means the time complexity you need to calculate all configurations possible with $\int p(x|z) p(z)$. When $z$ is large, it is computationally exhaustive to calculate the value. We are generally good with estimations.

As I mentioned above, this shouldn't be the reason as ELBO also involves an integral over z. It's an expectation with q(z).


text:
How to best implement Accelerate for vary large models
This was pointed out to me: https://huggingface.co/blog/accelerate-large-models
and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution script.
Does anyone know of a good library, or example code that implements Accelerate for large models to use less VRAM / RAM?
This is what I tried to run last:
from accelerate import infer_auto_device_map, init_empty_weights

from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig

import torch

checkpoint = "bigscience/mt0-small"

config = AutoConfig.from_pretrained(checkpoint)

with init_empty_weights():

model = AutoModelForCausalLM.from_config(config)

device_map = infer_auto_device_map(model, no_split_module_classes=["OPTDecoderLayer"], dtype="float16")

tokenizer = AutoTokenizer.from_pretrained(checkpoint)

device_map["model.decoder.layers.37"] = "disk"

model = AutoModelForCausalLM.from_pretrained(

checkpoint, device_map=device_map, offload_folder="offload", offload_state_dict = True, torch_dtype=torch.float16

)

Goal is to run Flan-T5 or BLOOMZ locally, using disk space, to run the models on lower resources.
This code throws the error:
ValueError: Unrecognized configuration class <class 'transformers.models.mt5.configuration_mt5.MT5Config'> for this kind of AutoModel: AutoModelForCausalLM

That is another issue, it would be good to know how to supply a configuration or convert the config to a supported type.
And not sure if the code I put together would work once that is addressed.
Any suggestions?

hi, do you solved? same problem here, i found another strategy bud did not work for me https://huggingface.co/docs/accelerate/usage_guides/big_modeling

Still foggy on this, but thanks for posting the link. It could be pretty useful. These days I would be most likely to use something like GPT4ALL or  oobabooga's webui if possible


text:
Why optimise log p(x) rather than log p(x|z) in a Variational AutoEncoder?
Background
The loss function in a Variational AutoEncoder is the Evidence Lower Bound (ELBO):
$\mathbb{E}_q[log$ $p(x|z)] - KL[q(z)||p(z)]$
And has this inequality:
$log$ $p(x) \ge \mathbb{E}_q[log$ $p(x|z)] - KL[q(z)||p(z)]$
It is said in the literature that we want to maximise $log$ $p(x)$, but instead, we maximise the ELBO, which we know is less than or equal to $log$ $p(x)$ (because this is an easier objective).
First Question
I am struggling to understand why the objective is to maximise $log$ $p(x)$ in the first place.
Why is it not enough to maximise $\mathbb{E}_q[log$ $p(x|z)]$, which is the reconstruction loss?
I think I don't understand what exactly $p(x)$ means here.
I thought $p(x)$ was just the distribution of $x$, which are the input variables we have observed.
So how can we maximise $p(x)$, if it is the actual distribution of data in the real world?
Main Question
The Variational AutoEncoder is trying to learn an encoding from $x$ to $z$ and decoding from $z$ to $x'$, such that the difference between $x$ and $x'$ is minimized.
How does $log$ $p(x)$ relate to this?
What does it mean and how does maximising it help?

TLDR: We're doing a maximum likelihood fit of our model. The VAE sets this up in a way that doesn't require evaluating the model likelihood, but instead expresses a lower bound in terms of reconstruction error.
Here's an explanation in two parts.
1. Variational inference in general
We have a generative model  $z \rightarrow x$ with latent variable $z \sim p(z)$, and observations, $x$, generated by $p(x|z)$. We want to get the posterior $p(z|x)$, but this inference may be computationally infeasible even if $p(z)$ and $p(x|z)$ are known because $p(x)$ is hard to evaluate. So, we instead approximate $p(z|x)$ by $q(z)$.
Denoting the ELBO by $\mathcal{L}(x)$, we have (by expanding $D_{KL}$):
$$ \log p(x) = D_{KL}\big(q(z) \big|\big| p(z|x) \big) + \mathcal{L}(x). $$
Note that $p(x)$ is part of the generative model so, given data $x$, $p(x)$ is constant regardless of the approximation $q(z)$. This means that increasing $\mathcal{L}(x)$ by changing $q(z)$ is equivalent to decreasing $D_{KL}\big(q(z) \big|\big| p(z|x) \big)$ and vice versa.
Thus, the max ELBO approximation is the best in the sense that $q(z)$ is closest to $p(z|x)$ in KL terms. This is the classic reasoning for maximizing the ELBO.
2. Variational autoencoder
For the variational autoencoder model, we

choose $p(z)$ for the latent space (eg. gaussian)
parameterize $p(x | z)$ by the decoder network
parameterize $q(z)$ by the encoder network (eg. $z \sim \mathcal{N}( \mu(x), \sigma(x))$)

Now $p(x|z)$ is not fixed, but rather fit. So, as we train, we want to improve both the approximate inference model (encoder) and the generative model (decoder). This means that $p(x)$ is not fixed either as we train, so by increasing $L(x)$, we can either increase $\log p(x)$ or decrease $D_{KL}\big(q(z) \big|\big| p(z|x) \big)$ or both.
Why is increasing $\log p(x)$ a good thing? $\log p(x)$ tells us how likely that data we observe are under our model, this is a max likelihood fit of our (decoder parameterized) model. However, like the true posterior, $p(x)$ is difficult to calculate closed form, hence the ELBO.
That's the key idea of the VAE: it expresses the model max likelihood objective (which is hard to compute) in terms of reconstruction loss (which is easy to compute), giving us a generative model and approximate posterior along the way.

I thought $p(x)$ was just the distribution of $x$, which are the input variables we have observed. So how can we maximize $p(x)$, if it is the actual distribution of data in the real world?

I think the main confusion is coming from what "maximize" means in all that. What you have is a family of functions $p_\theta(x)$, parametrized by some parameters $\theta$. It could be a neural network or some distribution in analytic form or some kind of graphical model. By varying the parameters $\theta$ you are getting different distributions over $x$.
Now, you also have a sample of $x$'es that you observed "in the real world".  And we would like to find such parameters $\theta$ that $p_\theta(x)$ reflects the real world distribution as close as possible. The standard way to do that is to find $\theta$ that maximizes the log-likelihood of the observed data $x$:
$$\theta_{best} = \arg \max_\theta \log p_\theta(x)$$
This is called the maximum likelihood estimation. There are many extra details about it in the context of variational inference and VAEs, but that's the basic idea of what "maximize" means in all these cases.

The Variational AutoEncoder is trying to learn an encoding from $x$ to $z$ and decoding from $z$ to $x'$, such that the difference between $x$ and $x'$ is minimized.

You've described a general case of autoencoder - "variational" adds some extra  details to it.
The "decoder" in VAE is not just a function from  $z$ to $x$ - it is a probability distribution $p_\theta(x,z)$. With $\theta$ being parameters of the decoding neural network. Given some values for latent variables $z$ you obtain a distribution over observed variables $x$.
For the "encoder" we would ideally like to just have a conditional probability distribution of $z$ given $x$:
$$p_\theta(z|x) = \frac{p_\theta(x,z)}{p_\theta(x)} = \frac{p_\theta(x,z)}{\int  p_\theta(x,z) dz}$$
The problem is in the denominator - the integral for marginal density of observations (also called evidence) $p_\theta(x)$ is computationally very hard to evaluate. This is what is called intractability of the evidence and the reason for the whole field of variational inference to exist.
VAE approximates the true posterior $p_\theta(z|x)$ with a variational approximation  $q_\phi(z|x)$. Where $\phi$ are parameters of the encoder network, the distribution  usually factorizes following mean field approximation approach.
The need for $q_\phi(z|x)$ to approximate $p_\theta(z|x)$ leads to the whole ELBO derivation and the KL divergence term in the VAE loss.

Thank you for your helpful answer.
Does this mean that p(x) is the entire model? i.e. it is the function that includes both encoder and decoder and takes an x as input and as output produces another x', which could be different each time due to the sampling from the latent space?
And we are trying to make sure that p(x) over all the x in our dataset produces a distribution of x's that are as similar as possible to our original dataset?

@TitusBuckworth I would say that the "entire model" is $p_\theta(x, z)$. (Again, I encourage you to write the optimization parameters like $\theta$ explicitly.) You observe only $x$ and you want to simultaneously calculate a good maximum likelihood estimation of $\theta$ and infer the latent variables $z$, which you don't observe. This is a hard problem due to intractability of the integral for the evidence.  Variational approximation $q_\phi(z|x)$ is used to deal with this intractability.

@TitusBuckworth See this answer https://ai.stackexchange.com/a/27991/20538 for in-depth math on that.

Thanks. So you say that $p_\theta(x)$ is a family of functions. What is the input and output of those functions? What is the input and output for $p_\theta(x)$ with optimal $\theta$?

@TitusBuckworth It is a family of probability distributions. I'd really recommend reading Wikipedia link on Maximum Likelihood Estimation.


text:
How do I know if image after image enhancement is better than before? (Image Preprocessing)
There are few common image enhancement:
1. brightness -> r = s + b
2. negative -> s = 255 - r
3. contrast -> scretching (flexible) dan thresholding (binary image)
4. smoothing -> generate blur
    penapis non linear (min, max, median)
5. sharpening (high pass filter), suit for getting edge

How do I know what should I enchance? Is it brightness, contrast, or what?
And how do I know if image is better after enhancement or not (evaluating result) objectively instead of subjectively by our perspective? Mainly for medical imaging such as USG, CT-SCAN, etc...
As far As I know, I can see the histogram, but I still don't know what histogram to evaluate the result.

"Better" has a very wide definition. Is it faster? Better accuracy? Better precision? First, you need to define what you mean by "Better"
In general, it depends on your task (mainly) and your model. Some image processing methods will give benefit in some cases, but they also degrade performance in other cases (e.g. changing rgb image to b/w). I suggest two approaches:

The easiest one is to use it in the downstream task, which means your main task, and see the evaluation result. For example, change the image brightness and see the accuracy of your model, whether it helps or not. With this approach, you need to do several trial-error experiments

If you don't want to use the downstream task, you can read the previous research or understand your problem domain and model behavior. It could give you some information (or intuition) about what's good and not. For example, a quick search leads me to this research that shows some techniques for data augmentation affect the model performance for MRI.

As said in other answers, it can be very difficult to quantify "better" and this depends on your problem domain. However, we can quantify "different" (with respect to the original image). This may be able to tell you how much of an effect your changes may have step-by-step.
A common metric for this is the structural similarity index, which is particularly concerned with image texture. Although regular (Shannon's) entropy isn't suitable for images, Larkin (2016) proposes a modification, dubbed "delentropy" for this task. I use an implementation from this gist, which I've condensed make it into a easy to use function:
def delentropy(image):
    # Using a 2x2 difference kernel [[-1,+1],[-1,+1]] results in artifacts!
    # In tests the deldensity seemed to follow a diagonal because of the
    # assymetry introduced by the backward/forward difference
    # the central difference correspond to a convolution kernel of
    # [[-1,0,1],[-1,0,1],[-1,0,1]] and its transposed, produces a symmetric
    # deldensity for random noise.
    if True:
        # see paper eq. (4)
        fx = ( image[:,2:] - image[:,:-2] )[1:-1,:]
        fy = ( image[2:,:] - image[:-2,:] )[:,1:-1]
    else:
        # throw away last row, because it seems to show some artifacts which it shouldn't really
        # Cleaning this up does not seem to work
        kernelDiffY = np.array( [ [-1,-1],[1,1] ] )
        fx = signal.fftconvolve( image, kernelDiffY.T ).astype( image.dtype )[:-1,:-1]
        fy = signal.fftconvolve( image, kernelDiffY   ).astype( image.dtype )[:-1,:-1]
    diffRange = np.max( [ np.abs( fx.min() ), np.abs( fx.max() ), np.abs( fy.min() ), np.abs( fy.max() ) ] )
    if diffRange >= 200   and diffRange <= 255  : diffRange = 255
    if diffRange >= 60000 and diffRange <= 65535: diffRange = 65535

    # see paper eq. (17)
    # The bin edges must be integers, that's why the number of bins and range depends on each other
    nBins = min( 1024, 2*diffRange+1 )
    if image.dtype == float:
        nBins = 1024
    # Centering the bins is necessary because else all value will lie on
    # the bin edges thereby leading to assymetric artifacts
    dbin = 0 if image.dtype == float else 0.5
    r = diffRange + dbin
    delDensity, xedges, yedges = np.histogram2d( fx.flatten(), fy.flatten(), bins = nBins, range = [ [-r,r], [-r,r] ] )
    if nBins == 2*diffRange+1:
        assert( xedges[1] - xedges[0] == 1.0 )
        assert( yedges[1] - yedges[0] == 1.0 )

    # Normalization for entropy calculation. np.sum( H ) should be ( imageWidth-1 )*( imageHeight-1 )
    # The -1 stems from the lost pixels when calculating the gradients with non-periodic boundary conditions
    #assert( np.product( np.array( image.shape ) - 1 ) == np.sum( delDensity ) )
    delDensity = delDensity / np.sum( delDensity ) # see paper eq. (17)
    delDensity = delDensity.T
    # "The entropy is a sum of terms of the form p log(p). When p=0 you instead use the limiting value (as p approaches 0 from above), which is 0."
    # The 0.5 factor is discussed in the paper chapter "4.3 Papoulis generalized sampling halves the delentropy"
    H = - 0.5 * np.sum( delDensity[ delDensity.nonzero() ] * np.log2( delDensity[ delDensity.nonzero() ] ) ) # see paper eq. (16)
    return H

In any case, I'll reiterate that downstream performance metrics are the way to quantify "better," but inspecting the effect of each filter will help guide you on what changes are could be impactful and which may not be.

I've worked on color enhancement and your question is very valid. In practice you have human experts qualitatively evaluating the algorithms. The other option is to use quantitative evaluation that is what your question points to. Quantitative evaluation depends on the task, as image enhancement includes denoising, color enhancement, sharpening, superesolution and more.
For color enhancement researchers use datasets that have i) pairs of images before and after enhancement (e.g. Adobe-MIT5K dataset) or ii) good images or bad images (can involve ratings). In NeurIPS 2022 a dataset involving edited images and natural language comments was introduced (link).
For the first case in which you have a target image the metrics are usually structural similarity, PSNR and norms in different colorspaces (e.g. CIELab).
There are many methods that do image enhancement and the main difficulty is to evaluate them. Once you have a way to automatically evaluate them you can probably optimize for that criteria. Don't forget that it's highly subjective, it's been shown that in average people in tropical regions (e.g. china) prefer more contrasted and vibrant colors that those in northern Europe.


text:
Transformer parallelization during training
What does it mean that the decoder can be parallelized during training?
Let's assume a transformer (with both encoder and decoder) is employed for a time-series prediction. I.e. from the input sequence x_0, ..., x_N we want to predict y_0, ..., y_N. Is this the way that parallelization occurs during training?

form the batch [], [y_0], ..., [y_0, ..., y_N-1]
feed this batch to the transformer, together with the input sequence
we will obtain the batch Y_0, Y_1, ..., Y_N
compare against y_0, ..., y_N and form the loss (*)

(*) here, some teacher ratio techiques may be employed, so that more passes may be required

Does this answer your question? Can the decoder in a transformer model be parallelized like the encoder?

In fact no, I asked my question after reading that post. @Minh-LongLuu


text:
Training model after Cross Validation: train one of the *k* split models plus the remaining data, or train from scratch?
So after a, say, 5-fold CV, you are left with 5 models, each trained on 80% of the data. You now want to have the best model possible, i.e. train it on all data. In order to save computation time, can you 'safely' train one of the 5 models on its respective validation data for the same amount of epochs and call it a day? Or do you need to train the entire model from scratch?

The purpose of k-fold is to perform hyperparameter tuning more than "finding the best model". We want the hyperparameters of the model to be able to generalize to all folds of the data. When you finish k-fold CV, in practice, you have to train a model from scratch again with the training + validation data, however, it is still not a bad idea to use all 5 models, and then averaging predictions on all models.


text:
Confusion in subscript for n-step TD(0)
This is n-step TD(0) update rule:
$v_{k+n}(x_k)=v_{k+n-1}(x_k)+\alpha [g_{k:k+n}-v_{k+n-1}(x_k)]$
Why is the subscript on the left hand-side of equation "k+n", not "k+n-1"? Does the subscript for $v$ denote time step or iteration number? I am confused.

n-step TD(0) is an extension of TD(0) that uses a sequence of forward n rewards and estimated state values to update the value estimate of the current state $x_k$, and TD(0) and Monte Carlo prediction are two extreme cases of it. In some reference such as Sutton & Barto’s the subscript is using $t$ instead of your $k$ and you can treat it simply as time step of the MDP though nothing prevents you from treating it as iteration number within a loop of an episode. And finally the stochastic approximation iterative equation above is probably not real code implementation thus it has to keep exact time step (k+n) on the LHS.


text:
Prioritized experience replay correction with off-policy estimators
Prioritized exeperience replay (PER) biases the sampling and introduces importance sampling (IS) correction to the Q-function update.
Weights are $w = \frac{1}{N P}^\beta$, where $N$ is the batch size and $P$ is the sampling probability.
Off-policy estimators like Retrace and V-Trace also use IS correction between the behavior and the target policy. Weights are $w = \pi / \mu$, where $\pi$ is the target policy and $\mu$ is the behavior policy. Weights are further clipped.
Recent methods like APE-X, Reactor, and LASER combine both techniques: they use a replay memory where n-step samples are drawn to compute value function target with off-policy estimators.
However, it is unclear to me how they combine off-policy correction and PER correction.
How are weights from off-policy estimators and weights from PER combined in methods like APE-X, Reactor, and LASER?


text:
How to detect outlier images?
Before I describe my challenge, I want to point out that I have searched extensively online for "outlier image detection", "anomaly images detection", etc., but all returned results are about finding anomalies or outliers within an image (e.g., a defect in a machined part, a tear in fabric, etc.).
This is not what I am looking for: I am looking to identify images (whole images) that depict an object that is different from the other images. For instance, I have a stream of images from a live camera. Each image depicts a single dog (different breeds) and each image is centered on the dog. A few images depict a cat, rather than a dog... I would like to detect these cat images (outliers).
This is not a simple classification problem, where I have a labeled data set of cats and dogs... because I don't want to define the 'outlier'. In other words, the approach must be adaptable to the data: if I were to suddenly add a few images of trash, the model must flag those trash images as novel/outliers because in comparison to the number of dog images, trash images (like cat images) are pretty rare (or new).
Would something like sequential k-means work for this problem? I thought about sequential k-means, since it operates on a data stream and will adapt to the number of clusters (i.e, k does not need to be pre-defined).
What other approaches would work? Any resources (e.g., links to tutorials, GitHub repositories, specific algorithms, methodologies) that might help?

Hi there @pookie you can approach this problem using unsupervised anomaly detection techniques. One such technique is to use an autoencoder neural network.
The idea is to train an autoencoder on a large set of dog images, and then use the trained network to encode (compress) each image into a lower-dimensional representation. The encoded representations should capture the key features of dog images. The network can then be used to reconstruct the original image from the encoded representation.
Now, when a new image (e.g., a cat image or a trash image) is presented, the network will not be able to reconstruct it well because it has not seen that kind of image before. The reconstruction error will be higher for the novel image, which can be used to identify it as an outlier.
You can use this approach to build a threshold-based anomaly detection system, where any image with a reconstruction error above a certain threshold is considered an outlier. Alternatively, you can use a more sophisticated anomaly detection algorithm like One-class SVM, Isolation Forest, or Local Outlier Factor.
Another option is to use clustering techniques like k-means or DBSCAN to cluster the dog images and identify clusters with very few points as outliers. However, this approach may not work well if the outlier images are visually similar to dog images.
Here are some resources that may help you get started:

Anomaly Detection with Autoencoders using Keras in Python: machinecurve/anomalyDetectionAutoencoder-using-Keras
One-class SVM for anomaly detection: scikit-learn/oneClassSVM
Isolation Forest for anomaly detection: scikit-learn/isolationForest
Local Outlier Factor for anomaly detection: scikit-learn/localOutlierFactor

If you found this helpful, don't forget to upvote :)

Thank you for the response -- very helpful! The autoencoder approach sounds like the right way to go. The autoencoder link you provided is broken, though.
Also, do you know of any good (and relevant) OC-SVM tutorials?
Also, welcome to AI SE!

You're welcome but I'm sorry to hear that the link I provided for autoencoder isn't working, here's another one [link] (https://machinelearningmastery.com/lstm-autoencoders/). You may also consider taking an online course on machine learning or deep learning that covers autoencoders and anomaly detection: link. And yepp, here's a resource that covers the basics of OC-SVM and provide examples of how to use it for outlier detection link

Thanks! Now, what if I want to update the model? For instance, after the model has seen loads of images of cats, cats should be considered "normal" (no longer an outlier). Can the model auto-update over time as it learns what is "normal" for the environment in which it is deployed?

Yes, u can update the model over time as it sees new data which is called online learning/incremental learning Ig, where the model is updated with new data as it arrives, rather than being trained on a fixed dataset.
One approach is to use a variant of the algorithms like One-Class SVM or autoencoders, that support incremental learning. Eg. in case of One-Class SVM, you could use an algorithm such as online SVM or stochastic gradient descent SVM to update the model as new data arrives.

Another approach is to use a streaming clustering algorithm such as CluStream or D-Stream, which can adapt to changing data distributions over time.
The key is to manage the trade-off between stability and flexibility. U want the model to be stable enough to capture the underlying patterns in the data, but on the other hand, you also want it to be flexible enough to adapt to changes in the data distribution. The right balance depends on the specifics of ur problem and the nature of the data u are working with.

I agree with Triple S' answer, but as a preprocessing step I suggest to use some pre-trained image classification network without the last fully-connected layer. This will give you robust features which represent characteristics of the input image. But since these representations are high-dimensional, I'd apply an autoencoder to these features.
Actually I have implemented and blogged about such approach, but I'm not sure whether links to personal blogs are discouraged or not. Well at least each answer should work as a stand-alone resource. In my case the images originated from videos, and I wanted to cluster either the frames themselves or video clips.

I used this for mountain biking videos, which resulted in this kind of 2D map:

Each line represents a different mountain biking / downhill clip. Anyway, in your case I'd use 2D coordinates for outlier detection. They seem to contain enough interesting data, and are so easy to visualize as well!
Link to my blog article: Image and video clustering with an autoencoder. It also has examples on clustering frames from either Arnold Schwarzenegger or James Bond films. It could group conceptually similar images together in the 2D plane, from different films.


text:
Why is Soft Q Learning not an Actor Critic method?
I've been reading these two papers from Haarnoja et. al.:

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement
Learning with a Stochastic Actor
Reinforcement Learning with Deep Energy-Based Policies

As far as I can tell, Soft Q-Learning (SQL) and SAC appear very similar. Why is SQL not considered an Actor-Critic method, even though it has an action value network (critic?) and policy network (actor?)? I also cannot seem to find a consensus on the exact definition of an Actor-Critic method.

Indeed SQL is very similar to actor-critic method which has a soft Q-function critic network with parameter $\theta$ and an actor policy network with parameter $\phi$, and in fact the paper "Equivalence Between Policy Gradients and Soft Q-Learning" by Schulman et al proves equivalence between the gradient of soft Q-learning within maximum entropy RL framework and natural policies with entropy regularization in its gradient estimator, and further talked about soft Q-learning:

Haarnoja et al. [2017] work in the same setting of soft Q-learning as the current paper, and they are
concerned with tasks with high-dimensional action spaces, where we would like to learn stochastic
policies that are multi-modal, and we would like to use Q-functions for which there is no closed-form way of sampling from the Boltzmann distribution $π(a|s) ∝ π(a|s) exp(Q(s,a)/τ)$. Hence, they use a method called Stein Variational Gradient Descent to derive a procedure that jointly updates the Q-function and a policy $π$, which approximately samples from the Boltzmann distribution—this resembles variational inference, where one makes use of an approximate posterior distribution.

Having said that, in terms of Haarnoja et al's own words about the subtle difference between SQL and actor-critic method in your first reference below, the estimated sampled action from SVGD network is not used directly to affect the next soft Q-function since the critic network uses minibatch experiences from its stored replay memory to update its parameter instead of the usual advantage function. If it's not sampled accurately enough it may not be stable and converge to an optimal final (stochastic) solution.

Although the soft Q-learning algorithm proposed by Haarnoja et al. (2017) has a value function and actor network, it is not a true actor-critic algorithm: the Q-function is estimating the optimal Q-function, and the actor does not directly affect the Q-function except through the data distribution. Hence, Haarnoja et al. (2017) motivates the actor network as an approximate sampler, rather than the actor in an actor-critic algorithm. Crucially, the convergence of this method hinges on how well this sampler approximates the true posterior.

Thank you for the detailed answer. If I'm understanding this correctly, is the main point that the sampled actions used for updates to the critic network have to come directly from the actor (on-policy) and not from a replay pool (off-policy) for it to be a true actor-critic method?

Not necessarily, as off-policy DDPG leverages DQN's replay buffer idea to uniformly sample minibatch experiences while being a actor-critic method (usually replay buffer only implies off policy not actor-critic). Here Haarnoja et al simply classifies true actor to be directly affect the Q-function like in DDPG via the usual n-step return advantage function, not via another gradient ascent of some performance metric for estimating Q-function in SQL indirectly. They also noted its MAP variant connection with DDPG in the same paper, thus it's not that essential clear cut for my point of view.

I see. I think that makes it clearer, thank you!


text:
Creating border around specific areas of images
I have some 1000+ image, containing data like this

Red area: Symbols
Grey area: Text describing the symbol
Note: I draw these red/grey boxes just for visualization only.
Each symbol is unique in this 1000+ images. I want to draw borders around those locations. Which algorithm/technique I need to use so that it can draw border around those places correctly. As these are unique symbols the only valid guess is these are starting from left side then some space then the text.

such segmentation technique is called Image Captioning; search in that direction


text:
Using root and modifiers in translation task
I am doing a project of translation task by finetuning T5 model. I am given sentences in Chinese, their translation to English, and for every English sentence I am also given its ROOT and some modifiers of the root. How can I use the ROOT and its modifiers in order to achieve better results?
I can't think about ideas for how to use this additional information.


text:
What reinforcement learning algorithm should I use for the following problem?
Environment
I have a static timeseries environment meaning the environment is the same.
This problem is a multi armed bandit problem.

Time
t0
t1
t2

State
s0
s1
s2

Score
10
0.1
0.2

Class
1
0
0

Keep work = 1
Discard work = 0.1
Class = (Score>9)
Rewards
Reward = $\frac{Score}{Work}$
I have various states in the dataset which are ordered chronologically and each state contains a reward from (0 -> inf)
rewards come in 2 types a very small reward then very sparsely a high reward is given.
To calculate the reward from a state is an expensive operation.
The model can discard states to increase the reward density
reward per work.
For example lets consider 2 cases\

model 0: do nothing
model 1: optimal

Model
Metric
t0
t1
t2

model 0
decision
keep
keep
keep

model 0
score
10
0.1
0.2

model 0
work
1
1
1

model 1
decision
keep
discard
discard

model 1
score
10
0
0

model 1
work
1
0.1
0.1

model 0: reward = $\frac{10+0.1+0.2}{1+1+1}$ = 3.43

model 1: reward = $\frac{10+0+0}{1+0.1+0.1}$ = 8.33

To avoid the model discarding most states to make work value low a penalty can be applied if the model discards too many states.
this is due to if keep/discard work is decreased reward will increase.
As $reward\propto\frac{1}{work}$
States
Each timestep state is related by being an input to a process and is monotonic
The state is equal to the process with the timestamp as the input
Sn = P(tn)
Domain Knowledge

P is unknown
high rewards are very sparse >1:10000

Attempts so far/Ideas
I have already tried using Bayesian optimisation on this problem with minimal luck.
there are many different types of reinforcement learning algorithms.
The ones applicable are

output is classification(category) for keep/discard
continuous state space -> discrete action space

stackoverflow

TD3
DDPG
SAC
PPO

output is decision boundary(value) for classification
continuous state space -> continuous action space

output is feature(value) which is then classified
continuous state space -> continuous action space

stackoverflow
CACLA paper

A2C
REINFORCE
CACLA

Questions

What reinforcement learning algorithm should I use?

given the problem is 1,2 or 3 most sensible?

any other considerations I should be aware of?

Thanks

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

If your problem is a multi-armed bandit then the algorithms that you listed are not suited for the job. Those algorithms are for situations where the agent's actions has an influence on the subsequent states encountered. You should look for solutions designed for multi-armed bandit scenarios such as LinUCB.


text:
Is the input embedding split along the embedding dimension so that every head of the multi-head-attention module just gets a part of the input data?
So I found two contradictory explanations of the MHA (multi-head-self-attention-module):
In the first approach, the input embedding (= the input matrix) is split along the embedding dimension and all heads are given a subset of the dimensions/features of each word.
Some websites supporting this theory:
https://medium.com/@smitasasindran/12-attention-mechanisms-multihead-attention-958041a35553
-> Quote: "The input has been split into multiple heads, and we are running the attention model separately on each of these heads."
https://towardsdatascience.com/how-to-code-the-transformer-in-pytorch-24db27c8f9ec#3fa3 -> Quote: "In multi-head attention we split the embedding vector into N heads, so they will then have the dimensions batch_size * N * seq_len * (d_model / N)."

The second approach assumes that all heads receive the entire input data, but different weight matrices are used for each head depending on the number of heads.
This theory is well explained on https://hungsblog.de/en/technology/learnings/visual-explanation-of-multi-head-attention/
-> Quote: "Each head is responsible to fully calculate the attention for the whole embedding, not just for a subset of it and creates h attention matrices"
I tend to the second explanation, but have not been able to find a satisfactory and contradiction-free answer so far.


text:
Fine-Tuning T5 with specific penalty
Currently I am finetuning transformers T5 model for translation task. As part of the dataset, I am given sentences in Japanese, their translation to English, and for every English sentence I am also given a few (English) words which need to be in the translated sentence.
Is there a way to modify the Seq2SeqTrainer and the loss being used in order to penalize the model (while training) for not including the given words in the translated sentence?

Actually I encountered this issue myself.
My solution was to wrap the model in my own class.
I then overrode the forward method. ( I was using pytorch lightning)
class WrapModel(pl.LightningModule):
    self.model = T5ForConditionalGeneration...

def forward(self, **inputs):
    outputs = self.model(**inputs)
    loss = new_loss_logic(outputs.specific_hidden_state)
    outputs.loss = loss

I am relatively new to the field, but I think this should work.
If you wish to stay within HF, you can always make the wrapping class inherit the T5Model, then the call to the underlying forward is self(**inputs) or can be down with super(), but the rest remains the same


text:
Are there public examples of AI models that predicted short-term price well?
This question is inspired by Is there any AI model that predicts short-term stock price well?.
As answered, game theoretically either nobody would reveal such a model or it would have been alread exploited. Still, it could be instructive to look at successfull past models, developed in the past and looking at data of that time. This model are expected not to perform well today, but they were profitable back then.
To be clear, developing a model at a given time and claiming success on historic data is not what I'm looking for here.
So, are there public examples of AI models that predicted short-term price well?

You can easily find dozens of tutorial examples with a quick google search of models that do exactly this.  I understand you are not interested in "historical data" and success using it, but are you willing to risk real money to find out if a model works well to effectively bet on a stock?

Are you sure that those model were market beating? I would be looking for more proof than "trust me, I made millions with this back then". Feel free to link the ones that satisfy this criteria. To your second question, no, I would not use those models at all, it is just to get insight into what worked back then and why it was market beating.

I would expect these models to be released by hedge funds or similar long after they were not profitable anymore. Or leaked somehow.

The short answer is, "No one has managed to create a model that predicts the future."

You don't need to predict the future, you need to extract correlations between datapoints before other parties figure them out too. Otherwise what would predicting in AI even mean?

@Rexcirus If it was this simple to beat the market, high-frequency trading companies would do it. In fact, they do do it. Which means they already take all the possible money using this method and you won't get any because yours is slower.

You can see when most people try to make an AI like this it usually ends up just predicting the next price to be the same as the current price

AFAIK, unfortunately no.
However, we can get a feel of these beasts from some clues:

They do machine learning. E.g. see this competition of Two Sigma on Kaggle, and this competition by Optiver. The point is they are willing to do much work just to predict 1 metric e.g. volatility.

They hear many signals. E.g. see Two Sigma's competition to predict with market news; there are a lot of potential signal sources such as Twitter. You can easily find lots of such attempts on the internet.

They know and act fast. E.g. in 2010, Morgan Stanley built a high-speed fiber optic cable connecting New York and Chicago, reducing the round-trip speed from 16 to 13 milliseconds. The project cost $300 million, which was considered well-worth.

On the other hand, I know a person with 20+ years experience in fintech sector. He said:

They make their own hardware to be fast: once they design a new algorithm/system, they would craft specialized hardware/chips to speed up computation. Commodity hardware are not fast enough.

Interestingly, he said they don't use neural network, not because of accuracy, but because it is not fast.

They watch their competitors' bids closely (obviously) and adjust theirs, just like playing poker.

All in all, we know that short-term trading is a zero-sum game - our gain is someone else's loss. With this in mind, even if we build a same system which predicts the same as these giants, we will still lose - by the time we get our prediction, these beasts have not only got the prediction, but also read your actions, placed the bid, took the money and quit as the price changed.
Sometimes we may get lucky and win, just like in a casino. However, the house always has the last laugh.


text:
What can GPT-4 do linguistics-wise?
I have no access to GPT-4, but I wonder whether it can do the following (where ChatGPT failed).

Make syntactic and morphological analysis of sentences in a language like Russian, marking cases, parts of speech and sentence, conjugations of verbs, etc. This is a standard school task, so I think this will be eventually possible (ChatGPT tried, but did it wrongly).

Reconstructing proto-forms of modern words based on known sound change laws.

Writing reconstructed texts in proto-languages based on the whole set of known grammar rules for those proto-languages, such as Proto-Indo-European (while of course highlighting arguable or unreliable choices). This to include stress and syllabification patterns, etc, things which while known to our science, are often omitted in recontructions because of the complexity.

Suggesting reconstructed sound correspondences, forms or patterns for proto-languages in automatic mode based on language's descendants. For instance, given vocabulary for Indo-Pacific languages to suggest most likely proto-forms for their proto-language.

So, here are some sentences GPT-4 generated in Russian (you can translate it with Google to English) and translated to PIE:

Я люблю лес. Он полон жизни и красоты. H₁éǵʰōm h₂lewh₂óm. H₂ésti pl̥h₁nós ǵʰéyewtis h₂ékʷeh₂s.
В лесу есть много деревьев. Они дают тень и кислород. H₂n̥h₁dʰgʷih₁és h₁estí h₂lewh₂óy. H₁óyti dʰeh₁núm h₃r̥h₃ḱwomkʷe.
Среди деревьев живут разные животные. Некоторые из них дружелюбны, а некоторые опасны. Médhyo dʰérwom bʰuh₂ónti pr̥h₃ḱwós bʰewgós. N̥sméy
gʷih₃wós ǵʰóstis, n̥sméy tréyes.
Я вижу зайца. Он бежит быстро по траве. Widhḗmi kʷeklosóm. H₁éǵheti spéudeti pér ǵr̥h₃nom.
Я слышу волка. Он выет на луну. Kléwō wl̥ḱwom. H₁ówyeti ná mḗnsm̥.

They have a lot of errors. But it is much better than ChatGPT who hardly could understand what Proto-Indo-European is.
One typical error is omitting diacritics, that is using k instead of ḱ and omitting superscript, that is kw instead of kʷ. I think, it is because of negligence in the training sources.
Anyway, I think that over not that long time AI would be able to generate grammatically correct text in PIE.


text:
Can humans surpass AI in game-playing?
In the realm of game-playing, such as in the cases of AlphaGo and Deep Blue, can humans ultimately surpass AI in skill? Despite the current dominance of machine learning, what factors may contribute to humans eventually catching up to, or even exceeding, the capabilities of AI game players?
For example, can AI-assisted training and a deeper understanding of how AI works provide effective approaches for humans to improve their game-playing abilities? What are the limitations of these methodologies and what other methods may be required?

Humans already do "surpass" AI in game-playing. AI-agents' difficulty at playing Montezuma's Revenge is an example. The question is usually the opposite: can AI surpass humans in game-playing skills? Hence the research dedicated to game-playing AI. The examples you provide are clear evidence that this is indeed possible, although for the moment not universal.
As for your second question, indeed there has already been a case where adversarial policies discovered by AI were used by a human to beat KataGo at the game of Go. See https://goattack.far.ai/human-evaluation. I'd recommend reading that paper and taking a dive into their related work if you're interested in this topic, there are references to other methods and limitations in the literature.
The rest of your question is asking for speculative answers about the future, which I have opinions on but will refrain from sharing here since that would be off-topic.


text:
Smaller, competitive LLMs that are not autoregressive transformers?
Large language models like GPT have been really successful lately. One downside is that they require a huge amount of resources to train, and still a lot of resources for inference, such that most models can't run inference on consumer hardware. Another problem is that they do not have an explicit world model, so that you can not be 100% sure if a response is factual or hallucinated.
Along the old neat-scruffy axis, I wonder if there are alternative architectures that are a bit more in the scruffy direction. For example, I could imagine that you use deep learning only for "language understanding", but for "world knowledge" you would have a fact database like a triplestore. The language model of such a system would be much smaller, only needing to reproduce basic grammar, and the facts would be explicit instead of implicit.
So are there any - potentially competitive - NLP models that are not of the autoregressive transformer type, scruffier, smaller, and yet operating in a similar space to GPT-3, LLaMA and co.?

It sounds like you're talking about KNN-LMs and Retrieval Augmented Generation.
KNN-LMs
These are models that, during inference time, look up the embedding representation of their context in a datastore. They use the retrieved next-token-prediction in addition to their regular output to make the final prediction.

The above figure is from the KNN-LM paper.
Retrieval Augmented Models
This is an extremely general class of models, which basically involve the retrieval of some information which gets added to the context before the model proceeds with regular autoregressive generation.
This retrieved information can be web pages (e.g., Bing Chat), documents from an existing database (see: open-domain question answering), or information from a knowledge graph. This paper does knowledge graph retrieval, then adds an encoded subgraph into the context.

External memory indexed from attention - https://arxiv.org/pdf/2112.04426.pdf
Regarding minimizing cost of autoregression, one could do encoder-decoder (eg T5) architecture, but with a twist: Make the decoder deliberately much smaller network than the encoder which is doing most of the heavy lifting, but running only once. Such asymmetry works on assumption that "speaking" a concept from the latent space vector is a simpler task compared to "understanding" it done in the encoder.
If this paper is correct, this allows you to run T5-FLAN at 6x inference speed at a cost of marginal (<5%) loss. The paper utilizes pruning (ie downsizing pre-trained model), better scaling might be possible if trained like this from scratch.


text:
Should softmax be in the model or in the loss function?
Suppose I have an image segmentation model with an output of [ 128, 128, 2 ], segmenting an input image into 2 parts.
Commonly, loss functions have the sigmoid or softmax needed to produce a probability distribution in the loss function itself, for example:
tf.nn.sigmoid_cross_entropy_with_logits
tf.nn.softmax_cross_entropy_with_logits

Alternatively, one could include the softmax or sigmoid call in the last layer fo a model:
model = tf.keras.Sequential([
    # ...
    tf.keras.layers.Conv2D(2, kernel_size=1, padding="same", activation="softmax")
])

My question is this: Why could you keep the softmax / sigmoid in the loss function instead of in the last layer of the model?
Wouldn't putting it in the last layer of the model ensure that the output of the model is constrained, enabling the model to more effectively learn the boundaries of the given input as smaller gradients would be needed for the model to change its mind on a given pixel?

Mathematically it does not matter at all. The results will be the same. However there is a strong reason to prefer it being in the loss function: numeric stability.
Because the loss function knows more information (ie what the loss is), it can compute the softmax and loss together using some tricks to improve numeric stability.

Thanks for the explanation. What are these tricks you mention here?

Here is a blog post explaining the basics. If the above answer was helpful please consider accepting it.
https://jaykmody.com/blog/stable-softmax/

Many thanks for that link, but I'm afraid I still don't understand. That log post mentioned log(softmax()) for some reason? It did not seem to cover my question

It did cover your question. Log softmax is more numerically stable. You can convert back to regular softmax after. Perhaps you do not have a sufficient mathematical background to understand the answer. But that link directly addresses your question. The trick is to use log softmax during the intermediate computations and then convert it back after. If this still does not make sense to you, make sure you have a good understanding of what numerical stability means for floating point arithmetic, and then the log softmax trick should be fairly easy to understand.

Right, I understand that, but it still doesn't explain why you wouldn't do the (log) softmax in advance in the last layer of the model.

You can. But then you cannot use cross entropy anymore since cross entropy is not expecting a log softmax. You would need to use a log loss instead. That is a fine alternative.


text:
Do Artificial Neural Network with non-linear activation only in the output layer follows linearity?
I am using a model with linear activation in the hidden layer and non-linear activation in the output layer. Could you please help to know whether such models exhibit linearity or not?
The non-linear activation is required to scale the output.

A non-linear layer is a matrix multiplication followed by an activation function. A linear layer is just a matrix multiplication.
You have a matrix multiplication, then another matrix multiplication, then an activation function.
Two matrix multiplications in series are equivalent to one matrix multiplication, so your network is equivalent to one non-linear layer.

You are speaking about generalized linear models (GLM)

Do the model GLM follow semi linearity? I am new to this. Could you please help to provide detail on how it works?

If you have a sigmoid activation in your output layer, and linearities before, you can interpret it as a logistic regression model for binary classification (if output dimension is one.)
Also, in practice, having multiple linear hidden layers is a waste of resources because you can define an equivalent linear layer whose weight matrix is the linear combination of those.
Moreover, intermediate linear layers can be useful to "adjust" (i.e. expand or decrease) the output dimension of the previous layer (including the input layer.)
In general, you want to have multiple non-linear layers (i.e. whatever layer with non-linear activation on top) in the middle of your model (i.e. as hidden layers) because the activation of the output layer is often (if not always) problem dependent: e.g., in regression it can be linear.
Having multiple non-linear functions in your neural-net allows it to model (learn) more complex functions, so exploiting the universal function approximation thing: indeed, assuming a correct architecture, loss function, and optimization procedure.


text:
Repainting a picture in the style of some painter (or of another picture)
It sounds like a straight-forward task for DALL-E (and GPT?) to present a painting and ask  to repaint it "in the style of Leonardo da Vinci". Like one can present texts and ask to rewrite them in the style of some author. Or even better: to present two paintings and ask to repaint the first in the style of the second. (You may replace "paint" by "draw".)
Can this already be achieved - and how? Or is it possibly in the pipeline?

This is quite a specific task and there is a solution: CycleGAN.
Of course, DALL-E, GPT-4 and similar systems can also do it, but if you are looking for the "how?", CycleGAN is specifically designed to solve this task and it is more st to understand.
How does CycleGAN work?
CycleGAN is specifically trained on two domains, e.g. Photos and Van Gogh Paintings. CycleGAN consists of 4 Neural Networks: One that transforms images of the first domain (e.g. photos) into images of the second domain (e.g. Van Gogh-Paintings). A second network does the backward transformation (e.g. Van Gogh --> Photo).
To ensure, that a transformed image still represents the same content, CycleGAN has Cyclic Objectiv that requires an image that is transformed forward and backward (e.g. Photos --> Van Gogh --> Photo) to be close to the original Photo. The same should also hold for backward-forward-transformations.
Two networks serve as discriminators that ensure that a transformed image look like an image of the other domain (e.g. a transformed photo looks like a Van Gogh).
For further readings:

The original CycleGAN Paper
Why is cycle consistency loss alone not sufficient to produce meaningful output?
There is also code availabe, so you can even try it on your own.

How Good is CyclicGAN:
These Examples are taken from the Paper:

Hi, thanks for the answer. Could you please include some explanation of how CycleGAN works within the answer itself? We generally expect all answers on Stack Exchange to be self-contained as much as possible, and saying "there's an explanation over there" is much less useful than giving an overview within the answer itself. Thank you!

Hi Mithical, of course, I can include some explaination, but I hesitate a bit to do so (and this is also the reason I did not do it in the first place): The question seems to be mainly about the abilities of AI Systems. Explaining a concrete Network Architecture would be on a much more technical level. To me this feel out of scope of this question. So "A network called CycleGAN" would (here) have been my answer to this question of "How". The links are ment as external references.
Having this said: I would appreciate your oppinion on this.

@Mithical: I edited the answer. I would appreciate some feedback from you if this meets your expectations.


text:
Layer Norm in a ResNet MLP
Where do you insert layer norm in a residual block?
After the addition or before the activation function (RELU in this case)?

Two things:

Layer Norm wasn't invented before ResNet. ResNet still uses the regular Batch Norm.
The model to use Layer Norm is residual block is ConvNeXt. Based on this line, it applies LayerNorm after the first Conv layer, before the activation layer.

I am aware of (1), but I should have added that I cannot use batch-norm because of a gradient penalty, which penalizes the gradient with respect
to each input independently, rendering batch-norm ineffective.
Regarding (2), I am not using any convolutional layers, as I don't work with image data. Do you know of any ResNet implemenatations with MLPs that use layer norm?

Usually you insert the normalization layer (be it BatchNorm, LayerNorm or whatever) after the convolutional layer and before the activation layer, i.e. Conv + Norm + ReLU.
The original ResNet applies the addition (skip connection) just before the last ReLU, but this design was revised in a follow-up paper by the same authors. You can also see here for figures and explanations.
Regarding using skip connections with fully-connected layers, I think you can take a look at the transformer architecture. The transformer encoder (and decoder) architecture uses skip connections around fully connected layers. It also uses LayerNorm instead of BatchNorm. However, with this architecture there are again debates where exactly to place the LayerNorm layer. This paper suggests using Pre-LayerNorm instead of Post-LayerNorm, because it stabilizes the training of the Transformer model. You can also see here for figures and explanations.


text:
Why "Exploratory moves do not result in any learning"?
In Chapter 1 of the book Reinformcement Learning An Introduction 2nd Edition by Richard S. Sutton and Andrew G.Barto, there is one statement "Exploratory moves do not result in any learning".
This sentence is in Figure 1.1.

Figure 1.1: A sequence of tic-tac-toe moves. The solid black lines represent the moves taken during a game; the dashed lines represent moves that we (our reinforcement learning player) considered but did not make. Our second move was an exploratory move, meaning that it was taken even though another sibling move, the one leading to e⇤, was ranked higher. Exploratory moves do not result in any learning, but each of our other moves does, causing updates as suggested by the red arrows in which estimated values are moved up the tree from later nodes to earlier nodes as detailed in the text.

It confuses me. In my understanding, exploration should contribute to learning in almost all RL algorithms. So, why does the book state "Exploratory moves do not result in any learning" in this case?

I believe this is a pedagogical decision. Because the sentence occurs in the first chapter of the book, I think the authors are trying to avoid the objection that a neophyte might make: learning from random movements seems like it will cause you to learn strange behaviors.
Certainly, the statement is inaccurate. We need only reach page 26 to see a counterexample: The Q-learning equation (2.1) sums over all actions, not just the greedy ones. This also applies to the temporal difference learning method specified in Chapter 6, which is the method Figure 1.1 is discussing.

I would say that the statement is inaccurate rather than incorrect. In off-policy learning with current greedy policy as target policy, exploratory actions are not backed up to cause value estimate updates (or if you prefer, the importance sampling weighting is zero), whilst greedy moves are. When I read the book, I took the statement in that sense.

@NeilSlater that's a good point. I'll update the answer.


text:
Generate Image with Artificial intelligence
I am pretty new to Artificial Intelligence programming, however i do understand the basic concept.
I have an idea in my mind:
Import a JPEG Image,
Convert this Image into a 2D Array (x,y values + r g b values).
Then create a second array with same (xy) values wit rgb all set to 0,0,0.
Now i want to build an AI Layer which will try to lower the error factor between the arrays until they are equal (the rgb values in the second array are equal to the first array (error factor 0) ).
I would prefer to do it in Java. Any suggestions to librarys or example that can help me get started? Thanks for any help.

To just copy image pixel values between arrays as you suggest you don't need any AI "layer". However, this kind of thing is strongly related to various image generating, classifying or compression work. It may help to explain where you want to take this. A very simply "toy" example might be for an AI to learn the function pixel(x,y) which given a co-ordinate in the image generates the right RGB colour array [r,g,b] - it's a simple, fun and informative task to use a neural network on, and you can animate the results. Would that sort of thing be what you had in mind?

Yes that sounds like a cool idea too. Do you have a framework or sdk or something in mind?

Not for Java. Someone else would need to suggest that. I wanted to clarify what kind of thing would count, since your description of your idea is a bit vague, and does not naturally lead to an AI solution. So it would be helpful to know your goals - e.g. are you learning about neural networks, or want to explore computational creativity? With this one idea it is only going to be a single small step towards any goal.

My idea is to recreate a given image by using ai, maybe you could add a twist like only lines are allowed etc. Im just messing around with ai because it seems really interesting to me.

That sounds more like a genetic algorthm approach e.g. https://www.youtube.com/watch?v=rGt3iMAJVT8 - I think you need to settle on a more specific goal than "just mess with AI" for this question (it's fine as a personal goal, but too broad for anyone to answer). This site cannot really give you the ideas for your project as answers because that is too open-ended. But we can answer more specific questions such as how to use neural networks or how to use genetic algorithms to solve a given problem.

For recreating an image exactly the same as the original, you can use an autoencoder. This basically use AI Layers to encode the image raw pixel values to a vector of floats, drastically decreasing the representing vector. Afterwards another AI Layer increases the dimensions back to the original image. The method does not required labels, as it only refer to teh image to encode it to a vector of features. For implementing in java, there is not a lot 9f resources. However, you can check this library out: https://deeplearning4j.org/
For the implementation, see this: https://github.com/eclipse/deeplearning4j-examples/blob/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/unsupervised/variational/VariationalAutoEncoderExample.java
For the method, you can see this python tutorial and implement it in java. https://towardsdatascience.com/autoencoders-in-keras-c1f57b9a2fd7
For generating completely new image, you can try GAN(Generative Adverserial Network). This generates completely new images from a random noise image. The noise image is passed through a generator which is a CNN(convolutional neural network) and get a result image. The result image is then feed to  a discriminator(CNN as well) to classify if that image is fake or real. The generator and discriminator compete and slowly gets better. For java implementation, see this: https://github.com/wmeddie/dl4j-gans
Hope I can help you and have a nice day!


text:
Which courses in computer science and logic are relevant to Machine Learning?
Although I have a decent background in math, I'm trying to understand which courses from CS and logic to look into. My aim is to get into a Machine Learning PhD program.

I worked as a professor for a time, and often advised students on this. For a PhD in machine learning, I think the ideal background is:

Core CS Courses

Programming (typically 3-4 courses). Language choice is not highly important, but Python, C++, Java, and perhaps JavaScript, are reasonable picks, if only because of their prevalence.
Core topics: data structures, algorithms, operating systems, databases
numerical linear algebra or numerical methods
advanced algorithm design and analysis

Together these will allow you to read and write the code that even highly optimized versions of ML algorithms are written in, and to understand what might be going wrong within them.
2. AI & ML courses, usually offered through a CS department

a broad survey course in AI (like AI:AMA by Russel & Norvig), usually offered to senior undergraduates.
A course applied machine learning or data mining.

You may also take other AI courses, but they are not as common to see offered to undergraduates, so many students wait until graduate school:

reinforcement learning
soft computing
computational learning theory
Bayesian Methods
Deep Learning
Multiagent Systems
Information Retrieval
Natural Language Processing
Computer Vision
Robotics

Together, these will give you the broadest possible background in AI & ML. These can allow you to find new applications of ML, or to pull AI techniques from one area into another as you need.
3. Statistics courses

a 1 or 2 term course in probability theory, ideally a version that requires and uses calculus.
at minimum a course in statistical hypothesis testing.

Much stronger would be to also take courses in:

regression
generalized linear models
experiment design
causal inference
Bayesian methods

These courses allow you to reason formally and comfortably about uncertainty. They also give you the correct framework for answering questions about whether your ML algorithm is working, and what patterns an ML algorithm uncovers mean.
4. Mathematics courses

3 semesters of calculus, going at least as far as multi-variate/vector calculus.
optionally, a more advanced course that builds on calculus, like real analysis, but only to reinforce calculus concepts.
at least 1, and preferably 2, courses in linear algebra
at least 1, and preferably 2, courses in discrete mathematics.
ideally, something like Knuth et al.'s Concrete Mathematics
ideally a course in advanced optimization techniques
optionally, courses in logic, but be aware that this is almost a fringe area in AI now, and essentially irrelevant to a PhD in machine learning. The parts you need are usually covered in a broad survey AI course.

These courses give you the basic mathematical fluency to understand most machine learning algorithms well.

I have found studying optimization techniques to be invaluable when trying to decide what technique to use in solving ml/ai problems

@nickw I agree. A survey course in AI, as well as a typical course in advance algorithm design will cover these topics pretty well, and probably well enough for someone to start on an ML PhD. That said, if they can do a mathematics course that is specifically about optimization, they'll be way ahead of the pack.

From the top of my mind roughly in order of priority excl. math:
Practical/applied CS: machine learning, artificial intelligence (incl. symbolic AI), data mining, algorithms, data structures
Theoretical CS: complexity theory
Programming: Python
Logic is highly relevant for symbolic AI but not so much for sub-symbolic approaches like ML.
For all topics mentioned in the first two categories you can find free online lectures from different universities.

In several projects, I found data analysis and data structures to be critical.  Machine Learning requires huge amounts of data and, most likely, the data will come from multiple sources. Prior to use, data  requires analysis, cleaning, interpretation, feature engineering (subject matter expertise), and structure.

Data analysis is not necessarily part of the core of computer science. It is more of a statistical sub-area.


text:
Why RNNs often use just one hidden layer?
Did I get it right, that RNNs most often have just one hidden neuron layer? Is there a reason for that? Will RNNs with several hidden layers in each cell work worse?
Thank you!!

Definitely you can have multiple hidden layers in RNN. One the most common approaches to determine the hidden units is to start with a very small network (one hidden unit) and apply the K-fold cross validation ( k over 30 will give very good accuracy) and estimate the average prediction risk. Then you will have to repeat the procedure for increasing growing networks, for example for 1 to 10 hidden units or more if needed.
However, in my experience, if you are interested to get best possible accuracy, you should start with small number of hidden layers and more simple structure, and if you are not satisfied with the corresponding accuracy, then we should go on increasing the learning rate by fixed but small small steps and each time start training fresh.