text:
Using deep reinforcement learning for malware detection; trained agent mostly performs the same action
I'm trying to implement this article:
Ransomware early detection using deep reinforcement learning on portable executable header
The article uses an unpublished dataset of benign and ransomware samples. I use ClaMP dataset instead which has a collection of benign and malware samples. The article proposes a deep reinforcement learning method for detection:

For each sample extracts 14 features from PE header. (For each sample I use 53 features from ClaMP which are also extracted from PE header)
For each sample a Boolean feature indicating the existence of a valid bitcoin address is computed. I do not use this feature since I use malware samples instead of ransomware samples.
Two average Manhattan distance features are computed. One is the average Manhattan distance of the current sample from previously correctly labeled ransomware samples. The other is the average Manhattan distance of the current sample from previously correctly labeled benign samples. I do the same.
The current state of the environment is composed of extracted features from the current sample. I do the same.
Agent action is either 1 to indicate classifying the current sample as benign or 0 to indicate classifying the current sample as ransomware. I use 1 is for malware and 0 is for benign. But this should not affect the results at all.
Reward is either -1 for misclassification or +1 for correct classification. I tried both -1,+1 and -100,+100.
According to the article a Deep Q Network (DQN) agent should achieve an accuracy of %96.

My trained agent mostly performs the same action which results in an accuracy of around 50%.
My implementation is available in this python notebook. The most important parts are:
class MalwareEnvironment(gymnasium.Env):
  def __init__(self, dataset):
    self.dataset = dataset.copy(deep=True)
    self.action_space = spaces.Discrete(2)
    self.observation_space = spaces.Box(low=0, high=2**63-1, shape=(55,), dtype=np.int64)
    self.state = self.dataset.iloc[0,:-1]
    self.state["Manhattan_TP"] = 0
    self.state["Manhattan_TN"] = 0
    self.label = self.dataset.iloc[0,-1]
    self.remaining_length = self.dataset.shape[0]
    self.time_step = 0
    self.TP_list = list()
    self.TN_list = list()
    self.positive_reward = 100
    self.negative_reward = -100

  def step(self, action):
    if self.label == action:
      reward = self.positive_reward
      if action == 1:
        self.TP_list.append(self.state[:-2])
      elif action == 0:
        self.TN_list.append(self.state[:-2])
    else:
      reward = self.negative_reward

    self.remaining_length -= 1
    if self.remaining_length == 0:
      done= True
    else:
      done= False

    if done == False:
      self.time_step += 1
      self.state = self.dataset.iloc[self.time_step,:-1]
      Manhattan_TP = Manhattan_distance(self.state, self.TP_list)
      if Manhattan_TP > 2**63-1:
        print("Manhattan overflow", Manhattan_TP)
        Manhattan_TP = 2**63-1
      Manhattan_TN = Manhattan_distance(self.state, self.TN_list)
      if Manhattan_TN > 2**63-1:
        print("Manhattan overflow", Manhattan_TN)
        Manhattan_TN = 2**63-1
      self.state["Manhattan_TP"] = Manhattan_TP
      self.state["Manhattan_TN"] = Manhattan_TN
      self.label = self.dataset.iloc[self.time_step,-1]

    info = {}
    return self.state.to_numpy(dtype=np.int64), reward, done, False, info

  def reset(self, seed=None, options=None):
    self.state = self.dataset.iloc[0,:-1]
    self.state["Manhattan_TP"] = 0
    self.state["Manhattan_TN"] = 0
    self.label = self.dataset.iloc[0,-1]
    self.remaining_length = self.dataset.shape[0]
    self.time_step = 0
    self.TP_list = list()
    self.TN_list = list()
    info = {}
    return self.state.to_numpy(dtype=np.int64), info

The above code defines the gymnasium environment.
train_env = MalwareEnvironment(train_dataset)
test_env = MalwareEnvironment(test_dataset)
check_env(train_env)
check_env(test_env)
model = DQN("MlpPolicy", train_env, verbose=1, train_freq=1,
            learning_starts=10, target_update_interval=1, buffer_size=32,
            batch_size=8, learning_rate=0.001, exploration_fraction=0.7,
            exploration_initial_eps=1.0, exploration_final_eps=0.1, gamma=0.99)
model.learn(total_timesteps=train_portion, log_interval=4)
#mean_reward, std_reward = evaluate_policy(model, model.get_env(), deterministic=True, n_eval_episodes=5)
#print(f"mean_reward:{mean_reward:.2f} +/- {std_reward:.2f}")

score = 0
obs, info = test_env.reset()
TP,TN,FP,FN=0,0,0,0
while True:
  action, _states = model.predict(obs, deterministic=True)
  print(action)
  obs, reward, terminated, truncated, info = test_env.step(action)
  #print(obs)
  score += reward
  if reward==test_env.positive_reward and action==1:
    TP+=1
  elif reward==test_env.positive_reward and action==0:
    TN+=1
  elif reward==test_env.negative_reward and action==1:
    FP+=1
  elif reward==test_env.negative_reward and action==0:
    FN+=1
  if terminated or truncated:
    break
accuracy = (TP + TN) / (TP + TN + FP + FN)
print(score)
print(accuracy)
print(TP, TN, FP, FN)

The above code performs training and testing of the DQN agent using stable-baselines3.
What am I doing wrong? Any help is appreciated.

I tried to find this article but I couldn't (only paid pdfs). What is the motivation behind using RL here? This looks like a simple supervised learning problem to me? Are the states in the dataset somehow connected, is there some order?

I agree with @pi-tau, as the action here does not affect the next state $s'$, it seems more sensible to phrase this as a classification task.  However, if you're set on reproducing this reinforcement learning work, I would suggest simplifying the state representation for debugging. For example, you can test whether your code works if you input the true label "malware" or "benign" as label.

@johannesack pi-tau Thanks for your comments. The action that we perform in the current state can potentially affect the Manhattan distance features calculated and incorporated into the next state.

@pi-tau According to the article: "To fully utilize the dynamic interaction characteristics of reinforcement learning, features based on Manhattan distance were constructed." As for motivation the article says: "deep reinforcement learning methods can adaptively learn and improve models to adapt to new and unknown ransomware threats ... In short, the deep reinforcement learning method has the advantages of high precision, self-adaptation, and real-time response in ransomware detection"

Also the article say "Properly prioritizing instances that are similar to these detected ransomware instances will improve performance. Additionally, the detected benign instances are equally important. Discouraging instances that are similar to detected benign instances can reduce the false positive rate." I have not implemented this prioritization which I believe is according to Manhattan distance.

Okay, I see. So the current action does have some inpact on the next state because of the Manhattan distance features. But still there is no particular order in which you must iterate over the states. Maybe you should shuffle the dataset before evey rollout? Is this mentioned in the paper?

@soosan123 Also, I see no reason why your Q-network should learn the value of the next state. Note that DQN uses the following backup equation $Q(s_t, a_t) = r_{t+1} +  \gamma max_a Q(s_{t+1}, a)$, but it seems to me that you should evaluate the current action based only on the current reward. So maybe you can try setting $\gamma = 0$. I see that you are using $0.99$, is this what they use in the paper?

Finally, you could consider regularizing the training to prevent convergence to suboptimal solutions. I would use L2 regularization on the weights of the neural network. Sorry, but all of my suggestions are toward modifying the training to look more like supervised learning. I still cannot see any reason to use RL here.


text:
Does splitting a model across multiple GPUs (using model parallelism) reduce model accuracy?
When training a model in PyTorch on multiple GPUs, it should ideally give the same results as using just one GPU. However, are there any specific things to be careful about during implementation?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to decide on size of state space?
I'm in a reinforcement learning class and we're working on a dice game problem where you profit the number rolled for certain digits and lose all your money on certain other digits.  You can choose to roll or quit and take your current winnings at any step.
The main question that I'm struggling with is how big exactly to make my state space.  I've set it to represent the profit ($0, $1, $2, $3, et cetera).  However, I don't exactly understand how to size my state space.  N = 100, N = 1000?  N= 30?  We haven't gotten much guidance on the question and it's gotten me hamstrung.
Another thing I'm stumped on is how exactly I'm generating my transition matrix.  Let's say that 4 out of 6 digits on the dice result in a reward, and the other 2 result in me losing all my winnings back down to zero.  Am I incorrect in assuming that the probability of increasing my bankroll in any state is 66.6667%, and 33.3334% for losing it all?  If I have a state space of N = 100, my probability matrix is 100 x 100?  I'm incredibly confused about transitions.  I'm also lost about how to factor in rolling or quitting into the equation once I'm actually iterating over values.

you cannot assume we know what's the setting/problem with as explanation "you profit the number rolled for certain digits and lose all your money on certain other digits", please provide an actual accurate explanation of the problem

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
MultiLayer Perceptron not working for regression problem, what could I try?
I am trying to learn the inverse kinematics of a robotic manipulator. To do that I have a simulator with which I acquired data.
My dataset is composed of positions in X, Y and Z and actuator variables (6 of them). That represents around 310 000 rows. Therefore, the inputs of my NN are the positions (3 inputs) and the outputs the actuator variables (6 outputs).
Just before (when my dataset were much simpler, 3 outputs only) I managed to obtain good results with a MLP. However, now that the dataset is much more complicated, I did not manage to obtain a similar result.
Being a beginner in deep learning, my question would be, what else could I try to obtain a better results than the one using MLP ?
First here is the code I used for the simpler case where I achieve a good result :
    import tensorflow as tf
    from sklearn.model_selection import train_test_split
    from sklearn.preprocessing import StandardScaler, MinMaxScaler
    from joblib import dump, load
    import matplotlib.pyplot as plt

    data = np.genfromtxt('dataprocess.csv', delimiter=',')

    scaler = MinMaxScaler()
    data[1:,[4,5,6,7,8,9]] = scaler.fit_transform(data[1:,[4,5,6,7,8,9]])

    trainset, testset = train_test_split(data[1:, [4,5,6,7,8,9]], test_size=0.2, random_state=0)

    X_train = trainset[:,[3, 4, 5]]
    X_test = testset[:,[3, 4, 5]]

    Y_train = trainset[:,[0, 1, 2]]
    Y_test = testset[:,[0, 1, 2]]
    print("Y train min", Y_train.min())
    print("Y train max", Y_train.max())

    input_layer = tf.keras.layers.Input(shape=(3,))  # Entrée : x, y, z
    hidden_layer1 = tf.keras.layers.Dense(512, activation='relu', kernel_regularizer='l1_l2')(input_layer)  # Couche intermédiaire
    hidden_layer2 = tf.keras.layers.Dense(512, activation='relu', kernel_regularizer='l1_l2')(hidden_layer1)  # Couche intermédiaire
    hidden_layer3 = tf.keras.layers.Dense(512, activation='relu', kernel_regularizer='l1_l2')(hidden_layer2)  # Couche intermédiaire
    hidden_layer4 = tf.keras.layers.Dense(512, activation='relu', kernel_regularizer='l1_l2')(hidden_layer3)  # Couche intermédiaire
    output_layer = tf.keras.layers.Dense(3)(hidden_layer4)  # Sortie : variables d'actionnement

    model = tf.keras.Model(inputs=input_layer, outputs=output_layer)

    model.compile(optimizer='adam', loss='mse', metrics=['accuracy'])

    History = model.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=150, batch_size=512, verbose=2, shuffle = True)
    print("history loss ", History.history['loss'])

    loss, accuracy = model.evaluate(X_test, Y_test, verbose=0)
    print("Test accuracy:", accuracy)
    print("Test loss:", loss)

    model.save_weights('modele_poids.h5')

    plt.figure(figsize=(12, 4))
    plt.subplot(1, 2, 1)
    plt.plot(History.history['loss'], linestyle='-', label='train loss')
    plt.plot(History.history['val_loss'], linestyle='-', label='test loss')
    plt.legend()
    plt.subplot(1, 2, 2)
    plt.plot(History.history['accuracy'], label='train score')
    plt.plot(History.history['val_accuracy'], label='test score')
    plt.legend()
    plt.show()

That gave my good results when I have 3 outputs but now that I have 6 of them it is no longer the case. Even if I add more neurons or more layers.
Moreover, here is what my dataset looks like (I removed some unused columns, that is normal if the index doens't match the dataset).

a big enough (your case falls in here) MLP is very likely powerful enough for your dataset, because at most it can learn by heart your data, so probably you messed up something (possibly in the data processing or the training)

Hi! You can better ask these kinds of questions on the data science stack exchange


text:
Understand Memory Usage of Pytorch Tensors for Inference
I plan on translating large text corpora from various languages to english with Large Language Models. Therefore, I tried messing around a bit to see the computational limits of my machine.
Specifically, I try to translate a text corpus by using the No Language Left Behind translation model of Meta with the following code:
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
device = "cpu"
tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M").to(device)

def translationPipeline(text):
    input_ids = tokenizer.encode(text, return_tensors="pt").to(device)
    outputs = model.generate(input_ids,forced_bos_token_id=tokenizer.lang_code_to_id["deu_Latn"])
    decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return decoded

# c = df_text.iloc[1,:].body.apply(translationPipeline)
with torch.no_grad():
    c = translationPipeline(df_text.iloc[0,:].body)

Here, df_text.iloc[0,:].body is one text corpus that equates to roughly 150,000 characters. The tokenizer transforms this to a Tensor of shape (1, 42809). If I execute the above code, I get a out-of-memory error on my 256 GB RAM machine.
I am aware that the token length of my text corpus exceeds the maximum length for this model. However, I am still wondering where this high RAM usage comes from. In its raw form, 150,000 characters should of course not consume 250 GB of memory.
Where does the additional RAM required come from? How much RAM would be needed to translate the whole text corpus at once?

This question is off-topic here. This is a typical systems programming question, so I think it should be asked on Stack Overflow. Having said that, there are tools that can help you with debugging memory issues. We focus on the theoretical aspects of AI.


text:
Training Diffusion Probabilistic Models
My question is in regards to the training process for Diffusion Probabilistic Models by Sohl-Dickenstein et. al. and also Ho et. al. and this blog post

For the derivation of the model log-likelihood lower bound $K$, my question is why do the authors replace $q(\mathbf{x}^{(t)} | \mathbf{x}^{(t - 1)})$ with $q(\mathbf{x}^{(t-1)} | \mathbf{x}^{(t)}, \mathbf{x}^{(0)})$.  Is there not already an analytic expression for $q(\mathbf{x}^{(t)} | \mathbf{x}^{(t - 1)})$ from the forward diffusion process? Specifically, when the authors go from Equation (46) to Equation (48) they have that

$$K = \sum_{t = 2}^{T} \int \mathrm{d}\mathbf{x}^{0, \ldots, T} q \left( x^{0, \ldots, T}\right) \log \left[ \frac{p(\mathbf{x}^{(t - 1)}|\mathbf{x}^{(t)})}{q(\mathbf{x}^{(t)} | \mathbf{x}^{(t - 1)})}\right] - \text{Other terms}.$$
The authors then use Bayes’ Rule and condition on $\mathbf{x}^{(0)}$ to get the expression in terms of $q(\mathbf{x}^{(t-1)} | \mathbf{x}^{(t)}, \mathbf{x}^{(0)})$
$$K = \sum_{t = 2}^{T} \int \mathrm{d}\mathbf{x}^{0, \ldots, T} q \left( x^{0, \ldots, T}\right) \log \left[ \frac{p(\mathbf{x}^{(t - 1)}|\mathbf{x}^{(t)})}{q(\mathbf{x}^{(t-1)} | \mathbf{x}^{(t)}, \mathbf{x}^{(0)})}\right] - \textrm{More terms}
$$
A similar technique is used in Ho et. al. in going from Equation (18) to Equation (20) and in the blog post in the Reverse Diffusion Process section

My second question is why the forward diffusion process is even needed during training?  I get that the authors are maximizing the model log likelihood and they derive a lower bound that depends on both $q(\cdot)$ and $p(\cdot)$

$$L = \int \mathrm{d}\mathbf{x}^{(0)} q \left( \mathbf{x}^{(0)} \right) \log p \left( \mathbf{x}^{(0)}\right)$$
Can we instead maximize based only on the joint probability $p(\cdot)$ as is done with a Markov chain?  Specifically, we can maximize
$$\hat{L} = \prod_{t= 1}^{T} p\left( \mathrm{x}^{(t - 1)} | \mathrm{x}^{(t)}\right)$$
where $p\left( \mathrm{x}^{(t - 1)} | \mathrm{x}^{(t)}\right) \sim \mathcal{N}(\mathbf{\mu_\theta}, \mathbf{\Sigma_\theta})$.  Is it that
a) Maximizing $\hat{L}$ solely based on $p(\cdot)$ is also ok but since we have additional information about the forward process in terms of $q(\cdot)$, incorporating this information will help with training
b) It is better to maximize the marginal probability $p \left( \mathbf{x}^{(0)}\right)$ in $L$ instead of the joint probability $\prod_{t= 1}^{T} p\left( \mathrm{x}^{(t - 1)} | \mathrm{x}^{(t)}\right)$ in $\hat{L}$


text:
Machine learning network design for 2D point prediction labeling
I'm looking to develop a basic neural network for labeling a detected set of 2D points. I've generated a random set of 3D points within a certain radius, all of which lie on a flat plane (Y value is 0). My aim is to train the network using different perspectives of this 3D point set, and subsequently input 'n' 2D points, predicting each point's label along with a confidence score.

Using a default camera matrix and some predefined camera positions I have created a pytorch dataloader. This will generate n number of training data samples.

Which neural network architecture should I explore to achieve this?

at first glance, I would say that transformers would be the best fit, just to be able to handle any number of points, and treating them as a set


text:
Challenges in Developing AI Algorithms for X-ray Image Analysis on Large Datasets
Hello everyone,

I'm currently working on a research project involving X-ray imaging
and the development of AI algorithms to detect diseases from large
representative datasets. I wanted to reach out to the community to
discuss some of the challenges I've encountered and see if anyone has
insights or suggestions.

I've been experimenting with various deep learning architectures,
such as convolutional neural networks (CNNs), for    disease
detection. While they've shown promise, I'm curious about any
recent breakthroughs or improvements in network architectures
specifically designed for medical image analysis.
I am currently looking for some interesting researchable areas of study for a PhD in AI for medical imaging.
I appreciate any input, experiences, or advice you can share.

it's not clear what's your current understanding of the architectures, what topic of CV you are interested, if segmentation, explainability, classification or what, your question is way to vague

I performed classification and I also want to do explainability.


text:
Can we derive the support vector machines dual formulation without directly using lagrangian duality theory?
Lagrangian duality theory allows us to derive the dual formulation for support vector machines and to show that the primal and the dual solutions are equivalent.
My question is: is it possible to derive the dual formulation by a different, simpler approach, such as variable substitution?
Consider the hard-margin problem alone, for simplicity. Could one start by assuming that the optimal hyperplane normal vector can be written as a linear combination of the data vectors (a sound and geometrically justifiable assumption), with the dual variables playing the role of coefficients or weights, and use the fact that non-support vectors don't contribute to the solution, to derive the dual formulation without explicitly applying the lagrangian machinery?
Update:
I finally figured out how to achieve such a derivation, by following a hint I found in a paper by Keerthi et al., "A fast iterative nearest point algorithm for support vector machine classifier design," in IEEE Transactions on Neural Networks, vol. 11, no. 1, pp. 124-136, Jan. 2000. In the following, I will sketch a brief outline of the derivation.
In the hard-margin version of SVM binary classification, the problem of finding a separating hyperplane with maximum margin turns out to be equivalent of finding the minimum distance between the convex hulls of the two classes of observations.
The convex hull $H_V$ of a set of vectors $V = \{\mathbf{x}_1, \mathbf{x}_2, ..., \mathbf{x}_n\}$ is the set of all weighted averages of the vectors $\mathbf{x}_k \in V$:
$$ H_V = \left{\mathbf{x} = \sum_k \beta_k \mathbf{x}_k \quad|\quad \mathbf{x}_k \in V, \beta_k \in \mathbb{R}, \beta_k \geq 0, \sum_k \beta_k = 1\right} $$


text:
RNN - Time To Stop extimation given non-categorical events sequence
I have a dataset which contains sequences of event of the following type:

Timestamp_s
Timestamp_e
Event_type

I've preprocessed the data to create sequences of the shape [seq_size, unique_codes_count], where each element describes the active events within a specific timeframe (1 for active, 0 for inactive). Note that events are not exclusive, so many events may be active at the same time (even 0), so this isn't a one-hot encoding; it's more like a "many-hot" encoding.
Additionally, I've computed an additional list of [seq_size] sequences, representing the time left until the end of the sequence.
The objective of the neural net is then to predict how much time until the next stop.
My initial approach was to use a simple LSTM model that takes the event codes as input and produces a prediction of the time to the next sequence end. To handle time prediction, I normalized the time to stop using a log function in order to:

Reduce the numbers to predict (since I calculate the time to stop in seconds)
give higher penalties the closer we are to zero, e.g. [0 day - 1 day] is to be penalized more than [10 days - 11 days].

For training, I used a Mean Squared Error (MSE) loss function, AdamW optimizer with a learning rate of 1e-4, weight decay of 1e-2, and gradient clipping with a norm threshold of 0.5.
No matter the number of layers, hidden size, activations and dropout I tried, every network seems to just settle on always predicting the average time left of all sequences, with occasional spikes in either the up or down direction when some events happen.
I'm asking for suggestions on improving my model architecture and data normalization techniques. Any insights or recommendations, papers or other questions on the exchange would be greatly appreciated.
Thank you all!
P.S. I've refrained from posting code since this is primarily a network design question. However, if sharing code snippets would help, please let me know. Also, I'm currently working with PyTorch 2.


text:
Pixel-wise regression only focus on edge
I am trying to use unet to learn pixel-wise regression from one image to one groundtruth with the same image size. The network seems to focus too much on the edge of the image, and it does not learn much on the inside. Loss is only calculated inside the valid region. The loss is in a good looking decreasing shape. The examples of groundtruth and prediction are shown below. Does anyone have any recommendation on the issue?

Groundtruth image

Prediction image

Epoch loss

Could you show the input image? Also share the architecture?

show the images as rgb/b&w not as 3d model

What loss function are you using?


text:
ROC curve for multiclassification - results sound not correct
I'm working on a multiclassification task using LSTM algorithm, i generated my roc curve plots but they give scores like 1 , 0.99, 0.97 however i have an accuracy of 0.97, Precision 0.65, Sensitivity/Recall 0.62, and
Specificity 0.62. is it normal ? here's my roc curve:

here's my code to generate the ROC curve
from sklearn.metrics import roc_curve, auc
import matplotlib.pyplot as plt
# Predict probabilities for each class
Y_pred_prob = model.predict(X_test_reshaped)

# Calculate ROC curve for each class
fpr = dict()
tpr = dict()
roc_auc = dict()

n_classes = Y_train_onehot.shape[1]  # Number of classes
class_names = {0: 'Benign traffic', 1: 'Brute force', 2: 'DDoS', 3: 'DoS',
               4: 'Mirai', 5: 'Reconnaissance', 6: 'Spoofing', 7: 'Web-based'}

for i in range(n_classes):
    fpr[i], tpr[i], _ = roc_curve(Y_test_onehot[:, i], Y_pred_prob[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])

# Plot ROC curves for each class with labels
plt.figure(figsize=(8, 6))

for i in range(n_classes):
    plt.plot(fpr[i], tpr[i], label=f' {class_names[i]} (AUC = {roc_auc[i]:.2f})')

plt.plot([0, 1], [0, 1], 'k--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve for Multiclass Classification')
plt.legend(loc="lower right")
plt.show()

Depends on the # of examples per class. Make sure X_test_reshaped and Y_test_onehot are lined up. Play with threshold to see how sens/spec changes.


text:
Optimizing Stop Loss Percentage for a Specific Model Based on Stock Price to Maximize Expected Value
I'm fine-tuning a specific trading model, and a crucial parameter I'm keen on optimizing is the stop loss percentage. The primary objective is to maximize the Expected Value (EV), formulated as:
$$EV = \text{probability of winning} \times \text{average gain %} - \text{probability of loss} \times \text{average loss %}$$
My dataset comprises a comprehensive list of trading signals from this model spanning a long period, which I utilize to compute the EV.
A unique challenge I've encountered is the dynamic nature of the optimal stop loss percentage, which appears to vary depending on the current stock price. A few observations include:

At a stock price of 1 dollar, the optimal stop loss might be 5%.
At 5 dollars, it might be 10%.
And at 5.5 dollars, it could be 7%.

I've been exploring Bayesian Optimization as a potential solution, yet I'm uncertain about its efficacy given the inherent dynamic nature of the optimal parameter. Another consideration has been using a neural network to predict the optimal stop loss percentage in relation to the stock price.
Given the constraints of my specific model and the aim to optimize dynamically based on stock price, are there methods or frameworks tailored for such dynamic parameter optimization, especially when the primary goal is maximizing EV?
EDIT
A signal could be as simple as {"symbol": "ABIX", "date": "2005-08-04", "signal": "SELL", "reason": "Overextended gap-down detected."}
EDIT
$$EV = EV(\text{parameter_1}, \text{parameter_2}, \text{stop_loss_%}, \dots)$$
and
there exits a function $f$ such that $f(\text{entry_price}) = \text{stop_loss_%}$. Be aware that the $\text{stop_loss_%}$ correspond to $\text{stop_loss_price} = \text{entry_price} * (1 + \text{stop_loss_%})$

Please, consider sharing data if not at least column label of it. It would help us to understand better. Regarding output varying by initial condition, stock price, I have seen Support Vector Machine with Kernel Trick used, https://towardsdatascience.com/the-kernel-trick-c98cdbcaeb3f.

@CloudCho I modified the question. Maybe you could build a full answer I could understand your point

Thanks for the sample data, but I don't see "current stock price" in your data. Does your model have to guess that as well? What is the output of your model between stop loss percentage and Expected Value?

@CloudCho From there, I just pull the minute OLHC data, calculate the entry point, exit point and stop loss point and compute the expected value for the model.

Thanks for explanation, so the "stock price" isn't one of input variables in you model?

@CloudCho I modified the question so you can understand a bit better.


text:
Are we missing correlation to achieve a artificial intelligence comparable to humans?
I don't think neural networks are enough to copy human intelligence because in the end a machine only using a neural network to decide on something still is unable to recognize patterns.
So I was wondering how do machines recognize patterns? With correlation! So what if we have two paths:

Path of a simple neural network
A path of just correlation.

And in the end which output will be given as the result is decided by the same neural network which output a value?

I really struggle to follow what you're saying...

Suppose we have a neural network modelled as a block(A).And a correlation function as a block(B).We give the input to both of these blocks and we get 2 outputs right?Now that we have 2 outputs we let A decide on which output from A or B we will output.

Please, share possible usage(s) of your suggested Two Path method. It would help us understand better.

@Cloud Cho should I post a diagram?I dont know how I came up with it tbh I like neural networks and neuroscience as a hobby but still havent been tought it at my university and I was studying about correlation in telecommunications for a exam and I dont know.

It sounds like that the concept of Correlation in signal brought your question. Diagram, theory, real life problem, possible problem in the future, and etc. Any possible usage, which you think, would help us.

I was just trying to understand how my brain works and can a):recognize patterns and b):learn through experience.I realized that neural networks cannot give the same pattern recognition with the capabilities of a human so I first thought :lets just add correlation which is pattern recognition essentially and see where it gets me.I realized that these processes must work in parallel but if you simply run 2 algorithms  at the same time with the same input and then choose 1 of the 2 outputs based on another algorith which doesnt change ,....

......then 1 will always prevail over the another and there wont be any room for "evolution".And we humans can understand when to discard a pattern (from correction by others) and when to apply its logic to solve a problem (from rewarding).

It is not clear what your question is. If you want someone to critique your idea or reference similar research, then right now the idea - or perhaps the presentation in the question - seems very vague and incomplete. Potentitally there are related topics - e.g. "Neurons that fire together, wire together" is a well known principle from neurobiology and there are ANNs that use that idea for training - https://neurosciencenews.com/wire-fire-neurons-19835/ - however that is only one way that some kind of self-correlation can occur, and doesn't seem to match your diagram

@NeilSlater It is a network with feedback.I am asking if this has been tried and if it is a effective way to copy the functions of a human brain.

Please make your request that you are looking for whether the idea has been tried in the question title. That would help a little, but not enough yet. Th eissue is that there are plenty of neural network architectures and training regimes using feedback. It is far from clear from your question and diagram what kind of feedback you are asking about and how it would relate to the neural network. You may be better off sitting on the idea for a while and waiting until you know more about the subject, so that you can refine the question to be something other people can understand

@NeilSlater well obviously the backpropagation function IS feedback but this is feedback coming from the 2nd path of the evaluation of the final value.

The main problem for me (and I suspect most readers) is that I have no idea what you are trying to say with statements like "the 2nd path of the evaluation of the final value". There might be something worth answering in that, but you are using vague statements that are not standard for neural networks or AI, and it is not possible to understand what you are saying. Are you able to express your idea mathematically, e.g. to describe the process that occurs during training in your model?


text:
Finding a value at which TD(lambda) and Monte Carlo Are Similar
Let's say that I'm trying to find a value of lambda for which running a TD(lambda) method will produce the same results as a Monte Carlo method (within a small margin or error, of course).  One of the ways that I've thought about doing this would be to calculate the Monte Carlo value (which is equivalent to TD(1)), and then assign lambda a small value and run TD(lambda) in a loop until the return converges with the MC value, adjusting lambda in small amounts until reached.  This seems computationally heavy though (imagine hundreds of iterations of calculated TD(lambda) until we reach the convergence value of lambda.
Is there a less computationally heavy way of doing this?

Interesting subject...I found Self Learning Monte Carlo paper and it looks like a few computer code in GitHub like https://github.com/sylvialee12/SelfLearning-MonteCarlo,  https://github.com/misawann/self-learningMC Also one research paper at https://dl.acm.org/doi/pdf/10.1145/162754.162763

Hi, could you please provide the formula for TD($\lambda$) and maybe Monte-Carlo so that there is no confusion.

I am confused, given that $\lambda=1$ is a solution to your question, what exactly are we looking for? Other $\lambda$s with the same property or something else?


text:
Query modification for search using AI
I have a problem statement that I'm struggling to formulate as a machine learning framework. There is a huge client database of documents - we're trying to come up with an efficient way of querying those documents.
The core problem is: let's say I want to find a document with certain attributes (these could include author, doc title, date published, subject matter, etc.) - I need to generate an appropriate query using these attributes such that the document with those attribute values is returned as one of the top search results.
A couple of issues: I don't have direct access to the doc database for obvious reasons - only whatever docs are returned via search. Plus the attribute values that the searcher is using to generate search query may have slight differences compared to what's there on the doc (e.g. date on doc may be "16 Sep 2023" and the searcher may use "sep 16 23").
Once I have a document from a search result, I can calculate its score judging how close it is to the intended search. Alternatively I can have a label saying whether the document is what the user intended to search or not.
I have a vague framework in mind -

There will be a query generator which just determines what preprocessing to do on the various document attributes and how to combine them (e.g. it'll make a decision on how the query is formed - is it just the document title? or just the author + doc title? etc.)
This query generator will not work on single doc search requests - instead it will work on a batch of several doc search requests at a time. For each request, search results will be returned
Top k search results will be taken for each doc search request and the aggregate quality of all search results for the whole batch of search requests can be calculate from individual scores/labels
This "aggregate quality score" can then be used as feedback to modify the rule set of the query generator

The issue is - I'm not sure exactly what's the best way to frame this as an ML problem. Should it be a deep learning framework? Or should it be an agent-action-reward based reinforcement learning problem? Or should it be an explore-exploit paradigm RL problem?
I'd really appreciate your opinions on this!

isn't this a normal document retrieval problem? look up on google "vector databases "

@Alberto: Thanks! I went through a link describing the vector database paradigm, but there's an issue. It requires creating embeddings for the content that we want to retrieve using search queries. I don't have access to the content, only the search results retrieved via queries. And I can't do bulk retrieval of content to create the vector database. So vector database is a good solution for queries retrieving content in my own organization. But not for retrieving content from my client's organization.


text:
Reinforcement learning: enviroment changes a lot while DQN is still computing actions
I'm implementing a agent to dodge skillshots. Here's the problem: while DQN is computing action based on observation, enviroment changed meanwhile. Since those skillshots are quite fast, the enviroment are quite different from observation for DQN. Sometimes when computing finished, my agent has already been hit.
My question is: do I have to do extra work to deal with these changes? Or model itself can adapt to it?

Welcome to AI Stack Exchange. Is the agent also running DQN training on minibatches, making it slower because it is training its neural network? If yes, then is the delay more acceptable when the agent is only having to calculate its next action and does not need to train?

@NeilSlater Thanks for your answer. I checked the source code from stable baseline, in which training phase are sepreate from prediction. Besides, prediction only spends around 0.05s and still makes the issue above. I'm quite curious about if there are a topic about this kind of problem.

The time between the start observation and environment state in your diagrams is 0.05 or less? So the agent has to start moving to avoid being hit in less time than that? Are there prior observations that may give some clue on how the trick short will progress before the fast changing phase?

usually in training you wait for the env update for the agent to act, however if when deployed you also have this problem, consider developing a lighter NN, or to give a look at Real Time Reinforcement Learning (RTRL)


text:
Model gets stuck at an almost static error
I am trying to solve MNIST. If I limit the training samples to just 0 and 1 everything works flawlessly. With the whole training model it gets stuck at a certain loss and all the predictions are just zeroes. My network structure is a convolutional layer with 32 5x5 filters and two fully connected layers from 32x24x24 to 100 to 10. Weights are initialized in a uniform distribution between -0.5 to 0.5. I train with mini-batch GD, 30 batch size, 20 epochs, 0.005 learning rate. Any help?

What is you loss function and last activation in the network?

Loss is MSE and last activation is a Sigmoid. I suspect the reason of these results would be a vanishing gradient, as it is a large network and using leakyReLU causes an exploding gradient, but I need confirmation, I am still learning.

You're trying to classify the digits, right? If so, you should use categorical_crossentropy (or sparse_categorical_crossentropy if not doing one-hot encoding of labels) as loss function, and softmax as the final activation. The rest seems fine. If this changes would solve your issue I can provide a more detailed answer, just let me know how it goes

Thank you, I did some research on my own and learned a lot, indeed those were issues, and now I also understand why (not going in depth, softmax gives me a probability distribution, and MSE is  not fit for classifiers). Thank you for confirming, I am currently working on implementing softmax and cross entropy as I am using a self-made ML framework.


text:
YoloV5 Calibration
I'm currently tackling the task of calibrating object detection models specifically tailored for medical imaging applications. Despite extensive online research, I've encountered difficulties in finding comprehensive information and resources that cater to the unique demands of this domain. While I have identified several relevant papers, a significant number of them lack the crucial details needed to reproduce their reported results.
To provide some context, I am working on a project aimed at enhancing the accuracy and reliability of detecting anomalies in medical images. This necessitates a well-calibrated model to ensure precise predictions. If anyone has experience or expertise in calibrating detection models, particularly within the context of medical imaging, your insights and recommendations for more comprehensive resources would be greatly appreciated.


text:
Does an RNN architecture exist where the output are actually samples drawn from a (non-parametric) probability distribution?
Does an RNN architecture exist where my input is a 10 dimensional real-valued vector and the output are 200 samples drawn from a probability distribution.
In essence, the RNN is actually learning a probability distribution but gives the output as multiple (200) samples drawn from this distribution.

Could you explain why you might want this? It would seem easy to have the RNN output the learned distribution and then sample from it outside of the NN (which is e.g. how translation models based on LSTMs worked - although nowadays they are all transformers, the principle is the same). What benefit are you hoping to realise by having the RNN do the sampling via the NN architecture?

Let me try to provide more context: I have a model that gives me the distribution from t = 0 to ,say, t = T at an interval of 1 minute.  I also have a time series signal (10-dimensional) that is sampled every second. I want to predict the distribution using my time-series signal,  by training it on the time-series and distribution.
I chose an RNN with the time-series as input and the distribution as output. I am unsure how to use distribution as an output during training (drawing samples from this distribution and somehow using them as an output is what I was initially thinking).

Is your source distribution model inaccurate? I don't otherwise understand why you would take a full distribution ground truth and also want to use a sample?

I am making an assumption that the source distribution it is accurate. In the meanwhile efforts are underway to validate this assumption on a different data set. The source distribution I am using is a normalizing flow model.

If the distribution model that you are learning from is accurate ground truth, then why do you want a sample from it as well? A correct distribution model effectively contains the data from a large number of samples directly, in a single function call. I am not saying that you don't need to do this, or that your goal is wrong, but I am having trouble understanding from the question and your comments why you are doing this, and what you are hoping to achieve. If I cannot understand that then I don't think I can answer the question for you.

Am I right in guessing that your source distribution model is very expensive to calculate? So you want to create something approximate but faster? Otherwise I guess you would just take samples from the existing distribution model, and would not need to train any neural network.

The source distribution is not expensive. Once trained, it is actually quite cheap to draw samples.  But, the caveat is, the samples used to train the distribution generating model is obtained after a very long time interval. However, the time-series signals are obtained in real time.  So, we want to use the time series to understand the probability distribution at any given time. The source distribution simply serves as a surrogate data generator for the time series to learn some systems dynamics.
P.S.: I spent summer 2017 at Durham University and stayed by the cathedral. I loved it there.

Sorry I am missing something and don't understand your need/use-case. Maybe someone else can understand and help. There are several possible architectures that can sort of sample, and could be added to RNN-based approach. None of them learn to sample as a function, something has to be inserted into or extend the neural network. However, I am not sure if any of these would actually help you, and they might be hard to implement. PS I moved to Durham in 2015, and would go past the palace green (with castle and cathedral) a lot back then - so maybe we have even passed by one time!

Thank you for your attention Neil.


text:
How to get GPT to complete all tasks before ending its turn?
I'm trying to figure out how ChatGPT and Code Interpreter (or anything else) gets GPT to complete an arbitrary number of actions before it ends its turn and hands it back to the user. I'm using a while loop to have it perform multiple actions but I can't figure out a way to end its turn when it has completed all tasks. I tried making an end_turn() button but it doesn't adhere to using it.
def get_some_value():
    some_value = "one plus one equals two."
    return json.dumps({"some_value_one": some_value})

def get_some_value_two():
    some_value = "two times two equals four."
    return json.dumps({"some_value_two": some_value})

def end_turn():
    return json.dumps({"message": "All tasks are completed."})

# Initialize the assistant and its state
# messages = [
#     {"role": "system", "content": "You can call the function get_some_value, get_some_value_two, and call the end_turn function when your tasks are complete to hand the conversation back to the user."},
#     {"role": "user",
#      "content": "activate the get_some_value function, then activate the get_some_value_two function, then tell me the value of both. "}
# ]
messages = [
    {"role": "system", "content": "You must quietly activate the end_turn function to give control back to the user after you are done with your current response"},
    {"role": "user",
     "content": "hi"}
]

# Describe the functions
functions = [
    {
        "name": "get_some_value",
        "description": "This function returns some value",
        "parameters": {"type": "object", "properties": {}}
    },
    {
        "name": "get_some_value_two",
        "description": "This function returns some value",
        "parameters": {"type": "object", "properties": {}}
    },
    {
        "name": "end_turn",
        "description": "This function ends the assistant's turn so the user can respond",
        "parameters": {"type": "object", "properties": {}}
    }
]

while True:
    # Make the API call
    response = openai.ChatCompletion.create(
        model="gpt-4-0613",
        messages=messages,
        functions=functions,
        function_call="auto"
    )
    # print(response)
    function_name = response["choices"][0]["message"].get("function_call", {}).get("name")

    if function_name:
        # Call the function based on its name
        if function_name == "get_some_value":
            function_response = get_some_value()
        elif function_name == "get_some_value_two":
            function_response = get_some_value_two()
        elif function_name == "end_turn":
            function_response = end_turn()
            print("Assistant says: All tasks are completed.")
            break

        messages.append({"role": "function", "name": function_name, "content": function_response})
    else:
        assistant_message = response["choices"][0]["message"]["content"]
        print("Assistant says:", assistant_message)


text:
Policy Gradient in Partial Observability
Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where $J=\mathbb{E}_{\tau\sim\pi_{\theta}}[R(\tau)]$.
My professor said something along the lines that this equation is also true in the case of partial observability. Can someone please explain how is this the case?
In particular, I need to prove $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|o_i)R(\tau)]$, where $R(\tau)=\Sigma_{t=1}^T r(s_t,a_t)$.


text:
Algorithm Suggestion for Diverging Data (Severity/Intensity Analysis)
I have four datasets for four different accidents; each dataset has the same parameters.  Some of the key parameters are changing their values from a "standard value". The more they change these key parameter values, the more the intensity increases. I want to compare the intensity of four accidents. I don't understand what algorithm to use. Please suggest any ML, DL, soft computing, or evolutionary algorithm.
I am already working on ANFIS (Adaptive Neuro Fuzzy Inference System), Please suggest more.


text:
Is my 1D signal using CNN & RNN regression reasonable?
I want to know if my impact-echo signals are proper with CNN or RNN regression model.
I got some simulated signal, as following shows.
In previous research, people mostly consider frequency or even time-frequency analysis. For deeper crack the signal response lower frequency .
Singal input are their 1D signal, and want output present their depths. But my train simulated crack are all create integer (e.g. 8, 10, 12, 14, 16, 18, 20, 22, 24, 26), so I'm wondering if these signals in time domain using CNN like this or RNN like this model do regression task is reasonable. (Is my RNN model wrong or BIG concept misleading)
And I hope the model can learn (e.g. train 10cm and 12cm depth crack signal, when the model input is a 11cm signal it can be predict as 11).
Also if it is reasonable, which paper should I read or some paper related to enhance knowledge.
If the question is not clearly describe, please told me (I'm not native in English). Thank you guys.

I see you have many points (X axis) in each depth crack. Do you need to put entire signal to neural network or could you cut to small period like 1024 points and feed to neural network?

@CloudCho Oh you are right!
In frequency domain analysis, we need to take a longer time to achieve higher frequency resolution. However, on the flip side, a longer time duration can make the signal more susceptible to interference from boundaries or other reflections. This is  one of the advantages that I think 1D signals can get good performance. Currently, I'm using 448 data points (μs) for each input signal.
I'm wondering what I'm doing (make CNN or RNN doing these regression) is reasonable.

448 point in X axis would be doable in CNN. By the way, how human detect the depth of crack visually from the graph?

On-site, I think people use FFT to check frequency peak (at high frequency). And decide if  there have a crack under impact point.

Then you may consider to use FFT plot instead of time series as your dataset.


text:
Unclear points on feature extraction for a set of scenarios
The following is an example from a book (An Introduction to Pattern
Recognition and Machine Learning by P. Fieguth, page 85) on feature extraction and selection.
Please consider the following figure.

I have the following questions:
Q1- Why in Scenario IV will the applicable distance metric in 3D be "very non-Euclidean"? The points in Scenario IV are also in a 3D space for which an Euclidian distance metric is valid in my opinion.
Q2-Why will feature projection to 2D be linear for Scenario 1?
Q3- Why is there only one meaningful feature direction for Scenario 1?

Please, one question per post.


text:
Independent parameter update in backpropagation
When we calculate the gradient wrt to each paramters, we consider the other parameters remain constant, but the moment their is a change in any of the other parameters, shouldn't all the other changes become invalid?
Edit: In the usual gradient descent we assume that all the weights and biases for all the layers to be the input parameters for the loss function which we try to minimize. In that, we inherently assume that the inputs are independent on each other, which in case of a multilayer NN not true, the params of a layer should be influenced by its previous layers, which is why the param updates should atleast be done on a layer by layer bases, i.e. although inefficient, the correct method should be, if we make a nudge in the params of a layer what would be the best param value for the next layer.
Am I missing, or misinterpreting something, how is gradient descent able to take into account the interdependence of the parameters of a layer on its own since we cannot manually define how the parameters across layers are dependent on each other?

https://ai.stackexchange.com/a/20315/69468 this should answer your question

Does this answer your question? Why do we update all layers simultaneously while training a neural network?

@nbro not really, have edited the question. See if that clears the doubt I have

@Inprogress... Can you please put your specific question in the title? "Independent parameter update in backpropagation" is not really a question. Thanks.


text:
High Accuracy ML.NET Image differentiation model
I have a relatively big dataset (100+GB) that has 35 categories. All of them are microscopic images with slight differences. Although ML.NET documentation itself declares that training time should be much higher, in all 3 times of my training it didn't last for more than an hour. the resulting accuracy number is 0.8 but when I test it, it usually gets 2 out of 10 images right. I would like to know the possible ways to improve my practical accuracy and get more images to their respective categories.


text:
Why are convolutions and pooling described as layers in a network?
Whenever I look at resources on convolutions and max-pooling in CNNs they always seem to describe these algorithms as being part of the network - a preliminary set of layers before the main computation. It just seems strange considering they have no learnable parameters and only need to be applied to a dataset once (at least in my understanding that's the case). Surely it is vastly more inefficient to run these algorithms every time a piece of input data comes up - especially if you are training multiple models or there is potential for input samples to come up in batches multiple times...
Am I missing something?

Can you specify what are the "algorithms" you refer to? Is it max-pooling or also the convolution operations? What is the "main computation"? Can you cite some resources? By the way, max-pooling is often used between convolutional layers, therefore it is part of the computational graph. There's no way to compute max-pooling without knowing the feature maps of the previous parameterized convolutional layer.

Sure yeah, when I say "algorithms" I am referring to the combination of max-pooling and convolutions in the way you describe - an alternating series of computing the convolution of input data with a given kernel, followed by a max-pooling algorithm - and by main computations I mean the feed through process of the dense layers with learning hyper-parameters.

Here is an IBM document which outlines the basics: https://www.ibm.com/topics/convolutional-neural-networks. They refer to convolutional & pooling layers, but why can't these be statically applied to the dataset beforehand - rather than being included in the network so that they are applied each feedthrough - it seems unnecessary since for a given input they will always compute the same output.

Convolutional layers do have learnable parameters/weights, the kernel and biases, so your claim here is incorrect.

Ah I see - that makes a lot more sense. Thank you!

I’m voting to close this question because it seems it was just based on a wrong assumption, which is now clarified in the comments.


text:
Approaches for multi-label classification with over 1,000,000 labels
I have billions of rows in some dataset and each row can be in any subset of about 1 million binary labels. So the number of overall classes would be $\sim 2^{1,000,000}$, if I were to think about it as a simple classification problem, which is obviously out of question due to feasibility. I am given the wonderful task to develop a model to assign new data into these 1m labels...
The first obvious thing to do is to train 1 model per label but then each model needs to be trained on each observation, and likewise, at the inference time, each model needs to run for each new observation. One improvement I can think of to make this solution somewhat more efficient is to use a hierarchical model structure so that not each of the 1m. models needs to see each observation:

10 binary classifiers decide whether a data point belongs to 10 super-groups of classes.
100 binary classifiers decide whether a data point belongs to 100 groups of classes, given that it belongs to the super-group that includes it.
1,000,000 binary classifiers decide whether a data point belongs to any of these binary classes, given that it belongs to the higher level of the label hierarchy.

(Labels have some inherit hierarchy but it only has 2 levels as it is right now.)
So my idea is to build this so that a single neural network (in PyTorch) does all the work. Is there any recommendations for how to build this model and make my life easier?
What are good/feasible alternatives to the hierarchical approach I described?

Would you explain "binary label" and "binary class"? Do you mean your label is like 10011100?

Yes, each observation's label would be a sequence of length 1 million that consists of 1's and 0's, which is why I don't think it would be feasible to consider it as a simple classification problem.

Regarding subgroup, do you already know, which one in subgroup or you expect neural network could separate for you?

I already have a hierarchy on the labels so I don't need the NN to cluster them or anything like that. They are divided into 10 groups and 100 subgroups.


text:
What is the policy model in RLHF for LLMs?
What is the policy model doing explicitly in an LLM with RLHF setup?
From my understanding, LLMs generate in a way that is no different from any of their predecessors: beam search decoding, potentially with some sampling.
Does this mean the policy model is specifying how to generate the next token?


text:
Seeking Advice on Local AI Model for Analyzing Personal Files
I'm embarking on a project to develop an application for personal use that can assist me in retrieving specific information from my files. I'd like to be able to ask questions like "What was my income last year?" and get accurate answers directly from my stored documents.
One of my primary concerns is privacy; I want to keep my data as secure as possible. Therefore, I am interested in utilizing a local AI model that doesn't rely on cloud-based services. As I am relatively new to the world of AI, I'm looking for suggestions on which models would be most suitable for my needs.
After some preliminary research, I've come across Langchain, which seems to fit my requirements well. However, I'm open to other suggestions too.
A potential complication is that my files are not in any specific format or structure; they range from PDFs and text files to PowerPoints and Excel spreadsheets. I can segregate them into separate folders if that would be helpful. My question is, would restructuring my files into a specific format improve the accuracy of the AI model giving me proper answer?
I'd appreciate any advice on the following:

What AI model would be best suited for my project?
Is Langchain a good choice for this type of task?
Would reformatting my files increase the model's accuracy?

My current hardware:

RAM: DDR5 64GB
GPU : 3090
CPU: I9 11TH

Thank you for your time and expertise!

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.


text:
Confusion about the code for choosing "stumps" in Adaboost algorithm
(I actually asked the following question on Stack Overflow and Cross Validated Exchange for more than a month:
https://stackoverflow.com/questions/76842431/confusion-about-the-code-for-choosing-stumps-in-adaboost-algorithm
https://stats.stackexchange.com/questions/623300/confusion-about-the-code-for-choosing-stumps-in-adaboost-algorithm
but, still, there has been no response so far.)
This question refers to the following step in the classical procedure of Adaboost classification.
For each boosting round $b$, we define
$$ c_b = \text{argmin}_{c \in \mathcal{S}} \Bigg[ \frac{\sum_{i=1}^n w^{(b)}_i \mathbf{1}_{\big\{ c(x_i) \neq y_i \big\}}}{\sum_{i=1}^n w^{(b)}_i }\Bigg],$$
where $W:= \big[\{ w^{(b)}_i \}_{i=1}^n \big]$ is the array of weights corresponding to each boosting round $b$ and $\mathcal{S}$ is the set of stumps (decision trees of depth 1 corresponding to one feature).
Suppose that we assign an array $W$ and generate training points $\mathbf{x}$ and $\mathbf{y}$ (with $\mathbf{y}$ only taking values -1, 1) as follows:
W = [0.05, 0.032732683535398856, 0.05, 0.05, 0.032732683535398856,
0.05, 0.05, 0.05, 0.032732683535398856, 0.05,
0.05, 0.05, 0.05, 0.05, 0.05,
0.05, 0.05, 0.032732683535398856, 0.032732683535398856, 0.032732683535398856]

from sklearn.datasets import make_blobs
x,y = make_blobs(n_samples = 20, n_features = 5, centers = 2, cluster_std = 20.0, random_state = 100)
y[y==0] = -1

Then my textbook uses the following code A to generate $c_b$.
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(max_depth=1)
clf.fit(x, y, sample_weight = W)  # Here clf is the weak classifier c_b.
training_pred = clf.predict(x)
print(training_pred)

However, the following code B based on the definition of $c_b$ gives a different result:
 import numpy as np
 from sklearn.tree import DecisionTreeClassifier

 error_rate = 100000

 for k in range(5):

        clf = DecisionTreeClassifier(max_depth=1)
        clf.fit(x[:,[k]], y)

        local_training_pred = clf.predict(x[:,[k]])

        local_error_rate = 0

        for i in range(len(x)):
            if (local_training_pred[i] != y[i]):
                local_error_rate += (W[i])/np.sum(W)

        if local_error_rate < error_rate:
            error_rate = local_error_rate
            training_pred = local_training_pred

 print(training_pred)

Here the code compares the error rate of each stump; selects the one with the lowest error rate and then computes the prediction of the training set $\mathbf{x}$ under that stump.
Nonetheless, Codes A and B do not return the same result for our choice of $W$. Does anyone know the reason behind this? Have I actually mistaken the definition of stumps?


text:
What reinforcement learning method tries to learn the model? Is it better?
So I haven't read through every reinforcement learning algorithm, so I am curious. The model-free methods I have read about so far basically just try things and see what works and what doesn't. They then value each thing based on how good they think it is. Then they just choose the things with highest value. This is basically like pressing random buttons in like Pac-man. Then seeing what orders of button works the best given where they are and where the opponents are. Then they just do that. They aren't learning that staying away from the ghosts is what you are supposed to do and pressing buttons that keep their distance far away from the ghosts. I guess they both eventually reach the same result, but I think learning the method is faster. I think this is more important in like the real world. For example race car driving, the ones that I have read would just turn in the direction the corner is turning based on a bunch of stuff. But a person understands their speed and understand what happens from their speed and the outcomes of decisions without having to do them. This allows them to transfer their learning easier and be able to learn things step by step more efficiently.
So is there a reinforcement learning method that learns the method first then starts evaluating stuff and getting actions? And is this method better than current model-free methods that don't try to learn the model?
Edit: I thought about it a bit. Do humans learn how the world works by doing stuff and seeing what it does to their current "state"? So that is kind of what the method is doing. But this doesn't hold up for the pac-man example. A human playing even without knowing the rules can quickly recognize that being touched by the ghost is bad and alter their decision making by simply making sure they are far away from the ghost. However, in my mind I picture that the agent would just go through lots and lots of iterations and decide how to proceed based on a bunch of different exact parameters. However there are thousands maybe even millions of combinations of observations to make a single state. A human can quickly place this newfound model and know how to avoid the ghosts anywhere, and through experience they can quickly fix this function. Humans can also generalize a function and just slightly alter it for another situation.

Does this answer your question? What's the difference between model-free and model-based reinforcement learning?


text:
Requesting resources on causal networks for 2D strategy game
I am requesting research, articles, abstracts or interesting opinions that will help me create a complex causal neural network. There are many detailed resources on causal discovery, image recognition, spatial invariance. However, none of them take into account an animated changing game state, and all are very specific and hard to use for complex problems. Here is the core of the problem:

I am making a 2D strategy simulation/game and would like to create a neural network that can learn and deduce all the game's rules. The game, similar to Conway's Game of Life, takes the form of a 2-dimensional image, with red green and blue values for each pixel/cell. The game goes in turns, where every turn, the pixels of the image are evaluated, and changed based on the rules dictated by their RGB values. The network will train on a large data set of animated images, representing the changes to the game's state, with each frame being a turn.
RULES PER PIXEL

if red is above 50%, all adjacent pixels have their blue values
halved.
if blue is below 75%, between 2-8 random pixels within a 10 tile
radius will increase their green values by 25%
if green value is divisible by 5, all adjacent pixels will turn
white (100%red, 100% blue, 100% green)
if pixel is black, 15% of the map's pixels will be turned white at
random locations.
if pixel is white, a random pixel on the image will have their RGB
values randomized.
if green value is above 80%, the first 3 pixels to the immediate north will all have their values switched with each other.

The rules are very arbitrary and feature a mix of conditional checks as well as random changes to the map. This is intended to challenge the network's spatial and causal recognition. There is also the intention of adding or removing rules in the future, with the network being flexible enough to learn the new rules or changes to old rules, if found.
The main constraint I placed is that there will be no feature engineering, and the game's rules will always be hidden from the network. The challenge is for this theoretical network to analyze short and long term changes to the game state, discover and deduce as much of the game's hidden rules as possible, in the time it's given to analyze the data. It will eventually need to design its own generated image, designed in order to maximize the lifetime of an arbitrary choice of color. For example, it may be tasked to keep as much of the image red as possible, or possibly keep the green values low. The idea is that giving it new design constraints should be intuitive and easy, without having to re-code or re-train the entire network every time. I would also like to avoid using pooling, sampling, or any other tricks for image-based neural networks to detect patterns. This is because some of the game's rules are arbitrarily random, and may not have direct spatial correlation, while other rules do.

great project, now what about how to implement it? where are the implementation details? how do you encode the input if you don't want images?

Welcome to AI Stack Exchange. In general, we don't do "requests for resources" - even if we did, I think your setup is so unique, especially with your self-imposed constraints, that there will be little to no papers, tutorials or code that could apply to the project in general. I suggest you ask a more specific question about your project setup or approach that could have an objective answer, and that is not "please find me some helpful resources".

thank you for the replies. i guess the difficulty lies in connecting all the resources i've found into a functional network. i'm going to do more personal research in the future so that i can narrow down my question into something simpler and more possible to answer.
applying machine learning to the realm of video game design should be easy in theory, but in practice there's so many major differences between game states and linear statistical data that makes it hard to translate.

how do you encode the input if you don't want images?

that's actually what i'm trying to figure out. i will be using images, but it will be several frames of the image changing according to the rules. i'm lost on how to feed sequential turn-based information into my network in a way that makes it feasible to break it into quantifiable, linear chunks.


text:
In $n$ dimensions, why do we need at least $n+1$ sample points to have covariance of sample data non-singular?
In $n$  dimensions, why do we need at least  $n+1$  sample points to have covariance of sample data non-singular?


text:
Deep Reinforcement Learning that takes action from two different sets
I am working on a problem where I want to schedule multiple activities (a1, a2, a3, ... aN) requiring different resource types (r1, r2, r3, ... rK), in order to get a minimum lifespan. Now, each resource type rk has urk independent units available for work, each having unique characteristics. I want to solve this problem using deep reinforcement learning but I am not clear how to create output layer such that the action signifies both which activity to pick - a2, along with which resource units - [r11, r14, r23, r25, r31] (considering a2 requires r1, r2, and r3 resource types. r11, and r14 being selected indicates their overall performance score is better than r12, and r13)

You have tagged this question with dqn. It is going to be very hard, maybe impossible, to deal with the flexibility of actions in DQN. Could you please clarify whether you want to fit this to a DQN agent, and give some estimate of the action space. For example if there are four main actions and five independently chosen resources allowed for each action, your action space would be 4 x 32 or 128. That number is still ok in DQN, but much higher would be a problem

No, I don't necessarily need to fit the model with DQN agent. I also started with the same thought to use DQN agent but later realized the same point you mentioned. So my thought went in a direction something like I could have 2 outputs : activity to choose as an integeter, and resource units selected as a boolean array [r11, r12, r13, r21, r22, r23, ... rkurk]


text:
What causes train loss to suddenly rise, and never fall back, starting at some epoch?
Please see the following train/val losses of some basic vision binary classifier training I am performing.

I thought train loss should always go down.
When dropping the learning rate, this still happens, but at a latter epoch.
I also recall this happening on entirely different data and architecture a while back, but I couldn't explain it back then as well.
This isn't overfitting, as both train and val loss drop together up to some epoch ("iteration" in the graphs), and then BOTH suddenly rise, not only val loss.

Does this observed phenomenon have a name?
What could cause it?
Is there a standard solution?
How to debug?

Is always at iteration (epoch) 100? Have you checked your dataset if label is good? Also if you share loss functions (it looks like train and validation loss function differ), it will help us to guess.

learning rate schedule, exploding gradient, ill-formation of the optimization

Your validation loss appears to be three orders of magnitude greater than training loss. Is that correct on the graph?


text:
Cooperation between pattern recognition and neural networks for better design of AI
I continue to try to understand how a human brain works and how we can use it to set up AI which would copy the behavior of a actual person. A few days ago I uploaded a question where I described a process of function of the brain of such a AI. However I have changed my mind on some things.What if let's say we have a correlation function and a neural network. My new idea is that first the correlation function will look for patterns between the input and some stored data or memories or logic. Then if the correlation coefficient is greater than some number let's say 0.7 then the neural network will process those data and do what a neural network does. This will ensure that we would have a dynamic pattern recognition process and that the AI could still learn from new experiences, although it would be able to understand the relevance of a input with a understood pattern(memory or logic). Is there such a mode in Machine Learning?

Hi. This question is a bit better than last, but I think it is still hard to join the dots well enough to figure out what you are asking, and what a useful answer could look like. I don't want to discourage you though, if you have a sincere interest in the subject. I think we could try to work on refining the question so that it still explores at least a part of your idea, and works somehow on the site. I'll have a think about that and comment further when I have more time

I think we probably need to open a chat room, but as a start, let's look at your first statement. "I continue to try to understand how a human brain works" - could you explain, briefly what you are doing to increase your understanding? What material are you watching/reading that is relevant to understanding how the human brain works, and which parts of that material have inspired this question?


text:
VAE for Motion Sequence Generation - Convergence Issue with Scheduled Sampling
I have implemented a Variational Autoencoder (VAE) in PyTorch for motion sequence generation using human pose data (joint angles and angular velocities in radians) from the CMU dataset. The VAE architecture consists of an encoder and a decoder, each with two layers, comprised of a Conv1D layer and an ReLu activation for each layer.

During training, I input a sequence of 121 poses (60 prev pose + current pose (p(n)) + 60 next pose in the dataset) and the VAE generates the next pose (p_hat(n+1)).
I also have tried with normalized joint angles and angular velocities but it worsens the convergence.
Here's an overview of my training process:
Loss Function:
Initially trained for 30 epochs using Mean Squared Error (MSE) loss by comparing the generated next pose with ground truth data from the CMU dataset.
loss = MSE(p(n+1), p_hat(n+1))
From epoch 31 to 60, I added the KL divergence to the loss function.
loss = MSE(p(n+1), p_hat(n+1)) + KL
Scheduled Sampling:
Starting from epoch 61, I applied scheduled sampling, gradually increasing the probability p from 0.0 to 1.0 over 20 epochs (epoch 61 to 80).
From epoch 81 onwards, p is set to 1, implying that the generated next pose is fed into the model as the current pose to generate the next pose.
The length of scheduled sampling is 8 (I autoregressively create next 8 poses, inputting the generated pose of the VAE)
The Issue:
The network converges nicely on the MSE loss, a bit slower on MSE+KL, but it fails to converge when scheduled sampling is applied.
My Questions:
Is there a potential reason why the model doesn't converge during the scheduled sampling phase?
Are there any adjustments or insights regarding the VAE structure or training parameters that could help resolve this issue and improve convergence during scheduled sampling?
VAE Structure and Parameters:
Encoder and Decoder: Each with two layers (Conv1D + ReLu activation)
Loss: MSE initially, then MSE+KL
Scheduled Sampling: Gradual increase of sampling probability p from 0.0 to 1.0 over epochs 61 to 80, then p set to 1 from epoch 81.

class Encoder(nn.Module):
    def __init__(self, latentDim, inputFeatDim, frameSequence, intermediate_channels):
        super(VariationalEncoder, self).__init__()

        #intermediate_channels = 256
        # layer 1
        self.convLayer1 = nn.Conv1d(in_channels = inputFeatDim,
                                    out_channels = intermediate_channels,
                                    kernel_size = 1,
                                    padding = 0,
                                    padding_mode = 'zeros',
                                    bias = True)

        # layer 2
        self.convLayer2 = nn.Conv1d(in_channels = intermediate_channels + inputFeatDim,
                                    out_channels = intermediate_channels,
                                    kernel_size = 1,
                                    padding = 0,
                                    padding_mode = 'zeros',
                                    bias = True)

        self.downSamepleLayer = nn.Linear(in_features= frameSequence, out_features=1, bias=True)

        self.muLayer = nn.Conv1d(in_channels=intermediate_channels, out_channels=latentDim, kernel_size=1, padding=0, padding_mode='reflect')
        self.logVarLayer = nn.Conv1d(in_channels=intermediate_channels, out_channels=latentDim, kernel_size=1, padding=0, padding_mode='reflect')

        self.normalDist = torch.distributions.Normal(0, 1)
        self.normalDist.loc = self.normalDist.loc.cuda()
        self.normalDist.scale = self.normalDist.scale.cuda()
        self.kullbackLeibler = 0
        self.latent = torch.zeros(1).cuda()

        #self.print_f = True

    def forward(self, x):
        input = x
        x = self.convLayer1(x)
        l1_output = x
        x = torch.relu(x)

        x = self.convLayer2(torch.cat((input, x),dim=1))
        x = torch.relu(x)

        x = self.downSamepleLayer(x)

        mu = self.muLayer(x) # input here must be(latentDim)
        logVar= self.logVarLayer(x)

        self.latent = mu + torch.exp(0.5 * logVar)*self.normalDist.sample(mu.shape)
        self.kullbackLeibler = ((torch.exp(logVar) + mu**2)/2 - 0.5 * logVar - 0.5).sum()/(logVar.size()[0]) # logVar size ----> [batch_size * latentDim * 1]
        return self.latent, self.kullbackLeibler

class Decoder(nn.Module):
    def __init__(self, latentDim, inputFeatDim, poseFeatDim, frameSequence, intermediate_channels):
        super(Decoder, self).__init__()
        self.LatentExpander = nn.Linear(in_features=latentDim, out_features=poseFeatDim)

        # entry layer
        entry_in_channels = latentDim + poseFeatDim
        self.entryLayer = nn.Conv1d(in_channels = entry_in_channels,
                                    out_channels = intermediate_channels,
                                    kernel_size = 1,
                                    padding = 0,
                                    padding_mode = 'zeros',
                                    bias = True)

        # hidden layer 1
        self.convLayer1 = nn.Conv1d(in_channels = intermediate_channels+entry_in_channels,
                                    out_channels = intermediate_channels,
                                    kernel_size = 1,
                                    padding = 0,
                                    padding_mode = 'zeros',
                                    bias = True)

    def forward(self, latent, cur_pose):

        cur_pose = cur_pose.unsqueeze(2)

        x = torch.cat([latent, cur_pose], dim = 1)
        input = x

        x = self.entryLayer(x)
        x = torch.relu(x)

        x = self.convLayer1(torch.cat((input, x), dim=1))
        x = torch.relu(x)

        x = self.finalLayer(x)
        return x

class VAE(nn.Module):
    def __init__(self, Encoder, Decoder):
        super(VAE, self).__init__()
        self.encoder = Encoder
        self.decoder = Decoder

    def forward(self, seq, cur_pose):
        latent, kullbackLeibler = self.encoder(seq)

        X_hat = self.decoder(latent, cur_pose)
        return X_hat, latent, kullbackLeibler

def train(VAE, data, device, optimizer, load_saved_model, epochs_before_KL, draw_pose, poseFeatDim, scheduled_sampling_length, epochs, lr, lr_init, lr_final, latentDim, frameSequence, first_train_stage_epochs, second_train_stage_epochs):
    N = int((frameSequence-1)/2) # pose sequences before and after current pose

    if load_saved_model==True:
        alpha = 1.0
    else:
        alpha = 0.0

    for epoch in range(epochs):
        epoch_KL = 0
        if epoch == epochs_before_KL:
            alpha = 1.0
        epoch_loss= 0

        if epoch > (first_train_stage_epochs+second_train_stage_epochs-1):
            lr = lr_init - (lr_init-lr_final)*(epoch-(first_train_stage_epochs+second_train_stage_epochs))/(epochs-(first_train_stage_epochs+second_train_stage_epochs))
        optimizer.lr = lr
        for X, target, cur_frame_idx in data:

            X = torch.permute(X, (0, 2,1)).type(torch.FloatTensor).to(device=device)

            X_hat = (X[:,:,N].clone()).unsqueeze(2).cuda()#to(device=device)
            for l in range(scheduled_sampling_length):
                train_loss = 0
                optimizer.zero_grad()

                cur_X = (X[:,:,l:frameSequence+l].clone()).to(device=device)

                cur_pose = (cur_X[:, 0:poseFeatDim, N].clone()).to(device=device)

                GT = (cur_X[:,0:poseFeatDim, N+1].clone()).unsqueeze(2).cuda()

                # scheduled sampling
                if load_saved_model==True:
                    p=0
                elif epoch<first_train_stage_epochs:
                    p = 1
                elif epoch<first_train_stage_epochs+second_train_stage_epochs:
                    w1 = (epoch - first_train_stage_epochs+1)/second_train_stage_epochs
                    w2= 1-w1

                    weights = torch.tensor([w1, w2], dtype=torch.float).cuda()
                    p = torch.multinomial(weights, 1, replacement=True).cuda().item()

                else:
                    p = 0

                input_pose = p * cur_pose.detach().cuda() + (1-p)* X_hat[:,0:poseFeatDim,:].detach().squeeze(2).cuda()

                X_hat, latent, KL_loss = VAE(cur_X, input_pose)

                recon_loss = (((GT - X_hat[:,0:poseFeatDim,:])**2).sum())/(GT.size(dim=0)) # GT shape ----> [batch_size, inputFeatDim, 1]
                train_loss = recon_loss + KL_loss * alpha

                train_loss.backward()

                epoch_loss += train_loss
                epoch_KL  += KL_loss
                optimizer.step()

        print("epoch: " + str(epoch)+" loss: " + str(epoch_loss.item()) + " KL: " + str(epoch_KL.item() * alpha))

    return VAE, X, latent[0,:,:].unsqueeze(0), X[0,:,N].unsqueeze(0), epoch_loss, optimizer

cuda available
data unit is radian
epoch: 0 loss: 30499.1171875 KL: 0.0
epoch: 1 loss: 4208.41015625 KL: 0.0
epoch: 2 loss: 498.6940002441406 KL: 0.0
epoch: 3 loss: 158.99220275878906 KL: 0.0
epoch: 4 loss: 78.93453216552734 KL: 0.0
epoch: 5 loss: 53.533302307128906 KL: 0.0
epoch: 6 loss: 38.02873611450195 KL: 0.0
epoch: 7 loss: 28.048128128051758 KL: 0.0
epoch: 8 loss: 23.194978713989258 KL: 0.0
epoch: 9 loss: 21.458599090576172 KL: 0.0
epoch: 10 loss: 20.632036209106445 KL: 0.0
epoch: 11 loss: 20.297395706176758 KL: 0.0
epoch: 12 loss: 18.75624656677246 KL: 0.0
epoch: 13 loss: 17.753822326660156 KL: 0.0
epoch: 14 loss: 16.912155151367188 KL: 0.0
epoch: 15 loss: 16.498188018798828 KL: 0.0
epoch: 16 loss: 15.184914588928223 KL: 0.0
epoch: 17 loss: 14.235843658447266 KL: 0.0
epoch: 18 loss: 13.30086898803711 KL: 0.0
epoch: 19 loss: 12.536004066467285 KL: 0.0
epoch: 20 loss: 11.863930702209473 KL: 0.0
epoch: 21 loss: 10.70985221862793 KL: 0.0
epoch: 22 loss: 10.140275001525879 KL: 0.0
epoch: 23 loss: 9.719818115234375 KL: 0.0
epoch: 24 loss: 7.877124309539795 KL: 0.0
epoch: 25 loss: 6.41648006439209 KL: 0.0
epoch: 26 loss: 5.2640767097473145 KL: 0.0
epoch: 27 loss: 4.675246238708496 KL: 0.0
epoch: 28 loss: 4.752994060516357 KL: 0.0
epoch: 29 loss: 4.260623455047607 KL: 0.0
epoch: 30 loss: 208.68763732910156 KL: 1771.3675537109375
epoch: 31 loss: 18.421226501464844 KL: 7.0814619064331055
epoch: 32 loss: 16.831327438354492 KL: 0.2619243860244751
epoch: 33 loss: 16.36933708190918 KL: 0.22026295959949493
epoch: 34 loss: 16.225860595703125 KL: 0.1161663681268692
epoch: 35 loss: 16.09817123413086 KL: 0.14859028160572052
epoch: 36 loss: 16.100046157836914 KL: 0.164580836892128
epoch: 37 loss: 15.891282081604004 KL: 0.13011851906776428
epoch: 38 loss: 15.863426208496094 KL: 0.1438782811164856
epoch: 39 loss: 15.77467155456543 KL: 0.0739947035908699
epoch: 40 loss: 15.756997108459473 KL: 0.1154341995716095
epoch: 41 loss: 15.682149887084961 KL: 0.13609440624713898
epoch: 42 loss: 15.646101951599121 KL: 0.14060918986797333
epoch: 43 loss: 15.596468925476074 KL: 0.06942499428987503
epoch: 44 loss: 15.487974166870117 KL: 0.13864728808403015
epoch: 45 loss: 15.456522941589355 KL: 0.09747464954853058
epoch: 46 loss: 15.596013069152832 KL: 0.10960092395544052
epoch: 47 loss: 15.446678161621094 KL: 0.09400694817304611
epoch: 48 loss: 15.414061546325684 KL: 0.07403453439474106
epoch: 49 loss: 15.446662902832031 KL: 0.07924196124076843
epoch: 50 loss: 15.337182998657227 KL: 0.07696129381656647
epoch: 51 loss: 15.423378944396973 KL: 0.1136254072189331
epoch: 52 loss: 15.3486967086792 KL: 0.09196256101131439
epoch: 53 loss: 15.432474136352539 KL: 0.11669618636369705
epoch: 54 loss: 15.23315143585205 KL: 0.08362749963998795
epoch: 55 loss: 15.270442962646484 KL: 0.0592842772603035
epoch: 56 loss: 15.257233619689941 KL: 0.08109745383262634
epoch: 57 loss: 15.207656860351562 KL: 0.058704279363155365
epoch: 58 loss: 15.246068954467773 KL: 0.08804851025342941
epoch: 59 loss: 15.179248809814453 KL: 0.06591930240392685
epoch: 60 loss: 16.24458122253418 KL: 0.05520284175872803
epoch: 61 loss: 18.20315170288086 KL: 0.07300713658332825
epoch: 62 loss: 20.9660701751709 KL: 0.10368426144123077
epoch: 63 loss: 26.014833450317383 KL: 0.1356126070022583
epoch: 64 loss: 35.390743255615234 KL: 0.1684873253107071
epoch: 65 loss: 32.68571090698242 KL: 0.14424605667591095
epoch: 66 loss: 52.215614318847656 KL: 0.26431578397750854
epoch: 67 loss: 189.5343017578125 KL: 1.0707039833068848
epoch: 68 loss: 75.52210235595703 KL: 0.23325027525424957
epoch: 69 loss: 143.2079620361328 KL: 0.38768690824508667
epoch: 70 loss: 157.3100128173828 KL: 0.49191996455192566
epoch: 71 loss: 192.56976318359375 KL: 0.829379677772522
epoch: 72 loss: 258.619873046875 KL: 0.6730182766914368
epoch: 73 loss: 521.1996459960938 KL: 3.7076361179351807
epoch: 74 loss: 330.8260803222656 KL: 0.9579944014549255
epoch: 75 loss: 604.3058471679688 KL: 1.2703361511230469
epoch: 76 loss: 475.0205078125 KL: 0.9360959529876709
epoch: 77 loss: 731.9593505859375 KL: 2.7841150760650635
epoch: 78 loss: 975.5214233398438 KL: 1.2265475988388062
epoch: 79 loss: 924.7633056640625 KL: 0.873565673828125
epoch: 80 loss: 940.7155151367188 KL: 0.5359449982643127
epoch: 81 loss: 855.8935546875 KL: 0.9077990651130676
epoch: 82 loss: 849.4100952148438 KL: 0.7129514813423157
epoch: 83 loss: 743.1096801757812 KL: 0.5308371782302856
epoch: 84 loss: 849.7276611328125 KL: 0.9092111587524414
epoch: 85 loss: 806.3848876953125 KL: 0.49240317940711975
epoch: 86 loss: 773.6209716796875 KL: 0.35794520378112793
epoch: 87 loss: 714.7335815429688 KL: 0.36182066798210144
epoch: 88 loss: 725.5518188476562 KL: 0.6665423512458801
epoch: 89 loss: 725.10498046875 KL: 0.3123415410518646
epoch: 90 loss: 749.900634765625 KL: 0.5664316415786743
epoch: 91 loss: 746.6582641601562 KL: 0.8775449395179749
epoch: 92 loss: 740.4017944335938 KL: 0.4976818263530731
epoch: 93 loss: 709.8568115234375 KL: 0.34913212060928345
epoch: 94 loss: 716.6048583984375 KL: 0.7065077424049377
epoch: 95 loss: 681.2711181640625 KL: 0.36696088314056396
epoch: 96 loss: 740.9374389648438 KL: 0.803412675857544
epoch: 97 loss: 646.1436767578125 KL: 0.2696443796157837
epoch: 98 loss: 664.8652954101562 KL: 0.37316083908081055
epoch: 99 loss: 614.1035766601562 KL: 0.2937750816345215
epoch: 100 loss: 703.1944580078125 KL: 0.4119395315647125
epoch: 101 loss: 644.4376220703125 KL: 0.36282405257225037
epoch: 102 loss: 673.5081176757812 KL: 0.35550656914711
epoch: 103 loss: 599.3011474609375 KL: 0.18692539632320404
epoch: 104 loss: 589.5043334960938 KL: 0.33308255672454834
epoch: 105 loss: 589.5310668945312 KL: 0.20958860218524933
epoch: 106 loss: 633.5597534179688 KL: 0.3015775978565216
epoch: 107 loss: 587.228271484375 KL: 0.2859556972980499
epoch: 108 loss: 633.8538818359375 KL: 0.3062727153301239
epoch: 109 loss: 576.3986206054688 KL: 0.3453579843044281
epoch: 110 loss: 605.309814453125 KL: 0.7614783048629761
epoch: 111 loss: 559.1953735351562 KL: 0.43579205870628357
epoch: 112 loss: 601.722412109375 KL: 0.31123608350753784
epoch: 113 loss: 591.31494140625 KL: 0.38346976041793823
epoch: 114 loss: 677.573974609375 KL: 1.5325040817260742
epoch: 115 loss: 535.7906494140625 KL: 0.2391374409198761
epoch: 116 loss: 550.9417114257812 KL: 0.5806562900543213
epoch: 117 loss: 565.160400390625 KL: 0.31043145060539246
epoch: 118 loss: 584.8384399414062 KL: 0.8044378757476807
epoch: 119 loss: 616.1946411132812 KL: 0.9010312557220459
epoch: 120 loss: 589.0029907226562 KL: 0.5001609325408936
epoch: 121 loss: 558.1272583007812 KL: 0.36073750257492065
epoch: 122 loss: 522.8496704101562 KL: 0.4064602553844452
epoch: 123 loss: 563.9342651367188 KL: 0.2904842495918274
epoch: 124 loss: 562.810791015625 KL: 0.5313525199890137
epoch: 125 loss: 608.248046875 KL: 0.7063066363334656
epoch: 126 loss: 517.7711791992188 KL: 0.2636258602142334
epoch: 127 loss: 525.2127075195312 KL: 0.2245425432920456
epoch: 128 loss: 576.1654663085938 KL: 0.6417035460472107
epoch: 129 loss: 583.733642578125 KL: 0.47674331068992615
epoch: 130 loss: 522.4052124023438 KL: 0.34901681542396545
epoch: 131 loss: 565.4308471679688 KL: 0.232156440615654
epoch: 132 loss: 553.7698364257812 KL: 0.323140025138855
epoch: 133 loss: 586.9306640625 KL: 1.2630860805511475
epoch: 134 loss: 488.27557373046875 KL: 0.43516507744789124
epoch: 135 loss: 527.9531860351562 KL: 0.3459720313549042
epoch: 136 loss: 548.0935668945312 KL: 0.4123835861682892
epoch: 137 loss: 543.787841796875 KL: 0.2853831350803375
epoch: 138 loss: 536.0159912109375 KL: 0.27312254905700684
epoch: 139 loss: 546.4530639648438 KL: 0.5541123151779175

And I continued training for another 100 epochs:
epoch: 0 loss: 544.358642578125 KL: 0.5914149880409241
epoch: 1 loss: 505.7805480957031 KL: 0.38028520345687866
epoch: 2 loss: 443.52166748046875 KL: 0.4431949555873871
epoch: 3 loss: 643.8592529296875 KL: 2.7874884605407715
epoch: 4 loss: 491.23486328125 KL: 0.34796473383903503
epoch: 5 loss: 572.5677490234375 KL: 0.47636649012565613
epoch: 6 loss: 538.56591796875 KL: 0.7826776504516602
epoch: 7 loss: 505.7750549316406 KL: 0.5553744435310364
epoch: 8 loss: 604.9829711914062 KL: 0.7179495692253113
epoch: 9 loss: 484.38641357421875 KL: 0.45356979966163635
epoch: 10 loss: 599.8277587890625 KL: 1.4839460849761963
epoch: 11 loss: 549.4735717773438 KL: 1.2648544311523438
epoch: 12 loss: 480.8637390136719 KL: 3.1619513034820557
epoch: 13 loss: 480.6685791015625 KL: 5.205479145050049
epoch: 14 loss: 506.57611083984375 KL: 4.864458084106445
epoch: 15 loss: 549.2564697265625 KL: 9.807156562805176
epoch: 16 loss: 613.8514404296875 KL: 9.113334655761719
epoch: 17 loss: 501.9906921386719 KL: 11.410046577453613
epoch: 18 loss: 477.0592346191406 KL: 12.0811128616333
epoch: 19 loss: 505.9403381347656 KL: 12.787254333496094
epoch: 20 loss: 469.2815856933594 KL: 12.78911018371582
epoch: 21 loss: 537.4259643554688 KL: 12.93073844909668
epoch: 22 loss: 453.3262939453125 KL: 13.678216934204102
epoch: 23 loss: 487.96478271484375 KL: 13.802533149719238
epoch: 24 loss: 510.89300537109375 KL: 14.773233413696289
epoch: 25 loss: 543.4826049804688 KL: 16.42165756225586
epoch: 26 loss: 565.0093383789062 KL: 16.566818237304688
epoch: 27 loss: 519.4603881835938 KL: 17.37101936340332
epoch: 28 loss: 518.1926879882812 KL: 18.62165069580078
epoch: 29 loss: 512.0256958007812 KL: 17.694440841674805
epoch: 30 loss: 478.7466735839844 KL: 19.09602165222168
epoch: 31 loss: 429.68878173828125 KL: 20.003482818603516
epoch: 32 loss: 497.278076171875 KL: 21.56376838684082
epoch: 33 loss: 503.67755126953125 KL: 24.277345657348633
epoch: 34 loss: 612.775146484375 KL: 22.738990783691406
epoch: 35 loss: 557.6171875 KL: 19.988445281982422
epoch: 36 loss: 539.9255981445312 KL: 25.974149703979492
epoch: 37 loss: 483.6654052734375 KL: 23.685108184814453
epoch: 38 loss: 515.800537109375 KL: 26.66796112060547
epoch: 39 loss: 525.6260986328125 KL: 23.748584747314453
epoch: 40 loss: 458.6383056640625 KL: 24.160545349121094
epoch: 41 loss: 462.0157470703125 KL: 24.52272605895996
epoch: 42 loss: 470.8816223144531 KL: 24.42452621459961
epoch: 43 loss: 426.82861328125 KL: 24.829322814941406
epoch: 44 loss: 545.5446166992188 KL: 25.695920944213867
epoch: 45 loss: 510.5130310058594 KL: 26.169050216674805
epoch: 46 loss: 527.9548950195312 KL: 26.36553192138672
epoch: 47 loss: 495.6191101074219 KL: 27.640392303466797
epoch: 48 loss: 545.8670654296875 KL: 26.163671493530273
epoch: 49 loss: 484.4084777832031 KL: 27.724061965942383
epoch: 50 loss: 514.3748168945312 KL: 29.91745376586914
epoch: 51 loss: 587.0398559570312 KL: 28.68882942199707
epoch: 52 loss: 480.8255920410156 KL: 30.606407165527344
epoch: 53 loss: 476.2035217285156 KL: 28.670635223388672
epoch: 54 loss: 439.28936767578125 KL: 29.135217666625977
epoch: 55 loss: 444.4865417480469 KL: 29.801206588745117
epoch: 56 loss: 446.6828308105469 KL: 29.028263092041016
epoch: 57 loss: 456.685791015625 KL: 29.68742561340332
epoch: 58 loss: 489.0218200683594 KL: 28.470592498779297
epoch: 59 loss: 440.5421142578125 KL: 31.14491081237793
epoch: 60 loss: 526.8546142578125 KL: 29.1456298828125
epoch: 61 loss: 502.2847595214844 KL: 31.19633674621582
epoch: 62 loss: 496.770263671875 KL: 32.510498046875
epoch: 63 loss: 504.4200134277344 KL: 32.934532165527344
epoch: 64 loss: 531.3699951171875 KL: 36.5645637512207
epoch: 65 loss: 445.373046875 KL: 35.74449157714844
epoch: 66 loss: 448.3841247558594 KL: 33.053104400634766
epoch: 67 loss: 411.0800476074219 KL: 32.45486831665039
epoch: 68 loss: 454.9539794921875 KL: 34.142574310302734
epoch: 69 loss: 467.9690246582031 KL: 33.53937911987305
epoch: 70 loss: 421.2640686035156 KL: 32.56943893432617
epoch: 71 loss: 427.9134216308594 KL: 31.36210060119629
epoch: 72 loss: 526.1041259765625 KL: 34.80607604980469
epoch: 73 loss: 494.36798095703125 KL: 36.99067687988281
epoch: 74 loss: 486.51971435546875 KL: 38.45036315917969
epoch: 75 loss: 396.05517578125 KL: 36.818748474121094
epoch: 76 loss: 411.96221923828125 KL: 36.27510070800781
epoch: 77 loss: 451.77825927734375 KL: 37.816123962402344
epoch: 78 loss: 472.1197814941406 KL: 37.120418548583984
epoch: 79 loss: 475.6114501953125 KL: 37.12137985229492
epoch: 80 loss: 446.37921142578125 KL: 39.191429138183594
epoch: 81 loss: 420.8083801269531 KL: 35.29905319213867
epoch: 82 loss: 413.5904541015625 KL: 37.87409591674805
epoch: 83 loss: 445.56201171875 KL: 36.565650939941406
epoch: 84 loss: 420.17236328125 KL: 36.324310302734375
epoch: 85 loss: 434.4346008300781 KL: 36.1773796081543
epoch: 86 loss: 392.82489013671875 KL: 36.10137176513672
epoch: 87 loss: 449.11004638671875 KL: 35.46767044067383
epoch: 88 loss: 442.52716064453125 KL: 35.53940963745117
epoch: 89 loss: 441.1539001464844 KL: 37.004539489746094
epoch: 90 loss: 425.94110107421875 KL: 38.571372985839844
epoch: 91 loss: 395.73858642578125 KL: 38.19450378417969
epoch: 92 loss: 489.399169921875 KL: 36.86212921142578
epoch: 93 loss: 410.8083801269531 KL: 38.584346771240234
epoch: 94 loss: 417.1232604980469 KL: 36.909812927246094
epoch: 95 loss: 399.3186950683594 KL: 38.01395797729492
epoch: 96 loss: 473.8252258300781 KL: 38.23799133300781
epoch: 97 loss: 425.333251953125 KL: 39.477622985839844
epoch: 98 loss: 411.835693359375 KL: 39.992000579833984
epoch: 99 loss: 396.6147155761719 KL: 39.25416564941406

please add evidence of this non convergence... imho, giving the network the future poses is not ideal, as the network has a easy life predicting what's between two poses by linearly interpolating them

Thanks @Alberto for your comment, I updated the post with my training function and the output of the training stage. And about including the future poses, my reasoning was that the VAE is going to be able to somehow encrypt the motion into the latent space, which later on would be more useful in the evaluation stage. Learning Motion Manifolds with Convolutional Autoencoders has used the same method for example. But I will try your suggestion, giving only the past poses at train stage.


text:
Build a global model based on thousand local models
I have 1000 meters for electricity usage. I want to train one global model for all of them to predict their consumption for the next few days. So, when I train the model with all of the meter data, the prediction is not good. But when I train the model for each one separately, the prediction is good. So, I am looking for a way to train 1000 model with these ids, and then with these 1000 model, I create one global model. Is it doable? Do you have any thoughts regarding this? Thank you.


text:
What options do I have when performing RAG on similarly phrased chunks?
I'm building a RAG pipeline to extract real estate phrases from excel documents. These phrases are short (2-5 words) and are often phrased differently.
I've manually added in different phrases to each prompt to suggest the LLM to look for them.
What else could I do? What more powerful options do I have?

Search for each term manually? -> slow due to loads of calls
Embed the terms in the excel doc and similarity search the embedding of 'Total Income' against those in the excel doc?

# Example prompt with 4 similar terms
income = f"""What is the Total Income?

It may also be called:
- Total Revenue
- Effective Gross Income
- Total Operating Receipts
- Total Operating Income

Or something similar

Return a signed number or nan. Do not write anything else."""
"""
```

If you know what keywords you're looking for, it might be easier to build an inverted index for keyword searching

@Karl can you please expand? What does building an inverted index involve? How would I do it? Code sample?

It's a database structure for fast text keyword searching
https://en.wikipedia.org/wiki/Inverted_index
https://www.elastic.co/elasticsearch/