text:
How can I formulate a nonogram problem as a constraint satisfaction problem?
I've just started learning CSP and I find it quite exciting. Now I'm facing a nonogram solving problem and I want to solve it using backtracking with CSP.
The first problem that I face is that I cannot quite figure out what such variables, domains, and constraints could be. My first thought was to make every field(those squares) a variable with such domain: $D_i = \{1,0\}$, where 1 means that a certain field has been colored black and 0 white.
So far I've been mostly learning binary constraints, and I was thinking of using the AC-3 or forward checking algorithm for propagation during the backtracking algorithm.
As far I know constraints of all arity can be represented as a set of binary constraints, so that would enable me to use the algorithms I mentioned. But that leads me to the problem of defining constraints. If every field was a variable then each line and column would be a constraint, based on how certain lines should be colored(numbers defining line for example 2,3,2).
But it's all new and quite hard for me to imagine and come up with. I've been reading some articles and papers on that but they were too advanced for me.
So, does anybody have an idea how can I formulate a nonogram problem as a constraint satisfaction problem?

You are trying to both learn to formulate the problem and write a solver at the same time. Have you tried formulating the problem in an existing solver (eg Minion)? This way you get practice formulating problems before you work on implementing solvers.

I try to implement a simple solver to understand algorithms and heuristics behind. That gave me better understanding how it all works.

But after all, I decided that, a variable will be a single row or column and their domains are every possible combination od that linę. Not very powerful but you have to start somewhere.


text:
What are examples of tutorials and blogs for beginners to master the cross-lingual information retrieval?
Currently, I am following the Dan Jurofsky NLP Tutorial and CS 224 Stanford 2019. Can you list tutorials and blogs for beginners to master the cross-lingual information retrieval?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the difference between model and data distributions?
Is there any difference between the model distribution and data distribution, or are they the same?

Yes. In Machine Learning we consider that the samples in your training set are sampled from an underlying distribution called the data generating distribution.
Generative models classify the samples by trying to learn the distribution of the data. In most cases, either the model is incapable of doing so, or the training samples aren't enough to properly describe the data-generating distribution, so the model learns an approximation of this. This is what you call the model's distribution.
You can find more info about these concepts in a more detailed answer I wrote. If you're familiar with GANs, you can also read this post, to see where these two concepts come into play when training the two networks.

This answer seems correct, but why are you talking about generative models? Are model distributions only associated with generative models? Is the terminology "data distribution" and "model distributions" only useful in the case of GANs or generative models? Is this terminology widely known in machine learning, or only Goodfellow's book uses it?

Generative models is a broad category of  ML models including the generator from GANs, naive bayes, etc. The other category are discriminative models, which don't try to explicitly model the data's distribution, i.e. there's no notion of a model's distribution.  The terms "generative" and "discriminative" models are widely used, as is the notion of the data-generating distribution. I'm unaware, though, if the latter was coined by Goodfellow or not (it might have also different names). Nowadays, though, it is the only widely-used terminology I'm aware of.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How was the DQN trained to play many games?
Some people claim that DQN was used to play many Atari games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was common to all those games? Only the architecture of the RL agent? Did the reward function change for each game?

I will provide an answer later (if nobody does it), but I think that, meanwhile, someone will provide an answer.


text:
What is the intuition behind grid-based solutions to POMDPs?
After spending some time reading about POMDP, I'm still having a hard time understanding how grid-based solutions work.
I understand the finite horizon brute-force solution, where you have your current belief distribution, enumerate every possible collection of action/observation combinations for a given depth and find the expected reward.
I have tried to read some sources about grid-based approximations, for example, these slides describe the grid-based approach.
However, it's not clear to me what exactly is going on. I'm not understanding how the value function is actually computed. After you take an action, how do you update your belief states to be consistent with the grid? Does the grid-based solution simply reduce the set of belief states? How does this reduce the complexity of the problem?
I'm not seeing how this reduces the number of actions, observation combinations needed to be considered for a finite-horizon solution.

I will attempt to provide an answer to your questions based on the information you can find in the papers A Heuristic Variable Grid Solution Method for POMDPs (1997) by Ronen I. Brafman and Point-based value iteration: An anytime algorithm for POMDPs (2003) by Joelle Pineau et al.
A grid-based approximate solution to a POMDP attempts to estimate a value function only at a subset of the number of belief states. Why? Because estimating the value function for all belief states is typically computationally infeasible for non-small POMDPs, given that the belief-space MDP (i.e. an MDP where the state space consists of probability distributions over the original states of the POMDP) of a POMDP with $n$ states has an uncountably large state space. Why? Because of the involved probability distributions.
How do we compute the value for the belief states that do not correspond to a point of the grid? We can use e.g. interpolation, i.e. the value of a belief state that does not correspond to a point of the grid is computed as a function of the value of the belief states that correspond to other grid points (typically, the neighboring grid points).
Why is this approach feasible? The assumption is that interpolation is not as expensive as computing the value of a belief state. However, note that you may not need to interpolate at every step of your algorithm, i.e. interpolation could be performed only when the value of a certain belief state is required.
How do you compute the value of a belief state that corresponds to a grid point? It can be computed with a value iteration (dynamic programming) algorithm for POMDPs. An overview of a value iteration algorithm can be found in section 2 of the paper Point-based value iteration: An anytime algorithm for POMDPs. Here's an example of the application of the value iteration algorithm for POMDPs.
The grid-based approach, introduced in Computationally Feasible Bounds for Partially Observed Markov Decision Processes (1991) by William S. Lovejoy, is very similar to the point-based approach, which was introduced in Point-based value iteration: An anytime algorithm for POMDPs. The main differences between the two approaches can be found in section 3 of Point-based value iteration: An anytime algorithm for POMDPs.
The idea of discretizing your problem or simply computing the desired value at a  subset of the domain has been applied in other contexts too. For example, in the context of computer vision, you can approximate the derivative (or gradient) of an image (which is thus considered a function) at discrete points of the domain (i.e. the pixels).
There's a Julia implementation of the first grid-based approximative solution to POMDP. There's also a Python implementation of the point-based approach. These implementations may help you to understand the details of these approaches.

Thanks, I am still confused on this approximation method though. Actions have associated probability distributions that produce belief states which may not be on the grid, basically your state can be transformed to something outside your state space, so how do you handle this?

@FourierFlux I don't understand what you mean by "outside of the state space" in "basically your state can be transformed to something outside your state space". Anyway, I've updated this answer to include more details and references.

Thanks, I will read them. I think my confusion is that the belief state is continuous and an action can take a point in the belief space grid and update it to no longer be in the grid. So it's not clear to me how this is addressed when it comes to computations. Even restricting your actions to a smaller subset won't mean they will always result in transforming a grid point to a grid point. When transforming the POMDP into a discritized MDP, what defines your set of actions and associated probabilities?

When you say "an action can take a point in the belief space grid and update it to no longer be in the grid". Well, it's true that an action may move the environment to a state that is not represented in the grid. That's why you may need to use something like interpolation. Are you familiar with the concept of interpolation?

Lets consider a MDP with a continuous state space, discretizing it still leaves the problem of your actions giving states not in the discretization, where exactly does the interpolation come in when it comes to finding a policy and how does it help? Do you define new actions and round the resulting state to the "closest" one? Doesn't interpolation just lead back to the same issue of having a continuous state space?

@FourierFlux I think your first question "Lets consider a MDP with a continuous state space, discretizing it still leaves the problem of your actions giving states not in the discretization" is related to the discretization of continuous space MDPs and how it works. Maybe you could ask a new question. You're very confused. First of all, do you understand that if you find the value function you can derive the policy from it? So, if that's the case, we just find a value function in order to solve an MDP. How? There is a value iteration algorithm for MDPs and there's also one for POMDPs.

@FourierFlux Then you say "Do you define new actions", but I don't understand why you would define new actions. Then you say "Doesn't interpolation just lead back to the same issue of having a continuous state space?", interpolation is an operation that allows you to find the values of a function at points where you don't have those values. For instance, if you have $f(a)$ and $f(b)$ and you want to find the value of $f$ at $a \leq c \leq b$, then you can SOMEHOW combine $f(a)$ and $f(b)$ to get $f(c)$.

In our context, we have the value of the belief states at certain grid points (we compute this with e.g. one value iteration algorithm for POMDPs). If we wanted to find the value of the belief states not at the grid points, we would compute them SOMEHOW by using the value of the belief states at the grid points (which we have!). You don't introduce any new action or whatever. To be more concrete, assume that we have the value at the belief states $b_1, b_3, b_5$ (which correspond to some grid points), i.e. we have $v(b_1)$, $v(b_3)$ and $v(b_5)$ (computed with value iteration).

Now, if we wanted to find $v(b_2)$, we could SOMEHOW combine $v(b_1)$ and $v(b_3)$, i.e. $v(b_2) = g(v(b_1), v(b_3))$, where $g$ is some function that combines the inputs. Note that I don't know if this is actually done in practice. I am just giving you the GENERAL idea behind interpolation!!


text:
Should I be trying to create a generic or specific (to particular game) reinforcement learning agent?
I'm creating an RL application for the game Connect Four.
In general, should I be aiming to create an application that's more generic, which would 'learn' different games, or specific to a particular game (e.g. Connect Four, by assigning greater rewards to certain token positions in the C4 grid)?
Does the difference between the two approaches just come down to adapting their respective reward functions to reward specific achievements or positions (in a board game setting), or something else?

If what you mean by a generic reinforcement-learning application is an application that can learn any game (or some games), then you can't do it. Why? Because the goal of each game isn't the same, so you have to adapt the rewards depending on the game. If you just want to make an AI for Connect Four, I suggest you to make a specific RL application for that game.
I want to mention another thing: you shouldn't give a reward based on token's position, because it's hard to know what token's position is the best. Instead, just assign a big reward to the winner. That way, you're generalizing your algorithm, and you're avoiding your AI to focus on the wrong goal.
You have to be careful, you don't know for sure what token's position is the best. Try to give a reward ONLY when a player wins the game, because you can know for sure that winning is the best thing to do. It may sound idiot said like that, but that way, you're ensuring you AI to learn by itself what token's position is the best.

Hey! Thanks for trying to contribute to our site :) Btw, I think you should also mention the case of DQN, if you are familiar with it. Some people claim that DQN was used to play many games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was common to all those games? Only the architecture? Did the reward function change for each game?

@nbro♦ Hi! Thanks for your reply! Sadly, I'm not familiar with Deep Q-Networks, but after reading you comment, I read some articles about DQNs. It's really interesting, but I found nothing about how DQNs are able to play many games. But the reward function definitely changes from a game to another, or maybe the goal of each game was the same. For example, the common goal to each game could simply be: surviving.

Hi Dara, thanks for your reply. Regarding your second paragraph – what's the difference between "you shouldn't give a reward based on token's position" and "giving a score for tokens that are aligned"? Aren't you saying we should give greater rewards when the tokens are in certain positions, this is the same, no?

nbro, thank you for your continued help (for me and the rest of the community, your assistance is invaluable). Regarding your reply and to clarify: is the only thing that changes from game to game the rewards assigned in the reward function?

I guess a follow-up question to the above main question is: should I assign the reward values, or let the AI work out what token positions are the best?

@mason7663 I think the question I asked above would be a good new post on the site. If you want, you can ask this question(s) in a new post: "Some people claim that DQN was used to play many games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was common to all those games? Only the architecture? Did the reward function change for each game?".

@mason7663 Regarding your follow-up question, are you asking if you should allow the agent to learn the actual reward function? This is also an interesting question that you can ask separately. In some cases, you are given only rewards and not necessarily a definition of the reward function. So this may be interesting to ask.

Asked here:
https://ai.stackexchange.com/questions/20004/in-rl-if-i-assign-the-rewards-for-better-positional-play-the-algorithm-is-lear

@mason7663 Actually, I think this is a different question and it's not completely clear to me what you are actually asking :P

@mason7663 Hi! I updated my answer, because I haven't understood what you meant by "assigning greater rewards to certain token positions". Anyway, please read my answer again, I'm suggesting something else than "giving a score for tokens that are aligned".


text:
Is maximum likelihood estimation meaningless for a dataset of only outliers?
From my understanding, maximum likelihood estimation chooses the set of parameters for the estimator that maximizes likelihood with the ground truth distribution.
I always interpreted it as the training set having a tendency to have most examples near the mean or the expected value of the true distribution. Since most training examples are close to the mean (since they have been sampled from this distribution) maximizing the estimator's chance of sampling these examples gets the estimated distribution close to the ground truth distribution.
This would mean that any MLE procedure on a dataset of outliers should fail miserably. Are this interpretation and conclusion correct? If not, what is wrong with the mentioned interpretation of maximizing likelihood for an estimator?


text:
How can artificial intelligence predict the next possible moves of the player?
When you play video games, sometimes there is an AI that attempts to predict what are you going to do.
For example, in the Candy Crush game, if you finish the level and you still have moves remaining, you get to see fishes or other powers destroying the other candies, but, instead of watching 10 minutes of your combos without moving at all after accomplishing a level, like this Longest video game combo ever probably, it tells an alert that says tap to skip, so basically the AI is predicting all the possible combos that will keep proceeding automatically and calculating every automatic move.
How can artificial intelligence predict such a thing?

Welcome to SE:AI!  (Apologies that the answers are so generic.  My personal sense, in relation to this particular problem--producing the set of Candy Crush combos, is likely a combinatorial approach. In terms of prediction, my guess would be you train a reinforcement algorithm to identify the biases of the algorithm that is producing the combos in the game, and exploit the identification of those biases to more and more accurately predict the most likely next combo.)

However, if you're pressing "skip", and the AI is "predicting your next moves", that is likely a misrepresentation, unless your gameplay specifically has been observed and modeled sufficiently by a reinforcement learning algorithm.  This is unlikely due to the computational cost (time, space, energy) of training such algorithms.

AI is trained to predict such thing because that is their purpose, they are given almost all possibilities of move they can do to current state of the game and chose the best possible outcome of the possible move, but not only that the AI also predict what happens after that and predict the outcome of that prediction, just like a chess AI that can predict how to checkmate a player just by one move made by the player, so they did not just predict what move to do now but also what move to do after that move has been done
this can be done with deep learning as you can read here : https://towardsdatascience.com/predicting-professional-players-chess-moves-with-deep-learning-9de6e305109e
https://electronics.howstuffworks.com/chess1.htm

In video games, usually the developer spend an dedicated time to train their AI by feeding them with learning data, provided by either the developer itself or the feedback from open/closed beta tester that participated, and from that data, the developer can model the learning pattern for the algorithm and proceed to train them with some sets of goal.

Artificial Intelligence can predict such a thing because before they release the game, the train the BOT or AI to play the game million times so they have a model or BOT that can predict every next move or combo that they can do or predict all moves that can finish the game.
for example snake game. what they do to predict moves is train the model or bot to play the game when the snake perform some action . the snake got reward which can be positive or negative reward. the goal of the snake is to learn what action that maximize the reward, given every possible state. States are the observations that the agent receives at each iteration from the environment
this is the link that can give you the detail : https://towardsdatascience.com/how-to-teach-an-ai-to-play-games-deep-reinforcement-learning-28f9b920440a

AI can predict such thing by reading a data that they previously store by play the game many times. Using the data the AI can learn which is the best action to do. For example AI can find the best path to evade all incoming bullet while shooting down all the enemies in bullet hell game.


text:
Is this a good approach to evaluate the game state with a neural network?
I've written a Monte Carlo Tree Search player for the game of Castle (AKA Shithead, Shed, Palace...). I have set this MCTS player to play against a basic rule-based AI for ~30000 games and collected ~1.5 million game states (as of now) along with whether the MCTS player won that particular game in the end after being in that particular game state. The game has a large chance aspect, and, currently, the MCTS player is winning ~55% of games. I want to see how high I can push it. In order to do this, I aim to produce a NN that will act as a game state evaluation function to use within the MCTS.
With this information, I've already tried an SVM, but came to the conclusion that the game space is too large for the SVM to classify a given state accurately.
I hope to be able to train a NN to evaluate a given state and return how good that state is for the MCTS player. Either with a binary GOOD/BAD or I think it would be more helpful to return a value between 0-1.
The input to the NN is a $4 \times 41$ NumPy array of binary values (0, 1) representing the MCTS players hand, MCTS face-up cards, OP face-up cards, MCTS no. face-down cards, OP no. face-down cards. Shown below.

Describes the np.array:

The np.array is made from the database entries of game states. An example of this information is below. However, I am currently omitting the TOP & DECK_EMPTY columns in this model. WON (0, 1) is used as the label.

This is my keras code:
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(2, activation=tf.nn.softmax))

model.compile(optimizer='adam',
             loss='sparse_categorical_crossentropy',
             metrics=['accuracy'])

model.fit(X_train, y_train, epochs=3)

This model isn't performing.

Do you think it is possible to obtain a useful NN with my current approach?

What layers should I look to add to the NN?

Can you recommend any training/learning material that I could use to try and get a better understanding?


text:
How much time does it take to train DQN on Atari environment?
I am trying to build a DQN model for the Atari Pong game, but I am not sure whether the model is learning at all.
I am using the architecture described in the paper Playing Atari with Deep Reinforcement Learning. And I tested the model on a simpler environment (like CartPole), which worked great, but I am not seeing any progress at all with Pong, I have been training the model for 2-3 hours and its performance is no better than taking random actions.
Should I just keep waiting or there might be something wrong with my code. Around how many episodes should it take before I see some positive results?

Don't bother with hours, everyone has different hardware. They say in paper that they train for 10 million frames which is 10 million timesteps in the environment. How many did you complete in 2-3 hours ?

@Brale_ 100 episodes takes roughly 2 hours, and each episode takes maximum 10,000 steps (I am running each episode until game is lost/won).

I don't expect same results as in paper in just 100 episodes, but I should be getting at least some improvements right?

Well for you it could take entire day to reach around 10 million timesteps. Leave it over night for 7-8 hours and see in the morning if it gets any better. If not then you might have some bug.


text:
Can I use deepfake to rotoscope for animation?
I'm a big fan of animation and have kept an eye on the deepfake's ability to replicate full body motion.
So I ask

Is there a deepfake software available I can use to gather animation from a video?
Are there publically released a deepfake body tracking tools out there? Even for normal video

Hi and welcome to AI SE! Please, ask just one question per post. What do you precisely mean by "gather animation from a video"?


text:
Why is my DQN model not getting better?
I've created a deep Q network. My model does not get better, and can't see what I'm doing wrong. I'm new to RL.
Replay Memory
class ReplayMemory(object):

def __init__(self, input_shape, mem_size=100000):
    self.states = np.zeros((mem_size, input_shape))
    self.actions = np.zeros(mem_size, dtype=np.int32)
    self.next_states = np.zeros((mem_size, input_shape))
    self.rewards = np.zeros(mem_size)
    self.terminals = np.zeros(mem_size)

    self.mem_size = mem_size
    self.mem_count = 0

def push(self, state, action, next_state, reward, terminal):

    idx = self.mem_count % self.mem_size

    self.states[idx] = state
    self.actions[idx] = action
    self.next_states[idx] = next_state
    self.rewards[idx] = reward
    self.terminals[idx] = terminal

    self.mem_count += 1

def sample(self, batch_size):
    batch_index = np.random.randint(0, min(self.mem_count, self.mem_size), batch_size)

    states = self.states[batch_index]
    actions = self.actions[batch_index]
    next_states = self.next_states[batch_index]
    rewards = self.rewards[batch_index]
    terminals = self.terminals[batch_index]

    return (states, actions, next_states, rewards, terminals)

def __len__(self):
    return min(self.mem_count, self.mem_size)

DQN Agent
class DQN_Agent(object):

  def __init__(self, n_actions,n_states, ALPHA=0.001, GAMMA=0.99, eps_start=1 , eps_end=0.01, eps_decay=0.005):
      self.n_actions = n_actions
      self.n_states = n_states

      self.memory = ReplayMemory(n_states)

      self.ALPHA = ALPHA
      self.GAMMA = GAMMA

      self.eps_start = eps_start
      self.eps_end = eps_end
      self.eps_decay = eps_decay

      self.model = self.create_net()
      self.target = self.create_net()

      self.target.set_weights(self.model.get_weights())

      self.steps_counter = 0

  def create_net(self):

    model = Sequential([
        Dense(64, activation="relu", input_shape=(self.n_states,)),
        Dense(32, activation="relu"),
        Dense(self.n_actions)
    ])

    model.compile(loss="huber_loss", optimizer=Adam(lr=0.0005))

    return model

def select_action(self, state):
    ratio = self.eps_end + (self.eps_start-self.eps_end)*np.exp(-1*self.eps_decay*self.steps_counter)
    rand = random.random()
    self.steps_counter += 1

    if ratio > rand:
        #print("random")
        return np.random.randint(0, self.n_actions)
    else:
        #print("not random")
        return np.argmax(self.model.predict(state))

def train_model(self, batch_size):
    if len(self.memory) < batch_size:
        return None

    states, actions, next_states, rewards, terminals = self.memory.sample(batch_size)

    q_curr = self.model.predict(states)
    q_next = self.target.predict(next_states)
    q_target = q_curr.copy()

    batch_index = np.arange(batch_size, dtype=np.int32)

    q_target[batch_index, actions] = rewards + self.GAMMA*np.max(q_next, axis=1)*terminals

    _ = self.model.fit(states, q_target, verbose = 0)

    if self.steps_counter % 10 == 0:
        self.target.set_weights(self.model.get_weights())

My training loop
n_games = 50000
agent = DQN_Agent(2, 4)

scores = []
avg_scores = []

for epoch in range(n_games):
    done = False
    score = 0
    state = env.reset()

    while not done:
       #env.render()
       action = agent.select_action(state.reshape(1,-1))
       next_state, reward, done, _ = env.step(action)

       score += reward

       agent.memory.push(state, action, next_state, reward, done)

       state = next_state
       agent.train_model(64)

   avg_score = np.mean(scores[max(0, epoch-100):epoch + 1])
   avg_scores.append(avg_score)
   scores.append(score)
   print(score, avg_score)


text:
Is the TD-residual defined for timesteps $t$ past the length of the episode?
Let $\mathcal{S}$ be the state-space in a reinforcement learning problem where rewards are in $\mathbb{R}$, and let $V:\mathcal{S} \to \mathbb{R}$ be an approximate value function. Following the GAE paper, the TD-residual with discount $\gamma \in [0,1]$ is defined as $\delta_t^V = r_t + \gamma V(s_{t + 1}) - V(s_t)$.
I am confused by the formula for the GAE-$\lambda$ advantage estimator, which is
$$
\hat{A}_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l = 0}^\infty (\gamma \lambda)^l \delta_{t + l}^V.
$$
This seems to imply that $\delta_t^V$ is defined for $t > N$, where $N$ is the length of the current trajectory/episode. It looks like in implementations of this advantage estimator, it is just assumed that $\delta_t^V = 0$ for $t > N$, since the sums are finite. Is there a justification for this assumption? Or am I missing something here?


text:
How can I find the appropriate reward value for my reinforcement learning problem?
I am wondering how can I find the appropriate reward value for each specific problem. I know this is a highly empirical process, but I am sure that the value is not set totally at random. I want to know what are the general guidelines and practices to find the appropriate reward value for any reinforcement learning problem.

When you say "reward value", you mean the reward function (i.e. the function that gives the reward value for every action in some state), right?

Yes, but I am mostly having doubts about the ''value" I should give for rewarded states, for example, the reward of a goal in the final state of an episodic task. like in a maze what should be the reward value of the goal, +1, +50 or +100 or any other number? how this is determined?


text:
Why can the core reinforcement learning algorithms be applied to POMDPs?
Why can an AI, like AlphaStar, work in StarCraft, although the environment is only partially observable? As far as I know, there are no theoretical results on RL in the POMDP environment, but it appears the core RL techniques are being used in partially observable domains.

See this related but more specific question: Can Q-learning be used in a POMDP?.

It's not even clear if SC2 can be represented as any type of markov model, observable or not but the system still works.

I am not familiar with AlphaStar or StarCraft to explain the success of AlphaStar, but this is not the first time that a certain assumption doesn't probably hold in the real-world problem, but the applied method (that assumes that such an assumption holds) still performs decently or even well. For example, if I recall correctly, naive Bayes makes some assumptions that are just unrealistic, but, in practice, it still works in many cases. Why does it work? I don't know because I am not an expert on naive Bayes and the problems it's been applied to.

However, I can say that, in many cases, people try to make the environment an MDP (or they try to make the Markov property hold) by doing some tricks. For example, a typical trick in RL is to combine different successive frames of a video (or video game) in order to build a state, rather than using only one frame as the state.

Also, note that POMDPs are not exactly what you are thinking of. In POMDP, the agent doesn't know the state it is in, so it maintains a "belief" (i.e. a probability distribution) over the possible states. But your question is still very interesting and it is definitely legitimate!

Practically speaking, the agent doesn't know its state. The true state of the environment would display every piece but only areas within a radius of its units can it see. Based upon prior observations it could develop a belief about possible unit positions.....

Yes, maybe you're right. Even if you combine multiple frames, that may still not represent a state. It could just be an approximation of the true state.


text:
What are some good alternatives to U-Net for biomedical image segmentation?
Soon I will be working on biomedical image segmentation (microscopy images). There will be a small amount of data (a few dozens at best).
Is there a neural network, that can compete with U-Net, in this case?
I've spent the last few hours searching through scientific articles that are dealing with this topic, but haven't found a clear answer and I would like to know what other possibilities are. The best answers I found are that I could consider using ResU-Net (R2U-Net), SegNet, X-Net and backing techniques (article).
Any ideas (with evidence, not necessarily)?

Check out this paper.

If you want to detect anomaly on image, it would be a less layer neural network structure like the VGG16 (ref: https://towardsdatascience.com/anomaly-detection-in-images-777534980aeb)

Hey i am working on my Bachelor thesis at the moment and use UNET in combination with a GAN for image segmentation. I spend the last 5 months on that, so on my tests, the new approach of januar 2020, called Multires-UNET is quite a good choice for more texture orientated segmentation. I use the current github implementation. Its quite nice, maybe you notice that you can easyly tweak the number of parameters with "alpha" in the implementation, to scale the multiple Resnets in the Unet structure.
I tried also other segmentation networks like Mask_RCNN with different backbones or tried to construct various types of CAE on my own, but always had to come back to a UNET like structure. Same goes for ResU-NET and R2U, the multires one worked better for my purposes cause i didnt need any kind of LSTM modules.
Some examples which may clarify the difference in performance on a specific task:
Original Image:

Ground-Truth:

Classical UNET++ (Unet with skip-connections)(2.5 million parameters) more parameters (wide) didnt change the result.

Multires-UNET (alpha=1.67, think was about 7 million parameters)

Can you show some of ur microscopy images, how complex is ur task and what do you want to segment ?

Hi there, may I ask, what does "more textured orientated segmentation" mean? How do you decide that a picture you are performing segmentation on is more textured orientated? :)
Will certainly check it out. I can't show the images right now, after this coronavirus-situation I will get my hands on those

Hope the images give you some insigth, i trained it on about 500 handmade GT (used ImageJ for preprocessing, very useful tbh). Works way better for colored images, but Grayscale is a hard test for texture identification.

With texture orientated i basicly meant that it searches for deeper or more color/rotation/graysclae - INVARIANT features. So features which you cant fool that easy with some augmentation for example like color shift.

@PaulHigazi Is your first layer 572 by 572 if yes how large of your input image? How about evaluation matrix of your output against ground truth? Did you use Intersection of Union?


text:
Does Rice's theorem prove safe AI is undecidable?
According to Wikipedia

In computability theory, Rice's theorem states that all non-trivial,
  semantic properties of programs are undecidable. A semantic property
  is one about the program's behavior (for instance, does the program
  terminate for all inputs), unlike a syntactic property (for instance,
  does the program contain an if-then-else statement). A property is
  non-trivial if it is neither true for every computable function, nor
  false for every computable function.

A syntactic property asks a question about a computer program like "is there is a while loop?"
A semantic properties asks a question about the behavior of the computer program. For example, does the program loop forever (which is the Halting problem, which is undecidable, i.e., in general, there's no algorithm that can tell you if an arbitrarily given program halts or terminates for a given input)?
So, Rice's theorem proves all non-trivial semantic properties are undecidable (including whether or not the program loops forever).
AI is a computer program (or computer programs). These program(s), like all computer programs, can be modeled by a Turing machine (Church-Turing thesis).
Is safety (for Turing machines, including AI) a non-trivial semantic question? If so, is AI safety undecidable? In other words, can we determine whether an AI program (or agent) is safe or not?
I believe that this doesn't require formally defining safety.

I think an answer to your question may actually depend on the definition of "safe AI". For example, if you define a safe AI as any program that can run on a TM that does not contain a while loop (of course, this is a stupid example), then "AI safety" is a syntactic property.

nbro♦: Is there any way to prove AI safety if "safe" is not a syntactical definition?

I would need to think about it. I had never thought about it.

Every program halts, or continues

Given N steps, enough time and space (*), halting within N steps is provable

3 (from 2). Halting always has proof: run program until halt; count number of steps; verify claim of halting (within number of steps)

(Program is safe) implies (program is proved safe)

(Safety proved) implies (public understands the proof)

(Program is safe) implies (program always halts (safely) or continues (safely)) [from 1, 6] and ((public) understands (safety proof)) [from 7]

(Public not understand claimed safety proof this moment) implies (don't run program this moment) [common sense]

(*) This universe is finite. Some numbers too big to be computed in this universe
Have you seen perfect software?
Have you seen software make mistakes?
Why trust life-death decisions to software?
Why trust government decisions to software?
Why trust business decisions to software?
If scientists may not recognize AI is intelligent, what if you don't recognize something beyond AI in front of you?
(After enough doubt, all you can do is TRUST)

authors: Σωτήρης Κούτρας

authors: Апостол Тонев


text:
What is the difference between using dense layers as opposed to convolutional layers in my networks when dealing with images?
I am thinking about developing a GAN.
What is the difference between using dense layers as opposed to convolutional layers in my networks when dealing with images?

In general, conv layers are able to learn spatial relationships more efficiently and to a better degree. The specifics are more complicated.

Hi and welcome to AI SE! This question is indeed a bit unclear. You mention the GAN, but why is it relevant? Are you interested in 1. the structural differences between dense layers and convolutional layers, 2. the functional difference or, 3. both the structural and functional differences? Also, are you interested in some particular context? Again, why do you mention the GAN? What is your goal and why do you want to know the "difference" between the two?


text:
Is it normal to have the root mean squared error greater on the test dataset than on the training dataset?
I am new to deep learning.
I am training a model and I am getting a root mean squared error (RMSE) greater on the test dataset than on the training dataset.

What could be the reason behind this? Is this acceptable to get the RMSE greater in test data?

I think you should provide more details such as 1. your model architecture (I assume you are using a neural network) and 2. a plot of the evolution of the error and performance (on both the test and training data) throughout the epochs.

It is common to have root mean squared error (RMSE) greater on the test dataset than on the training dataset (this is equal to having accuracy/score higher for model in training dataset than test dataset). This normally happens because the training data are assesed on the same data that have been learnt before, while the test dataset may have data that are unknown / not common that may give more errors or misclassification when doing prediction.
But if your model shows your test dataset have way too high RMSE result rather than your training dataset RMSE result, it may indicates that overfitting happens.
If overfitting happens, there are a lot of reasons this could happen.
Referenced from https://elitedatascience.com/overfitting-in-machine-learning, some factors that causes overfitting are:

Complexity of data (e.g. there are irrelevant input features). This can be solved with removing irrelevant input features.
Not enough training data. This can be solved by training with more data (Eventhough this may not always succeed. Sometimes it may give noise towards data), etc.

I concur with the comment from @Angela Marpaung. You will always are going to have a higher RMSE in testing than training because testing hasn't been seen by the model.
Remember models tend to memorize the answer so showing new data to the model makes them struggle to find the answer in the figurative sense.
If you have a way disparate higher value of RMSE in testing that may indicate overfitting, but looking at the scale of your values in training/testing for RMSE it looks like the model can generalize. In this situation your model is a generalized model.
Hope this helps.

RMSE stands for Root Mean Squared Error. As the name suggests, it is calculated by taking the square root over the mean of the squared errors of individual points.
It is normal for the test error to be higher than the train error and in most cases, the test error will be greater than the train error.

I am training a model and i am getting test results greater than train results.

You don't give us too many details, but most probably it's underfitting.

What could be the reason behind this?

Underfitting is often a result of an excessively simple model.
Too much regularization techniques were used.

Is this acceptable to get the RMSE greater in test data?

Yes, but that indicates a problem with model, so you should be aware of the consequences.

The training error (on any error metrics, not only for RMSE) will usually be less than the test error because the same data used to fit the model is employed to assess its training error. In other words, a fitted model usually adapts to the training data and hence its training error will be overly optimistic (too small). In fact, it is often the case that the training error steadily decreases as the size of the model increases.


text:
What are the AI technologies currently used to fight the coronavirus pandemic?
The ongoing coronavirus pandemic of coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), as of 29 September 2020, has affected many countries and territories, with more than 33.4 million cases of COVID-19 have been reported and more than 1 million people have died. The live statistics can be found at https://www.worldometers.info/coronavirus/ or in the World Health Organization (WHO) site. Although countries have already started quarantines and have adopted extreme countermeasures (such as closing restaurants or forbidding events with multiple people), the numbers of cases and deaths will probably still increase in the next weeks.
Given that this pandemic concerns all of us, including people interested in AI, such as myself, it may be useful to share information about the possible current applications of AI to slow down the spread of SARS-CoV-2, to help infected people or people in the healthcare sector that have been uninterruptedly working for hours to attempt to save more lives, while putting at risk their own.
What are the existing AI technologies (e.g. computer vision or robotics tools) that are already being used to tackle these issues, such as slowing down the spread of SARS-CoV-2 or helping infected people?
I am looking for references that prove that the mentioned technologies are really being used. I am not looking for potential AI technologies (i.e. research work) that could potentially be helpful. Furthermore, I am not looking for data analysis tools (e.g. sites that show the evolution of the spread of coronavirus, etc.)

Related at Kaggle: https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge and possibly a good source of information for anyone answering

Here are other possibly useful resources Open Source Toolkit for Predicting Covid-19 Vulnerability and Resources and channels to help with COVID-19 research.

In the United Kingdom a major difficulty has been with the allocation of people requiring a test for COVID-19 to a testing station which has tests available. Some cases have been reported where people have been told to travel several hundred miles to be tested. The optimal allocation of people in real-time to available resources constrained to minimise travel time would appear to be ideal for an AI based optimisation technique such as multi objective genetic algorithms. An added problem would be to predict the likely future resource needs as the infection rises and wanes, and feed that in to o

According to the Baidu Research's blog post How Baidu is harnessing the power of AI in the battle against coronavirus (12-03-2020), there are already some artificial intelligence tools or algorithms being used to fight the coronavirus.
Given that I cannot confirm that these AI tools and algorithms I will mention are really being used in practice, I will only quote the parts of the blog post that potentially answer my original question.
To give some context, similar to HIV viruses, the virus that is causing the coronavirus pandemic, SARS-CoV-2 is capable of rapidly mutating, making vaccine development and virus analysis difficult.
AI-powered and non-contact infrared sensor system

Baidu has developed several tools that are effective in building awareness and screening populations, including an AI-powered, non-contact infrared sensor system that provides users with fast multi-person temperature monitoring that can quickly detect a person if they are suspected of having a fever, one of the many symptoms of the coronavirus. This technology is currently being used in Beijing's Qinghe Railway Station to identify passengers who are potentially infected where it can examine up to 200 people in one minute without disrupting passenger flow.

AI-powered pneumonia screening and lesion detection system

By leveraging PaddlePaddle and the semantic segmentation toolkit PaddleSeg, LinkingMed has developed an AI-powered pneumonia screening and lesion detection system, putting it into use in the hospital affiliated with XiangNan University in Hunan Province. The system can pinpoint the disease in less than one minute, with a detection accuracy of 92% and a recall rate of 97% on test data sets.

Automated HealthMap system

The Boston Children's Hospital used an automated HealthMap system that scans online news and social media reports for early warning signs of outbreaks, which led to the initial awareness that COVID-19 was spreading outside China.

Autonomous vehicles carry out non-contact tasks

Access to health care and resources at a moment's notice is vital for battling the spread of the coronavirus. Autonomous vehicles are playing a useful role in providing access to necessary commodities for health-care professionals and the public alike by delivering goods in infected areas and disinfecting hospitals, effectively minimizing person-to-person transmission and alleviating the shortage of medical staff.
Apollo, Baidu's autonomous vehicle platform, partnered with a local self-driving startup called Neolix to deliver supplies and food to the Beijing Haidian Hospital.

Not really something that slows it down but currently Mauricio Santillana at Harvard is working on modeling the pandemic and has shared some of his approaches. He explained that they have used google search trends to try to predict the number of actual cases (there is a delay between people being sick and getting tested). Looking for search terms like, "how to use an inhaler?" can reveal areas affected by the outbreak and is useful for modeling.
Paper can be found here
Towards Data Science has several articles listing potential AI applications for helping in the fight against COVID
Including:

Identify who is most at risk
Diagnose patients
Develop drugs faster
Predict the spread of the disease
Understand viruses better
Map where viruses come from
Predict the next pandemic

How to fight COVID-19 with machine learning
Though there are probably many creative applications that AI can help with I think most of them are modeling related at the moment.

I asked "What are the existing AI technologies that are already being used to tackle these issues", so I wasn't interested in "research work", but actual stuff that is useful. I had also found some research work.

What are the AI technologies currently used to fight the coronavirus pandemic?

You don't define what is AI for you. Let's suppose it means advanced informatics (but perhaps also abstract interpretation).
In France we have polling websites like https://covidnet.fr/ ... Does that count as AI?
And we also have Deep Tech related Covid19 specific call for proposals, e.g. here ... does that count as AI?
Does the French StopCovid platform count as AI? It is an open source digital project (using Bluetooth) to trace potential Covid-infected people. The current political debate is involving intense discussions about its legitimacy and deployement.
At last, development of Covid specific ventilators (e.g. this open hardware / open software project) involves a lot of embedded software which could be analyzed by static analyzers such as Frama-C ? Does that count as AI ?
My employer CEA (a French scaled down equivalent to US DoE) is an applied research institution participating to several Covid related projects.
And several French research institutions (e.g. INSERM, CNRS, INRIA, LIP6, etc...) are participating to Covid related research programmes. Also, major health organizations like AP-HP.
So is the European Union which is partly funding several of them. Most of the Covid related research projects have strong digital aspects. In particular, related to genome decoding and bioinformatics.
PS. My GitHub helpcovid ongoing project (a free software web application related to Covid19)  does not claim to be AI, but certainly is a digital application. See also these slides about RefPerSys (which is an ongoing free software ambitious AI project).

By "AI" I didn't mean websites that just keep track of the statistics (that's not even advanced informatics), otherwise, I would have mentioned the sites I am already mentioning as one of the tools. By "AI tools" I mean computer vision, robotics, machine learning, etc., tools. I was also not asking for research work (I know there's already some research work). I was asking for AI-based technology that is already being used to fight the coronavirus.

authors: Дана Ибрагимова


text:
Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?
I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation.

As per my knowledge, my derivation is correct for $dZ, dW, db$ and $dA$, but, if I refer to Andrew Ng Coursera stuff, then I am seeing an extra $\frac{1}{m}$ for $dW$ and $db$, whereas no $\frac{1}{m}$ in $dZ$. Andrew's slides on the left represent derivative and whereas the right side of slides shows NumPy implementation corresponding to the right side equation.

Can someone please explain why there is:
1) $\frac{1}{m}$ in $dW^{[2]}$ and $db^{[2]}$ in Andrew's slides in NumPy representation
2) missing $\frac{1}{m}$ for $dZ^{[2]}$ in Andrew's slides in both normal and NumPy representation.
Am I missing something or doing it in the wrong way?

TL;DR: This has to do with the way A. Ng has defined back propagation for the course.
Left Column
This is only with respect to one input example and so the $\frac{1}{m}$ factor reduces to 1 and can be omitted. He uses lower case to represent one input example (eg a vector $dz$) and upper case with respect to a (mini-)batch (eg a matrix $dZ$).
The $\frac{1}{m}$ factors in $dW,db$
In this definition of backprop, he "defers" multiplying by the $\frac{1}{m}$ factor until $dW,db$ rather than "absorbing" it into $dZ^{[2]}$. That is, the $dZ^{[2]}$ term is defined in a way that it does not have $\frac{1}{m}$.
Observe, if you move the $\frac{1}{m}$ factor to be in the definition of $dZ^{[2]}$ and remove it from the definitions of $dW,db$ you will still come out with the same values for all $dW,db$.
Speculation
This "deferred" multiplication might have to do with numerical stability. Or simply a stylistic choice made by A. Ng. This might also prevent one from "accidentally" multiplying by $\frac{1}{m}$ more than once.

Thanks for clearing the confusion. Can you please let me know where I can read this numerical stability thing and why following my approach (list in image 1 ) causes numerical stability ?

@user110244 I was only speculating as to why A. Ng presents back prop in this way. To better understand numerical stability I'd recommend searching around on the math stack exchange - surely there will be an answer better than any I could provide here.


text:
How to predict an event (or action) based on a window of time-series measurements?
I have an input vector $X$, which contains a series of measurements within a period, e.g. 100 measurements in 1 sec. The goal is to predict an event, let's say, moving forward, backward or static.
I don't want to predict the output just by looking at one series of measurements, but by looking at a window of $n$ vectors $X$ of measurements, making it dependant on the previous measurements, because of the noise in the measurements.
Is there a way RNN can help me with this? Many to one architecture? LSTM?
CNN of 1D + LSTM + dense?

Hi and welcome to AI SE. So, to clarify, you want to feed to the model at once $n$ vectors $X_1, X_2, \dots, X_n$, each of which contains e.g. $M=100$ measurements, i.e. $|X_i| = 100$, for all $i$?

Hi, basically I want to predict the class based on these X vector with 100 measuments, but also depend on the previous X inputs I fed. Is there a model which takes into account previous states? Or do I need to feed 'n' number of X in a certain window of states? I heard about RNN and Lstm.

Yes, RNN may be helpful for your problem. Please, edit your post to answer my questions above. Also, don't just ask "Is there a way RNN can help me with this?", because people could say either "yes" or "no". You could ask "How specifically could I design an LSTM to solve this problem?" What should the input of this LSTM be? What should be the output? And so on. These are more specific questions.


text:
What does the statement with the max do in the recursive best-first search algorithm?
I am reading the book "Artificial Intelligence: A Modern Approach" by Stuart and Norvig. I'm having trouble understanding a step of the recursive best-first search (RBFS) algorithm.
Here's the pseudocode.

What does the line s.f <- max(s.g + s.h, node.f) do?
Here's a diagram of the execution of RBFS.

You don't understand only the line s.f <- max(s.g + s.h, node.f) or all the algorithm?

Only the specific line.

I've edited your post to make it clearer then. Make sure that's still consistent with your original intent.

It is computing the cost for the successors, it will be max of the current incurred cost (node.f) or sum of s.g and s.h (as far as I remember, it is the edge cost+estimate cost to goal, I am not sure, the book might have used a similar notation before)

This is probably more easily understood as the collapse/restore macro. The idea is that the previously explored state was collapsed and only the minimum f-cost from the sub-tree was stored. This represents the best unexpanded state in the subtree that was collapsed.
When restoring the portion of the collapsed tree, the f-cost of the restored node could either be the original f-cost (g+h), or it could be the stored f-cost if it is larger. By taking the max, the code ensures that states that are restored maintain at least the cost of the previously best unexpanded state. (If the g+h cost is larger, then we know the state wasn't previously expanded and it wasn't previously the state on the fringe with the minimum edge cost.)
The linked paper gives several examples where similar ideas are used during search.

If the current paths is better than the previous one. E.g.  heruistic cost is lower. Got it! Thanks.


text:
How can I get to a final output of shape $224 \times 224$, without FC layers, from a tensor of specific shape, in OpenPose?
I am approaching the implementation of the OpenPose algorithm for realtime human body pose estimation.
According to the official paper OpenPose: Realtime Multi-Person 2D Pose
Estimation using Part Affinity Fields, $L$ and $S$ fields (body part maps, and part affinity fields) are estimated. These have the same size as the input image, and, according to the paper, these fields should be outputted at a given step in the forward pass (after a given number of $L$ stages, and $S$ stages), but, since before entering these stages the image is passed through the initial layers of the VGG-19 model, the spatial dimension is encoded and the features that finally enter the $L$ and $S$ stages have other dimensionality.
All the network is convolutional, there's no FC layer at all. The VGG-19 part is the only one that contains MaxPooling layers, hence affecting the spatial relations and size of the receptive fields.
My point is, after stage execution, I get tensors of shape [batch_size, filter_number, 28, 28]. The issue is that the paper is not stating how to decode this information into the $L$ and $S$ maps of size $224 \times 224$.
Following a traditional approach and decoding the final tensors with a linear net from, let's say, $15000 \rightarrow (224 * 224 * \text{ number of body parts }) + (224 * 224 * \text{ number of limbs } * 2)$~A very huge number!, is out of question for any domestic computer, I presume I should have the least 128GbRAM installed, and is not the case.
Another solution is to remove the max-pooling layers from the VGG-19 part, but then although the map size is preserved to $224$, instead of $28$, the huge amount of computations and values that need to be stored also lead to memory errors.
So, the problem is, how can I get to a final output of $224 \times 224$ without FC layers, from a tensor of shape [batch_size, bodyparts, 28, 28]?
Not an easy answer. I will check a TensorFlow implementation I have seen around to see how the problem was solved.
Any parallel ideas are greatly welcome.

Sorry nbro, but that looked more to my english teacher corrections than a ML related answer. What was so necessary about your corrections?. I am not a native english speaker, but the overall meaning and sense of my question, is fully understandable I guess.

BTW I think in other approaches the output of the S and L stages is upsampled to 224px to make it compatible with annotated images. So what I do is to downsample the annotations to the spatial dimensions of the output filters, (28,28) for the Loss function (MSE) to work. Now it seems to run correctly and there's no need to flatten or decode the results to compare results, which seems the right path to me. Any ideas?

Please, ping me if you want me to see your messages, i.e. use @nbro. I didn't edit your post to offend you, but e.g. to improve the title so that it is more informative. I am sorry if you felt offended or upset. I was just trying to further improve your post. Unfortunately, I am not familiar with OpenPose, so I cannot help you now. I also didn't fully understand your description, probably because I am not familiar with OpenPose. Eventually, you could also ask this question at https://stats.stackexchange.com or https://datascience.stackexchange.com.

I didn't feel offended at all. I am aware my english expressivity is not fully accurate. Need to increase my lr. :D... no worries I fully understand...


text:
Are there principled ways of tuning a neural network in case of overfitting and underfitting?
Whenever I tune my neural network, I usually take the common approach of defining some layers with some neurons.

If it overfits, I reduce the layers, neurons, add dropout, utilize regularisation.
If it underfits, I do the other way around.

But it sometimes feels illogical doing all these. So, is there a more principled way of tuning a neural network (i.e. find the optimal number of layers, neurons, etc., in a principled and mathematical sound way), in case it overfits or underfits?


text:
What is the difference between evolutionary computation and evolutionary algorithms?
A book on evolutionary computation by De Jong mentions both the term evolutionary algorithms (EA) as well as evolutionary computation (EC). However, it remains unclear to me what the difference between the two is. According to Vikhar, EA forms a subset of EC. However, it remains unclear to me what sort of topics/algorithms would be considered EC but not EA. Is there a clear difference between the two? If so, what is this difference?

As you can find on Wikipedia:

Evolutionary algorithms form a subset of evolutionary computation in that they generally only involve techniques implementing mechanisms inspired by biological evolution such as reproduction, mutation, recombination, natural selection, and survival of the fittest.

This means that other types of evolutions, which are not necessarily a biological evolution, are found in evolutionary computation but not in evolutionary algorithms. For example, learning classifier systems are in EC as they are evolutionary, but not completely in EA as they are not biological.

Thanks for your input, good point that it is indeed mentioned that EA use biological evolution. However; I am not quite convinced by your example. In the paper that introduced Grammatical Evolution, the authors classify it as an EA: "We present grammatical evolution, an evolutionary algorithm that can evolve complete programs". Source: https://pdfs.semanticscholar.org/469e/bfd13737504d1ae223b01140d2ddbc318e14.pdf . Are there in general other forms of evolution that are not biological? I'm sorry if I'm being overly pedantic here, I'm just trying to get my head wrapped around this.

@DaniëlWillemsen no problem. You're right. LCS can be a better example. It's a rule-based algorithm and has some differences with EAs.


text:
What are the guidelines for defining a reward function in reinforcement learning (bandit problem)?
I'm working currently on a problem and I'm using RL (bandit problem).
In my system, I have an agent that chooses an action among $k$ possible actions, and a user that decides whether the agent chooses the right action or not. If the user is satisfied with the decision made by the agent, he rewards with $+1$, otherwise $-1$.
Is this is a good reward function, knowing that in my problem the values are in the range $[0, 1]$?
Are there any guidelines to follow for defining the reward function? Are there any references (books or articles) that tackle this problem and present a solution?

Why would you let a user decide if the action is good or not? if you have a classical bandit problem where each arm returns a value between 2 values. And you are saying these 2 values are 0 and 1. Why not return this value the arm returns as reward?

I need the user to be part of the system, it's the same thing to say that the arm return a value or the user send the value.

No, because now you need the user to decide on every action. You should automate the reward function so you can run through thousands of training steps in seconds.

For the training, I automated it but when the system will be deployed the user will be involved.

And how did you automate it for training?

each arm returns 1 or -1


text:
What are the pros and cons of supervised, semi-supervised and unsupervised relation extraction in NLP?
I am following the NLP course taught by Dan Jurafsky. In the video lectures Supervised Relation Extraction and Semi Supervised and Unsupervised Relation Extraction Jurafsky explains supervised, semi-supervised and unsupervised relation extraction.
But what are the pros and cons of every relation extraction method compared with the other two relation extraction methods?

Supervised
Pros:

highest accuracy

Cons:

need a large human-labeled training set
brittle (doesn't work well with examples that are in a different genre from the training set)

Semi-supervised
Relation bootstrapping
Pros:

only requires a small set of labeled data (seed relations)

Cons:

complex iterative process

Distant supervision
Pros:

training happens in one go (no iterative process)

Cons:

requires a big database of relations

Unsupervised
Pros:

don't need any labeled data

Cons:

need to process a huge quantity of unlabeled data (usually web crawling)


text:
Why are conics important in computer vision?
The book Multiple View Geometry in Computer Vision by Richard Hartley and Andrew Zisserman talks about lines, points and conics. A conic is a curve described by a second-degree equation in the plane, so a parabola would be an example of a conic. The purpose and usage of points and lines in computer vision are quite clear. For example, a set of points defines an object, or we can project a 3-dimensional point to a 2-dimensional point in the image plane, or a line represents the space to look for the corresponding point in the second plane of another point in the first plane (epipolar geometry). However, probably because I haven't yet read the part of the book related to the applications of conics in computer vision, it's not clear why do we even care about conics in computer vision.
So, why are conics important in computer vision? Note that I know that conics are defined by points and, given the point-line duality, they can also be defined by lines, but this doesn't still enlightens me on the purpose of conics in computer vision. So, I am looking for applications where conics are used to define the underlying CV model, in a similar way that points and lines are used to describe the pinhole camera model.


text:
What is the difference between "out-of-distribution (generalisation)" and "(meta)-transfer learning"?
I'm trying to develop a better understanding of the concept of "out-of-distribution" (generalization) in the context of Bengio's "Moving from System 1 DL to System 2 DL" and the concept of "(meta)-transfer learning" in general.
These concepts seem to be very strongly related, maybe even almost referring to the same thing. So, what are similarities and differences between these two concepts? Do these expressions refer to the same thing? If the concepts are to be differentiated from each other, what differentiates the one concept from the other and how do the concepts relate?

Hi and welcome to AI SE! What is "Moving from System 1 DL to System 2 DL"? Can you provide a link to the resource that talks about "out-of-distribution" in the context of "Moving from System 1 DL to System 2 DL"


text:
Do deeper residual networks perform better or worse?
If you have an $18$ layer residual network versus and a $32$ layer residual network, why would the former do better at object detection than the latter, if you have both models are training using the same training data?

Just by having more parameters, the deeper model has a higher capacity than the smaller one. This means that theoretically it can learn to extract more complex features from the data. Additionally, more layers means that the model can extract even higher-level features from the data.
So, generally speaking, deeper models will most of the times outperform shallow ones for more difficult tasks.
The downside is that if you have a small amount of data, a high-capacity model has the ability of memorizing the training set, which would lead to overfitting.
Besides performance, deeper models require better hardware and larger training times. So, there are plenty of reasons for one to prefer a more shallow model to a deeper one.


text:
Why are reinforcement learning methods sample inefficient?
Reinforcement learning methods are considered to be extremely sample inefficient.
For example, in a recent DeepMind paper by Hessel et al., they showed that in order to reach human-level performance on an Atari game running at 60 frames per second they needed to run 18 million frames, which corresponds to 83 hours of play experience. For an Atari game that most humans pick up within a few minutes, this is a lot of time.
What makes DQN, DDPG, and others, so sample inefficient?

I will try to give a broad answer, if it's not helpful I'll remove it.
When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model of the environment.
In general I would say that there are two issues related to sample efficiency:
1 the size of the 'action'+'environment states' space 2 the exploration strategy used.
Regarding the first point, in reinforcement learning is really easy to encounter situations in which the number of combinations of possible actions and possible environment states explode, becoming intractable. Lets for example consider the Atari game from the Rainbow paper you linked: the environment in which the agent operate in this case is composed of rgb images of size (210, 160, 3). This means that the agent 'see' a vector of size 100800. The actions that an agent can take are simply modifications of this vector, e.g. I can move to the left a character, slightly changing the whole picture. Despite the fact that in lot of games the number of possible actions is rather small, we must keep in mind that there are also other objects in the environment which change position as well. What other object/enemies do obviously influence the choice of the best action to perform in the next time step. To a high number of possible combinations between actions and environment states is associated a high number of observations/interaction required to learn a good model of the environment. Up to now, what people usually do is to compress the information of the environment (for example by resizing and converting the pictures to grayscale), to reduce the total number of possible states to observe. DQL itself is based on the idea of using neural networks to compress the information gathered from the environment in a dense representation of fixed size.
For what concern the exploration strategy, we can again divide the issue in subcategories: 1 how do we explore the environment 2 how much information do we get from each exploration. Exploration is usually tuned  through the greedy hyper-parameter. Once in a while we let the agent perform a random action, to avoid to get stuck in suboptimal policies (like not moving at all to avoid to fall into a trap, eventually thanks to a greedy action the agent will try to jump and learn that it gives a higher reward). Exploration comes with the cost of more simulations to perform, so people quickly realise that we can't rely only on more exploration to train better policies. One way to boost performance is to leverage not only the present iteration but also past interactions as well, this approach is called experience replay. The underline idea is to update the q-values depending also on weighted past rewards, stored in a memory buffer. Other approaches point to computation efficiency rather than decreasing the amount of simulations. An old proposed technique that follow this direction is prioritised sweeping Moore et al. 1993, in which big changes in q values are prioritised, i.e. q-values that are stable over iterations are basically ignored (this is a really crude way to put it, I have to admit that I still have to grasp this concept properly). Both this techniques were actually applied in the Rainbow paper.
On a more personal level (just pure opinions of mine from here) I would say that the problem between RL agents and humans is the fact that we (humans) have lot of common sense knowledge we can leverage, and somehow we are able, through cognitive heuristics and shortcuts, to pay attention to what is relevant without even being aware of it. RL agents learn to interact with an environment without any prior knowledge, they just learn some probability distribution through trial and errors, and if something completely new happens they have no ability at all to pick up an action based on external knowledge.
One interesting future direction in my opinion is reward modelling, described in this video: https://youtu.be/PYylPRX6z4Q
I particularly like the emphasis on the fact that the only true thing that human are good at is judging. We don't know how to design proper reward functions, because again, most of the actions we perform in real life are driven by reward of which we are not aware, but we are able in a glimpse to say if an agent is performing a task in a proper way or not. Combining this 'judging power' into RL exploration seems to be a really powerful way to increase sample efficiency in RL.

This is mostly because humans already have information when they start learning the game (priors) that makes them learn it more quickly. We already know to jump on monsters or avoid them or to get gold looking object.
When you remove these priors you can see a human is worse at learning these games. (link)
Some experiments they tried in the study to remove these priors where replacing all notable objects with colored squares, reversing and scrambling controls, changing gravity, and generally replacing all sprites with random pixels. All these experiments made human learning much harder, and increased death rate, time needed and states visited in the game.
If we want the reinforcement learning algorithm to perform as good as a human, we will have to somehow include these priors we have as a human before training the network. This of course is not yet done (as far as I know)


text:
Interpretation of inverse matrix in mean calculation in Gaussian Process
The formula for mean prediction using Gaussian Process is $k(x_*, x)k(x, x)^{-1}y$, where $k$ is the covariance function. See e.g. equation 2.23 (in chapter 2) from Gaussian Processes for Machine Learning (2006) by C. E. Rasmussen & C. K. I. Williams.
Oversimplifying, the mean prediction of the new point $y_*$ is the weighted average of previously observed $y$, where the weights are calculated by the $k(x_*,x)$ and normalized by $k(x,x)^{-1}$.
Now, the first part $k(x_*, x)$ is easy to interpret. The closer the new data point lies to the previously observed data points, the greater their similarity, the higher will be the weight and impact on the prediction.
But how to interpret the second part $k(x, x)^{-1}$? I presume this makes the weight of the points in the clusters greater than the outliers. Am I correct?

Mathematical Interpretation
Note that equation (2.23) is simply calculating the conditional distribution of equation (2.21) and then finding the mean. Your question reduces to:
"Given normal variables $X$ and $Y$, why is $\mathbb{E}[Y|X] = \mu_y + Cov(X, Y)Cov(Y, Y)^{-1}(x - \mu_x)$? (note: in the book, $\mu_x = \mu_y = 0$)
Deriving the conditional probability mean is complicated (see The Bivariate Normal Distribution, page 3). A more intuitive look can be seen in the first graph in this page.
Here, the mean of $Y|X$ is linear in what value $X$ takes. The line starts at the intercept $\mu_y$, and increases with slope $\rho (\frac{\sigma_y}{\sigma_x}) = \frac{Cov(X,Y)}{\sigma_x \sigma_y}\left(\frac{\sigma_y}{\sigma_x}\right)=\frac{Cov(X,Y)}{\sigma_x^2} = \frac{Cov(X,Y)}{Var(x)} = \frac{Cov(X,Y)}{Cov(X,X)}$. So the mathematical interpretation of $Cov(X,Y)Cov(X,X)^{-1}$ is that it is the slope of the relationship between the mean of $Y|X$ and the value of $x$ that you are given. As you are given a higher value $x$, say $x + \delta$, then the mean of $Y|X$ raises by $Cov(X,Y)Cov(X,X)^{-1}\delta$.
Why is there even a $Cov(X,Y)Cov(X,X)^{-1}$ term there? For some reason, multiplying $Y|X$ by a $Cov(X,Y)Cov(X,X)^{-1}$ term makes $Y|X$ completely independent of $X$ (which makes sense as the definition of "conditional probability", because you are already given a value of $X$). This is just a mathematical property, I don't know if there's an intuitive explanation as to why.
Human Interpretation
In case your post just want an intuition as to why there is a $Cov(X,X)^{-1}$ in the prediction of a Gaussian process (and ignoring the conditional probability fluff), I don't think there's a real basis for this, it would only be coincidental as the authors simply used the conditional probability mean formula, but I would guess $Cov(X,X)^{-1}$ somehow normalizes the values of covariance matrix $Cov(X_*, X)$.
For example, if the training set $X$ has a lot of outliers and therefore extremely high variance (e.g. all non-diagonal entries in millions), then it is very likely that $Cov(X_*, X)$ would also be extremely high as $X_*$ follows the same distribution as $X$ (unless each data in $X_*$ matches the exact same variance in $X$). It doesn't make sense to multiply $y$ by millions though, as $y$ is already a somewhat decent estimator/prior.
It makes more sense to normalize $Cov(X_*, X)$ by dividing it with the training data variance $Cov(X,X) = Var(X)$ so that the ratio $Cov(X_*, X)Cov(X, X)^{-1}$ tends to be more closer to 1 when $X_*$ follows the same distribution as $X$ (which should be the case). If the ratio is exactly 1, then $X_*$ has the exact same distribution as $X$, so you just return the prior estimate $y$. If the ratio is far away from $1$, then the test set distribution is wildly different than the training set distribution, so you return a number far away from $y$.

I interpret this as the following, I will use the uppercase matrix notation $\mathbf{K_{*}}$, etc...
The covariance matrix $\mathbf{K_{xx}}$ summarizes everything we know about the input feature space. I think of it as a unique signature that describes the data we have in $\mathbb{R}^{d_x}$. Along with the example training data, we have the labels $\mathbf{y}$, which give us a concrete definition of what the prediction values for $\mathbf{K_{xx}}$ should be.
We can then ask the question "What can we multiply the unique signature $\mathbf{K_{xx}}$ by in order to get the training outputs?" This takes the familiar form of $\mathbf{A}\mathbf{x} = \mathbf{b}$...
$$
\begin{aligned}
\mathbf{K}_{xx}\mathbf{z} &= \mathbf{y} \\
\mathbf{z} &= \mathbf{K}_{xx}^{-1}\mathbf{y}
\end{aligned}
$$
$\mathbf{z}$ then represents the vector that the covariance matrix transforms into the outputs. Then all we have to do is multiply the new covariance of the train/test points by that same vector to get the predictions...
$$
\mathbf{\hat{y}} = \mathbf{K_{*x}}\mathbf{K_{xx}^{-1}}\mathbf{y}
$$
I should add that I am relatively certain about this, but I am still learning this stuff myself so I am not 100


text:
Are there any deep learning techniques to use the content of an image for another image?
Are there any machine learning or deep learning techniques to use the content of an image for another image?
More specifically, suppose I take a photo of a notebook. I get the angle, lighting, and perspective perfect. Now I copy an image I found online that contains text or handwriting. Is there a machine learning technique that would now draw this writing in my notebook?
Just asking if it's possible before I attempt to hire someone.

What do you mean by "draw this writing on my notebook"? Do you mean a computer physically drawing/painting an image onto a piece of paper?

Is your goal only to repoduce "flat" images of text/drawings/photos in situ in placeholder images with existing or blank pages? What is the scale of number of placeholder images (the perfect shots of notebooks) and number of content images (the text/handwriting)?


text:
Are these two TRPO objective functions equivalent?
In the TRPO paper, the objective to maximize is (equation 14)
$$
\mathbb{E}_{s\sim\rho_{\theta_\text{old}},a\sim q}\left[\frac{\pi_\theta(a|s)}{q(a|s)} Q_{\theta_\text{old}}(s,a) \right]
$$
which involves an expectation over states sampled with some density $\rho$, itself defined as
$$
\rho_\pi(s) = P(s_0 = s)+\gamma P(s_1=s) + \gamma^2 P(s_2=s) + \dots
$$
This seems to suggest that later timesteps should be sampled less often than earlier timesteps, or equivalently sampling states uniformly in trajectories but adding an importance sampling term $\gamma^t$.
However, the usual implementations simply use batches made of truncated or concatenated trajectories, without any reference to the location of the timesteps in the trajectory.
This is similar to what can be seen in the PPO paper, which transforms the above objective into (equation 3)
$$
\mathbb{E}_t \left[ \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_\text{old}}(a_t|s_t)} \hat A_t \right]
$$
It seems that something is missing in going from $\mathbb{E}_{s\sim \rho}$ to $\mathbb{E}_t$ in the discounted setting.
Are they really equivalent?

As you point out, they are not equivalent. I guess you could store the time index for each state visited, but there are two problems with this.
First, if you sample states according to their time index, sampling from the replay memory will become more cumbersome and probably much slower (you'd have to sample the time index and then a specific state with that time index). This is definitely something undesirable. If you choose to add the importance sampling term, then you will make really small most of the terms if $\gamma$ is not so close to 1.
Second, while the original objective is very nice to obtain theoretical results, you may ask yourself if that objective is really what you want to maximize. Do you really care more about the performance at the beginning of the trajectory than at the end?
While I have no rigorous proofs, I like to think that a better definition for the expected discounted reward is a time average:
$$\eta(\pi) = \lim_{N\to\infty}\frac{1}{N}\sum_{k=1}^N\mathbb E_{s_k,a_k,s_{k+1},\cdots}\left[\sum_{t=k}^\infty\gamma^{t-k}r(s_t)\right].$$
Since each term of the average satisfies the original proofs, the only difference with the equation 14 is that the density is $\tilde\rho_{\theta_{old}}(s) = \lim_{N\to\infty}\frac{1}{N}\sum_{k=1}^N\rho_{k,\theta_{old}}(s)$, where $$\rho_{k,\theta_{old}} = P(s_k = s)+\gamma P(s_{k+1}=s) + \gamma^2 P(s_{k+2}=s) + \dots.$$
You can notice that for a large $N$ you basically give the same importance to all of the time-steps. So, the usual implementations actually optimize some complicated function similar to the one I defined, where you give a similar weight to all samples, irrespective of their corresponding time-step.


text:
How can I combine the readings of multiple lidars into 1 point cloud?
I have a car with 8 lidars, each with a field of view of 60 degrees. My car looks like this:

How can I merge all the lidar readings into 1 point cloud?

Hi Josh. I think you should clarify how this problem is related to AI.

A high-level solution is to use the NVIDIA DriveWorks SDK.  NVIDIA's Developer Blog has a good post, "Point Cloud Processing with NVIDIA DriveWorks SDK".
The NVIDIA DriveWorks SDK contains a collection of CUDA-based low level point cloud processing modules optimized for NVIDIA DRIVE AGX platforms. The DriveWorks Point Cloud Processing modules include common algorithms that any AV developer working with point cloud representations would need, such as accumulation and registration.
I believe you don't need the AGX hardware for the problem you posted.


text:
In image classification, why do we usually minimize a cost function rather than maximizing it?
I was watching a video about policy gradients by Andrej Karpathy. At 10:00, it shows an equation for supervised learning for image classification.
$$\max\sum _{i} \log p(y_i \mid x_i)$$
I have worked with image classification models before, but I always minimized a cost function (aka loss function). I have also never seen someone maximizing a cost function for image classification in the wild.

So, what are the advantages of a minimizing loss function over a maximizing loss function in image classification?

Other than RL, which problems do we solve by maximizing a cost function?

There is really no difference between minimizing a cost function and maximizing a value function.  One can be the reciprocal of the other, or the negative of the other, for example.

It is worth noting that the function that the OP gives is in fact the negative of the usual multi-class loss function. I would hazard a guess that Andrej used it to make a more direct comparison with RL.


text:
What are sim2sim, sim2real and real2real?
Recently, I always hear about the terms sim2sim, sim2real and real2real. Will anyone explain the meaning/motivation of these terms (in DL/RL research community)?
What are the challenges in this research area?
Anything intuitive would be appreciated!

The abbreviations sim2sim, sim2real and real2real refer to techniques that can be used to transfer knowledge from one environment (e.g. in simulation) to another one (e.g. in the real world).

sim2sim stands for simulation-to-simulation,
sim2real stands for simulation-to-real, and
real2real stands for real-to-real.

In sim2sim, knowledge acquired during one simulation is transferred to an agent (or robot) in another simulation. Similarly, in sim2real, knowledge acquired during the simulation is used in a real-world problem (or environment). Finally, in real2real, knowledge acquired in a real-world problem can be transferred to another agent in another real-world problem.
The main challenges are related to the differences that exist between one environment and the other (either in simulation or in the real-world). For example, in sim2real, the simulation is almost never a perfect model of the real-world environment, so an agent trained in a simulation will probably not behave optimally in the real-world environment, which is often a lot more complex than the simulated environment. However, it is often the case that a robot needs to be trained in simulation, given that a robot trained in a real-world environment is subject to crashes.

I defined real2real based on the concepts of sim2sim and sim2real as used in Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via
Randomized-to-Canonical Adaptation Networks (2019) by Stephen James et al., given that I've not found any paper that uses the expression real2real.


text:
How do you game an automatic trading system by messing with data, as opposed to hacking the algorithm itself?
There was a recent question on adversarial AI applications, which led me to start digging around.
Here my interest is not general, but specific:
How do you game an automatic trading system by messing with data, as opposed to hacking the algorithm itself?

I think these questions are interesting, but it seems to me that you're asking slightly different questions (even though related), given that financial markets and automated trading systems are not necessarily the same, even though my knowledge of the field is like zero. I would focus on one of these questions and maybe ask other sub-questions, but you should clarify how these sub-questions related to the main one. Anyway, I think your main question should be also in the title.

@nbro I wanted a flashy title for the question, but the idea is that, the method of exploitation I'm asking about involves manipulating the data set (in this case, the financial markets) to mislead automated trading systems.  Thus "hacking the financial markets", with the caveat that it involves automation.

I don't have a proper source for this, as i've only read this from an online forum: everything i say is just hearsay and I am very uneducated on the subject. With that being said...
As you may know algorithmic trading relies on strategies, i.e. I trade a certain way once I see certain indexes move in a certain way. If your procedure is known by other people for some reason, then other adverse agents can release information or influence the market to manipulate your bot into making trades that the adverse agent then takes advantage over.
I'm not sure how this works in detail, and frankly I find it quite unlikely that a single adverse agent can target a single known strategy with the entire market in-between the two. But I've heard that a process known as the alpha algorithm can be used to discover hidden trading strategies. But again, I wasn't able to find any articles that directly linked this method towards financial trading. So my usual warning of me talking out my butt again applies.
Just a fun thought. A friend of mine told me (unfortunately, again I wasn't able to find a proper source for this) that Jane Street found that the most effective trading strategies were based on relatively simple ML techniques (I recall the specific example mentioned was just a common form of regression). The important follow question would be "What are their input metrics", I don't think I was told this, but I would believe that it would also just be something computed by relatively well-founded stock market indices. If this is the case, than again it would be very difficult for a single adverse agent to influence that particular bot since the force of the entire market is in between them.
On the opposite end, we have this redditor(finally, a source!), who amazingly analyzed about 130+ trading strategies, each of which were "big brained" in kind. This redditor concluded that all papers were essentially p-hacked, and none of the results were significant. Of course there is the possibility that authors with profitable results would not publish them, but the conclusion remains: keep your strategies simple. Again, this implies that it would be harder to "hack" a specific strategy used in the market by a particular individual.
P.S. There are other "big brained" methods that are kept well secret and thus difficult to analyze: I heard a rumour that a Chinese company paid people to take pictures inside shopping malls, to use to analyze shopping behavior, and how vibrant the economy is; I also hear that some people use satellites photos to look at, say, car dealerships, to see how much of their stuff is being purchased. Again its unclear if these techniques are profitable or not, but certainly a lot of money is being thrown into it.

actually i think this question is better asked at the quant.SE :)

You make a great fundamental point about robustness of simple strategies.  PS I specifically asked over here because it involves automated trading, a form of AI

i agree that it could be viewed as a form of AI, but this specific instance might be better studied over there.


text:
Neural networks for sports betting
I want to design a neural network that can be used for predicting sports scores for betting, specifically for American football.  What I’d like to do is create a kind of profile for each game based on the specific strengths and weaknesses of each team.
For example, let’s say two teams have the following characteristics:
Team A:

Passing Offense Rating: 5
Rushing Offense Rating: 2

Team B:

Passing Defense Rating: 3
Rushing Defense Rating: 4

I’d like to be able to search for historical games where two teams have similar profiles.  I could perhaps then narrow it down to games with profiles that have statistically significant historical outcomes (i.e., certain types of matchups are likely to result in similar results).
In reality, I’d have dozens of team characteristics to compare.  I would then need to assign weights of importance to each characteristic, which could be used to further ensure the effective selection of similar games.
I think I could do this like a convolutional neural network where there is an additional filter applied to the characteristics for the weights.
Are there any other ways that are specifically applicable to this strategy?

This is probably not going to work well as a way to make money. People with far larger budgets, and far more training, are already milking out any money to be made this way. This is probably their day job, and they are good at it.
That said, here are some ideas:

You do not need or want to use a convolutional network for this. Convolutional networks are useful when you want to detect a pattern invariant to translation in complex inputs. The only translation you have is that the teams could appear in either order. Just input them in both possible orders if this is a concern.
You want to find similar games. You do not need or (generally) want to use a neural network to do this. As a starting point, normalize the features and compute the nearest neighbors of an input point directly. Algorithms for doing this are fast and will return all similar points. You can even (with small modifications) output the degree of similarity as a numeric value.
If you want to predict who will win, you will probably have much better luck doing that directly. Build a feed-forward neural network that predicts which team won from the features you gathered. This is a routine classification task, and the resulting model will probably work better than finding similar games and then trying to manually determine what to do.
If you care about scores instead of just who won, use a feed forward network for regression instead of classification. The task is almost identical.

A CNN is probably the incorrect term since the filter is an entire game and the data being filtered is all games in the training set, but it’s easier to visualize that way.  Creating an arbitrary methodology to determine whether a past game is sufficiently similar isn’t hard - the crux here is finding the correct weights for each team category.  I know that I could use a feedforward network to estimate scores, but I like the idea of finding some number of similar games and looking at summary statistics.

@SuperCodeBrah It sounds to me like you want to do regression with a feed-forward network.

If that could work, then sure, but it seems hard to get a grasp of any nonlinearity that exists in the universe of potential matchups, so I’m trying to turn it into more of a categorization problem.  You correctly surmised that I’m new to this.  What I’m saying is that I don’t have a good sense of how exactly a feedforward network should be structured for this type of potentially nonlinear problem.  I could come up with something really messy for a nearest neighbor type of solution, but I’m wondering if there’s an existing methodology that’s cleaner.

@SuperCodeBrah I'm not completely sure I've understood what you want, but it sounds like maybe you are after something like dimensionality reduction? That is, you'd like to come up with weightings of the features that allow you to make simple comparisons of games in a principled way. Maybe you want PCA? https://en.wikipedia.org/wiki/Principal_component_analysis

I'm not looking for dimensionality reduction exactly. If I understand correctly, the relative importance of each input gets lost in PCA (i.e., it only ranks attributes). Assuming I've already normalized each characteristic (let's say 1-100), I could calculate a cumulative distance between a given game and each game in the historical set, however, for football, that would lead to me treating passing offense equally with something like kickoff return average, which would obviously be wrong. So the solution would be to further normalize each characteristic by its importance to outcome.

Okay, assuming I could correctly structure a NN as a non-linear classifier for this problem (I very much struggle with the intuition for how to tailor NNs for a given problem other than maximum layers, nodes, and connections), I think the answer is to train the NN and then extract the weights of the connections between the input and first hidden layer, and use those to calculate cumulative distances between each characteristic of the game in question and each game in the historical set - ||G1 - H1|| * W1 + ||G2 - H2|| * W2... There's probably a better way, but I think that would work.

One other point - the advantage to doing it this way is that I'm left with not just a prediction model but a means to come up with a rough estimation of statistical significance of past outcomes relative to the spread on a game, i.e., 1) the greater the difference between the average of historical outcomes and the current spread and 2) the lower the variation on those outcomes, the more confident I can be in the pick.  So I wouldn't just bet on every  game - I'd have a filtering mechanism.  There's obviously more to think about and I could be flat out wrong, but that seems like a start.

@SuperCodeBrah You are mistaken about PCA: it produces a weighting for each input, such that a linear combination of weighted inputs yields a good approximation of the "shape" of the original dataset. I'll also recommend against trying to interpret the weights in the first layer of a NN as statistically meaningful. It really sounds like you want logistic or linear regression if that's your goal. Good luck!

Okay, all good stuff, thanks.  I think the secret sauce is in determining how to rank each team for each characteristic.  If I wanted to get really crazy, I might be able to train on raw stats rather than team rankings.  Sounds like I just need to really play around with it and see what sticks.


text:
Is it possible to use the GPT-2 model for time-series data prediction?
Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?

Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not the architecture, and I don't think it would transfer properly to time series.

GPT-2 is a decoder, not an encoder.

In TS tasks, Transformers can capture long-range dependencies effectively through their self-attention mechanism, which can potentially lead to better forecasting performance compared to LSTMs. However, they are computationally more expensive than LSTMs.

It's worth noting that their self-attention mechanism is quadratic in time and space with regard to the input length. This may be intractable for very long sequences (>> 1k). However, solutions to this problem have already been proposed, such as the Sparse Transformer.