text:
Is an autoencoder model encoder-only or encoder-decoder?
I'm writing up about different model architectures used in NLP, namely encoder-only models, encoder-decoder-only models, and have come across what seems to be a naming inconsistency. For decoder-only models it seems that they can be referred to as autoregressive models since they autoregressively predict the next token and encoder-decoder models can be referred to as sequence-to-sequence since they map a sequence of tokens to a different sequence of tokens.
However, I found various sources conflicting each other as to what an autoencoder is. While many sources claim that encoder-only models are autoencoders, others also refer to encoder-decoder models as being autoencoders.

You will find that the nomenclature in this field is not particularly rigorous. One of the main reasons I visit this site is to keep up with the new names professors (and others) are using for existing things.
That said, an autoencoder is comprised of an encoder, a latent space (some type of information choke point), and a decoder. Based on this and given the options you mention, a strong argument can be made that it is an encoder-decoder model, however it is not a sequence-to-sequence model at all. It is simply attempting to reconstruct the input, not predict a future value, token, or anything else.

Based on your explanation, you're indicating that an encoder-decoder model is not a sequence-to-sequence model. However, when you consider for instance machine translation which is a sequence-to-sequence task, this is often modelled using encoder-decoder models.

LSTMs and RNNs tend to be sequence to sequence encoder-decoder models. My point is that Autoencoders are encoder-decoder models but they are not used for sequence to sequence tasks.


text:
A question about the Wolpertinger algorithm (Deep RL in Large Discrete Action Spaces paper)
I am trying to reproduce the recommender task experiment from this paper.
The paper suggests to embed discrete actions into continuous action space and then to use the proposed Wolpertinger agent.
The Wolpertinger agent is as follows:
DDPG produces so called proto action $f(s)$, then KNN finds k nearest embeddings of discrete actions to this proto action, and then we choose the one of these $k$ embeddings, which has the highest Q-function value. The whole is a full policy, $\pi(\cdot)$.
While training we optimize the critic using only the full policy (DDPG + a choice of a neighbour).
The actor is optimized using proto action output in order to be differentiable, $Q(s, f_{\theta}(s)) \rightarrow \max_{\theta}$.
The problem is that the critic does not know that it is used to optimize continuous output of the algorithm. It is trained only to value the embedded actions. As I understand, we hope that continuity of critic will help us with it, but what I have is that proto actions constantly appear to be in some corners with no real actions and where the Q-function unreasonably has greater values (because it is simply untrained in such domains). The DDPG output is normalized to match the embeddings bounds to make these empty spaces not so large.
It seems for me that there is a way to make embeddings more appropriate for the task and achieve higher reward. However, when I use $k = |
\mathcal{A}|$, proto actions are not considered and algorithm works pretty well.
Usually I use $|\mathcal{A}| = 100$ and $k = 10$.
I have trained them with skip-gram, based on the users history.
Below are 2d projections of my embeddings to the first 10 axes (embeddings are from $\mathbb{R}^{20}$). And undependently of the state, proto actions are about the same. The blue is a proto action for the some fixed state.
Having some state fixed, $Q(s, f(s))$ value is always higher than $Q(s, a)$ for any $a \in \mathcal{A}$.

Would be glad to get any help, especially the help of people familiar with this algorithm.
Do I need to make embeddings fill proto actions range (some hyperrectangle in case we have the tanh activation it the actor)? What is the way to fill such a domain with embeddings?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Most of state-action pairs remain unvisited in the q-table
In building my first Q-learning algorithm for OpenAI gym's CartPole problem, many of my states remain unvisited. I believe it is the reason that my agent does not learn.
Can I be told of the reasons I can look into why that may happen? I have read and seen tutorials and I know a lot has already been done for this problem. My goal here is to learn and hence this simple implementation with Q-learning.
PS. The specific question to my problem is asked here.
PSS. As an edit, I am inserting my whole code in the following.
import numpy as np
import gym
import matplotlib.pyplot as plt

env = gym.make('MountainCar-v0')

EPISODES = 5000
LEARNING_RATE = 0.1
SHOW_AFTER = 1500
DISCOUNT_FACTOR = 0.95
EPSILON = 0.1
NUMBER_OF_BINS = 10
OBSERVATION_SPACE = 2
MAX_STATES = 100 # e.g. 23 means obs one is 2, obs two is 3

'''This function breaks the continuous states into discrete form'''
def digitize_states():
    bins = np.zeros((OBSERVATION_SPACE,NUMBER_OF_BINS))
    bins[0] = np.linspace(-1.2, 0.6, NUMBER_OF_BINS)
    bins[1] = np.linspace(-.07, 0.07, NUMBER_OF_BINS)
    return bins

'''This function assign the observations into discrete bins using
   digitize function and the bins that we created using digitize_states()
'''
def assign_obs_to_bins(obs, bins):
    states = np.zeros((OBSERVATION_SPACE))
    states[0] = np.digitize(obs[0], bins[0])
    states[1] = np.digitize(obs[1], bins[1])
    return states

'''This function merely make the states in form of the strings so that we can
   later use those strings (i.e. number of states) as the KEYs in our q-table
   dictionary.
'''
def get_all_states_as_strings():
    states = []
    for i in range(MAX_STATES):
        states.append(str(i).zfill(OBSERVATION_SPACE))
    return states

'''Convert the current state into string so that it can be used as key for dictionary '''
def current_state_to_string(state):
    current_state = ''.join(str(int(e)) for e in state)
    return current_state

'''This function iniquation the q-table to zeros'''
def initialize_q():
    states = get_all_states_as_strings()
    q = {}
    for state in states:
        q[state] = {}
        for action in range(env.action_space.n):
            q[state][action] = 0
    return q

def initialize_Q():
    Q = {}

    all_states = get_all_states_as_strings()
    for state in all_states:
        Q[state] = {}
        for action in range(env.action_space.n):
            Q[state][action] = np.random.uniform(-.5, 0.5, 1)
    return Q

'''This function returns the maximum Q-value from Q-table'''
def return_max_from_dict(dict_var, action = None):
    '''    Arguments
    # dict_var: Dictionary variable, which represent the q-table.

    # Return
    # max_key: Best Action
    # max_val: max q-value for the current state, taking best action
    '''
    if(action == None):
        max_val = float('-Inf')
        for key, val in dict_var.items():
            if val > max_val:
                max_val = val
                max_key = key
        return max_key, max_val
    else:
        return dict_var[action]

'''Main code starts here'''

bins = digitize_states()
all_states = get_all_states_as_strings()

q = initialize_Q()

Total_reward_matrix = []
_testing_action_matrix = []
_testing_state_matrix = []
_testing_states = []
_testing_random = 0
_testing_greedy = 0

for episode in range(EPISODES):

    done = False
    cnt = 0

    # Reset the observations -> then assign them to bins
    obs = env.reset()

    if episode%SHOW_AFTER == 0:
        print(episode)

    total_reward = 0

    while not done:
        current_state = current_state_to_string(assign_obs_to_bins(obs, bins))
        _testing_state_matrix.append(int(current_state))

        if np.random.uniform() < EPSILON:
            act = env.action_space.sample()
            best_q_value = return_max_from_dict(q[current_state], action = act)
            _testing_random+=1
        else:
            act, best_q_value = return_max_from_dict(q[current_state])
            _testing_greedy+=1

        obs, reward, done, _  = env.step(act)
        _testing_action_matrix.append(act)

        q[current_state][act] = (1-LEARNING_RATE)*q[current_state][act] + LEARNING_RATE * (reward + DISCOUNT_FACTOR * best_q_value)
        cnt+=1
        total_reward += reward

        if done and cnt > 200:
            print(f'reached at episode: {episode} in count {cnt}')
            Total_reward_matrix.append(total_reward)
        elif done:
#             print('Failed to reach flag in episode ', episode)
            Total_reward_matrix.append(total_reward)

env.close()

Could you please add: How have you discretised the state? What approach are you using for exploration?

I am using epsilon-greedy method. The states are divided into 10 bins (later I tried 100 bins as well) each. They are discretised using standard Pythong function with the help of number of bins.

Thanks. I think we need more than that. When you say the states "are divided into 10 bins" I want to see in more detail how you are doing that - are there 10,000 states here (10 in each dimension of position and velocity for cart and pole?), and what are you treating as extremes for each one? IIRC CartPole reports incorrectly +-Inf for one or more of these dimensions which would be a problem if you use it as-is. Also, what values of epsilon are you using? Pleae use [edit] to add to your question

In fact this answer by me might answer your original question and this one: https://ai.stackexchange.com/questions/21872/why-do-my-rewards-fall-using-tabular-q-learning-as-i-perform-more-episodes/21878#21878 - it  could be that you are hitting the same problem.

I have edited my question. the problem I am facing a similar problem with the CatPole as well. There is something very seriously wrong that I am doing, and I cannot put my finger on that. I have seen my code so many times that I have lost the count and could not find anything wrong in the logic and algorithm (following straight from the book of Sutton and Barta)

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning
I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0.
However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) of more than 30. My question is, is it not possible to get successful completion of Cartpole without using sophisticated algorithms such as Deep learning etc.?
I am glad to share my code, but I am sure no one would have time to check it.

PS. I know there are many implementations out there, but I have learned from them but I want to implement my own code for learning purpose and do not just want to copy-paste.
PSS (Edit): I have added the code in the answer to this question for reference.

The problem has continuous states. Probably the most critical part of your implementation using tabular Q learning is how you have approximated these states into discrete values. Could you add an explanation for that?

I discretize them into 50 discrete blocks as below
X_position = np.linspace(-2.4, 2.4, 50)
Velocity = np.linspace(-5, 5, 50)
Angle = np.linspace(-0.7295476, 0.7295476, 50)
Angular_vel = np.linspace(-5,5,50)

Then fitting any continuous state value into the respective discrete block isn't hard. For example if X is continuous state for location, then int(np.digitize(X, X_position)) can do the continuous to discrete conversion.

OK, so you have done that for all 4 state variables, giving a discrete state space size of ~ 6 million? Is your choice of 50 a hyper-parameter that you can easily change in your code?

Yes. It's a hyper-parameter that can be changed easily. For reference, I have added the code in the answer ( adding code to the question would make it difficult to read and understand).

The code to my question is as below, for reference:
import gym
import numpy as np
import matplotlib.pyplot as plt

# Discretize the contiuous space
DISCRETE_POINTS = 50
X_position = np.linspace(-2.4, 2.4, DISCRETE_POINTS)
Velocity = np.linspace(-5, 5, DISCRETE_POINTS)
Angle = np.linspace(-0.7295476, 0.7295476, DISCRETE_POINTS)
Angular_vel = np.linspace(-5,5,DISCRETE_POINTS)

# Fit the instantnous state into any discrete box
def get_state(_state):

    X, X_bar, Angle_, Angle_bar = _state

    X = int(np.digitize(X, X_position))
    X_bar = int(np.digitize(X_bar, Velocity))
    Angle_ = int(np.digitize(Angle_, Angle))
    Angle_bar = int(np.digitize(Angle_bar, Angular_vel))

    return (X, X_bar, Angle_, Angle_bar)

def epsilon_greedy_action(s, epsilon):

    '''   Input argument: state 's' tuple in the form (4,0,1,0)    '''
    # if np.random.uniform() < epsilon:
    if np.random.random() > epsilon:
        a = env.action_space.sample()
    else:
        _,a = find_maxQ_value(s)

    return a

def find_maxQ_value(state):
    '''
    Input argument:
    state: should be a tuple of form (0,0,0,0) or (1,0,0,1) etc.

    Output argument:
    best_value: best q-value of the current state-action pair
    choosen_action: best action corresponding to current state. It depends on the best q-value
    '''

    for act_ in range(env.action_space.n):
        A = [Q[state,0], Q[state, 1]]
        best_value = np.max(A)
        choosen_action = np.argmax(A)

    return best_value, choosen_action

def plotRunningAverage(totalrewards, N, n_avg):

    running_avg = np.empty(N)
    for t in range(N):
        running_avg[t] = np.mean(totalrewards[max(0, t-N):(t+1)])
    return running_avg

if __name__ == '__main__':

    env = gym.make('CartPole-v0');

    EPISODES = 1000;
    no_actions = env.action_space.n

    # Hyper parameters
    alpha = 0.001 # Learning rate
    gamma = 0.99 #Discount Factor
    epsilon = 1 # For Epsilon-Greedy algorithm
    epsilon_decay_factor = 0.99;
    min_epsilon = 0.1;

    states = []
    for i in range(len(X_position)):
        for j in range(len(Velocity)):
            for k in range(len(Angle)):
                for l in range(len(Angular_vel)):
                    states.append((i,j,k,l))

    #Initialize Q-table :
    # 1. We make the Q-table in form of a dictionary
    # 2. We initialize Q-table values as zero in this
    Q = {}
    for s in states:
        for n_a in range(no_actions):
            Q[s, n_a] = 0

    Running_reward = [];
    l_action_cnt = 0;
    r_action_cnt = 0;
    wrong_action =0;

    #Q-Learning agent episodes
    for e in range(EPISODES):

        cn_state = env.reset()
        ds_state = get_state(cn_state)

        done = False
        ep_reward = 0
        ep_len = 0

        while not done:

            action = epsilon_greedy_action(ds_state, epsilon)
            if action == 0:
                l_action_cnt+=1
            elif action == 1:
                r_action_cnt+=1;
            else:
                wrong_action+=1;

            cn_next_state, reward , done , ep_len = env.step(action)
            ep_reward += reward

            ds_next_state = get_state(cn_next_state)

            # Update the Q-table based on the action
            Val_Q_bar, _ = find_maxQ_value(ds_next_state);

            Q[ds_state, action] = (1-alpha)*Q[ds_state, action] + alpha*(reward + gamma*Val_Q_bar)

            ds_state = ds_next_state;

        Running_reward.append(ep_reward)

        if e%100 == 0:
            print('Episode : {}, Episode reward: {}, Epsilon: {}'.format(e, ep_reward, epsilon))

        if epsilon >= min_epsilon:
            epsilon*=epsilon_decay_factor;

plt.plot(Running_reward);
plt.xlabel('episodes')
plt.ylabel('episodic reward')
plt.grid('ON')
running_avg = plotRunningAverage(Running_reward, EPISODES, 50)
plt.plot(running_avg);
plt.legend(['Episodic rewards', '50-Episode moving-average reward'])

print('The ratio of left to right action is : {}'.format(l_action_cnt/r_action_cnt))


text:
How does an episode end in OpenAI Gym's "MountainCar-v0" environment?
I am working on OpenAI's "MountainCar-v0" environment.  In this environment, each step that an agent takes returns (among other values) the variable named done of type boolean. The variable gets a True value when the episode ends. However, I am not sure how each episode ends. My initial understanding was that an episode should end when the car reaches the flagpost. However, that is not the case.
What are the states/actions under which the episode terminates in this environment?

The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the _max_episode_steps attribute of the environment.

To answer your question, the specifics of some of the OpenAI Gym environments can be found on their wiki:

The episode ends when you reach 0.5 position, or if 200 iterations are reached.

There is a deeper question in what you asked, though:

My initial understanding was that an episode should end when the Car reaches the flagpost.

The environment certainly could be set up that way. Limiting the number of steps per episode has the immediate benefit of forcing the agent to reach the goal state in a fixed amount of time, which often results in a speedier trajectory by the agent (MountainCar-v0 further penalizes long trajectories through the reward signal). Also, the underlying learning algorithm may only perform policy updates after completion of the episode. If the agent will never reach the goal state under its current policy (i.e. the policy is very bad, lacks much randomness, etc.), then terminating the episode after a fixed amount of time will ensure that the agent is able to perform a policy update and try a new policy on the next episode (alternatively, the learning algorithm could perform a policy update during the episode). There is a branch of tasks called continuing tasks which never terminate (see Section 3.3 of Sutton and Barto), so the choice to limit the number of steps per episode is heavily dependent on the task at hand and the choice of learning algorithm.


text:
Number of parameters in Keras/Tensorflow Dense layers
I am a bit confused about how the number of parameters are calculated in Dense model for the Kera/Tensorflow.
For example, in the figure below I thought that both the statements were the same, but I found a different number of parameters for both.  In particular, I am talking about model.add(Dense(...)) command.

Check the documentation for Dense layer:

Note: If the input to the layer has a rank greater than 2, then Dense computes the dot product between the inputs and the kernel along the last axis of the inputs and axis 1 of the kernel (using tf.tensordot). For example, if input has dimensions (batch_size, d0, d1), then we create a kernel with shape (d1, units), and the kernel operates along axis 2 of the input, on every sub-tensor of shape (1, 1, d1) (there are batch_size * d0 such sub-tensors). The output in this case will have shape (batch_size, d0, units).

That is what happening in your first case - for input dimensions (4,1) you've got d0=4 and d1=1. So it creates a kernel of shape (1,32) that gets applied along the axis of dimension 4. That's why your output shape is (4,32) and you've got 32 weights + 32 biases = 64 parameters.
In second case you've got a "standard" 32 * 4 fully-connected weight matrix + 32 biases = 160.


text:
How to deal with a moving target in the Lunar Lander environment with DDPG?
I have noticed that DDPG does rather well at solving environments with a static target.
For example, the default of Lunar Lander, the flags do not change position.  So the DDPG model learns how to get to the center of the screen and land fairly quickly.
As soon as I start moving the landing position around randomly and adding the landing position as an input to the model, the model has an extremely hard time putting this connection together.
A few questions/points about adding this complexity to the environment:

Would more nodes per layer or more layers help in figuring out this connection?  I have tested this but seems the bigger I go, the harder it is to learn anything.
Is it a common RL AI issue that it has a hard time connecting data?
I realize that I could change the environment to always have a static target and instead change the position of the lunar lander ship, which in effect accomplishes the same thing, but want to know if we could solve it with moving target
Is there any good documentation on Actor/Critic analyzing models? I have some results where my critic target is falling out but my critic loss is going down nicely. At the same time my actor target is going up and up and eventually plateau.  It is hard to really understand what is happening and would be great to understand actor loss vs critic loss vs critic target.

Essentially, I added a random int (left side of flags), added 1 to get x position middle of landing and add 1 more to get distance of flags to be 3 out of 11 chunks.
rand_chunk = random.randint(0, CHUNKS-3)
self.x_pos_middle_landing = chunk_x[rand_chunk + 1]
self.helipad_x1 = chunk_x[rand_chunk]
self.helipad_x2 = chunk_x[rand_chunk + 2]
height[rand_chunk] = self.helipad_y
height[rand_chunk + 1] = self.helipad_y
height[rand_chunk + 2] = self.helipad_y

Old State:
state = [
        (pos.x - VIEWPORT_W/SCALE/2) / (VIEWPORT_W/SCALE/2),
        (pos.y - (self.helipad_y+LEG_DOWN/SCALE)) / (VIEWPORT_H/SCALE/2),
        vel.x*(VIEWPORT_W/SCALE/2)/FPS,
        vel.y*(VIEWPORT_H/SCALE/2)/FPS,
        self.lander.angle,
        20.0*self.lander.angularVelocity/FPS,
        1.0 if self.legs[0].ground_contact else 0.0,
        1.0 if self.legs[1].ground_contact else 0.0
        ]

Added to State, same as state[0] but using middle of 3 for landing
(self.x_pos_middle_landing - VIEWPORT_W/SCALE/2) / (VIEWPORT_W/SCALE/2)

And update the obs space from 8 spaces to 9
self.observation_space = spaces.Box(-np.inf, np.inf, shape=(9,), dtype=np.float32)

Rewards need to be updated, Old Rewards:
shaping = \
        - 100*np.sqrt(state[0]*state[0] + state[1]*state[1]) \
        - 100*np.sqrt(state[2]*state[2] + state[3]*state[3]) \
        - 100*abs(state[4]) + 10*state[6] + 10*state[7]  # And ten points for legs contact, the idea is if you

New Rewards:
shaping = \
        - 100*np.sqrt((state[0]-state[8])*(state[0]-state[8]) + state[1]*state[1]) \
        - 100*np.sqrt(state[2]*state[2] + state[3]*state[3]) \
        - 100*abs(state[4]) + 10*state[6] + 10*state[7]  # And ten points for legs contact, the idea is if you

DDPG Model
    import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import gym
from tensorflow.keras.models import load_model
import os
import envs
import time
import scipy.stats as stats
# from stable_baselines.common.policies import MlpPolicy, MlpLstmPolicy
from stable_baselines.common.vec_env import SubprocVecEnv, DummyVecEnv, VecCheckNan
from stable_baselines.common import set_global_seeds, make_vec_env
from stable_baselines.ddpg import DDPG
from stable_baselines.ddpg.policies import MlpPolicy
# from stable_baselines.sac.policies import MlpPolicy
from stable_baselines import PPO2, SAC
from stable_baselines.common.noise import NormalActionNoise, OrnsteinUhlenbeckActionNoise, AdaptiveParamNoiseSpec

if __name__ == '__main__':
    num_cpu = 1  # Number of processes to use
    env = SubprocVecEnv([make_env(env_id, i) for i in range(num_cpu)])
    env = VecCheckNan(env, raise_exception=True, check_inf=True)
    n_actions = env.action_space.shape[-1]

    #### DDPG
    policy_kwargs = dict(act_fun=tf.nn.sigmoid, layers=[512, 512, 512], layer_norm=False)
    param_noise = None
    # action_noise = OrnsteinUhlenbeckActionNoise(mean=np.zeros(n_actions), sigma=float(0.5) * np.ones(n_actions))
    action_noise = NormalActionNoise(0, 0.1)
    model = DDPG

    # # Train Model
    model.learn(total_timesteps=int(3e5))
    model.save('./models/lunar_lander')

Full Code:
    """
Rocket trajectory optimization is a classic topic in Optimal Control.

According to Pontryagin's maximum principle it's optimal to fire engine full throttle or
turn it off. That's the reason this environment is OK to have discreet actions (engine on or off).

The landing pad is always at coordinates (0,0). The coordinates are the first two numbers in the state vector.
Reward for moving from the top of the screen to the landing pad and zero speed is about 100..140 points.
If the lander moves away from the landing pad it loses reward. The episode finishes if the lander crashes or
comes to rest, receiving an additional -100 or +100 points. Each leg with ground contact is +10 points.
Firing the main engine is -0.3 points each frame. Firing the side engine is -0.03 points each frame.
Solved is 200 points.

Landing outside the landing pad is possible. Fuel is infinite, so an agent can learn to fly and then land
on its first attempt. Please see the source code for details.

To see a heuristic landing, run:

python gym/envs/box2d/lunar_lander.py

To play yourself, run:

python examples/agents/keyboard_agent.py LunarLander-v2

Created by Oleg Klimov. Licensed on the same terms as the rest of OpenAI Gym.
"""

import sys, math
import numpy as np
import random

import Box2D
from Box2D.b2 import (edgeShape, circleShape, fixtureDef, polygonShape, revoluteJointDef, contactListener)

import gym
from gym import spaces
from gym.utils import seeding, EzPickle

FPS = 50
SCALE = 30.0   # affects how fast-paced the game is, forces should be adjusted as well

MAIN_ENGINE_POWER = 13.0
SIDE_ENGINE_POWER = 0.6

INITIAL_RANDOM = 1000.0   # Set 1500 to make game harder

LANDER_POLY =[
    (-14, +17), (-17, 0), (-17 ,-10),
    (+17, -10), (+17, 0), (+14, +17)
    ]
LEG_AWAY = 20
LEG_DOWN = 18
LEG_W, LEG_H = 2, 8
LEG_SPRING_TORQUE = 40

SIDE_ENGINE_HEIGHT = 14.0
SIDE_ENGINE_AWAY = 12.0

VIEWPORT_W = 600
VIEWPORT_H = 400

class ContactDetector(contactListener):
    def __init__(self, env):
        contactListener.__init__(self)
        self.env = env

    def BeginContact(self, contact):
        if self.env.lander == contact.fixtureA.body or self.env.lander == contact.fixtureB.body:
            self.env.game_over = True
        for i in range(2):
            if self.env.legs[i] in [contact.fixtureA.body, contact.fixtureB.body]:
                self.env.legs[i].ground_contact = True

    def EndContact(self, contact):
        for i in range(2):
            if self.env.legs[i] in [contact.fixtureA.body, contact.fixtureB.body]:
                self.env.legs[i].ground_contact = False

class LunarLander(gym.Env, EzPickle):
    metadata = {
        'render.modes': ['human', 'rgb_array'],
        'video.frames_per_second' : FPS
    }

    continuous = False

    def __init__(self):
        EzPickle.__init__(self)
        self.seed()
        self.viewer = None

        self.world = Box2D.b2World()
        self.moon = None
        self.lander = None
        self.particles = []

        self.prev_reward = None

        # useful range is -1 .. +1, but spikes can be higher
        self.observation_space = spaces.Box(-np.inf, np.inf, shape=(9,), dtype=np.float32)

        if self.continuous:
            # Action is two floats [main engine, left-right engines].
            # Main engine: -1..0 off, 0..+1 throttle from 50% to 100% power. Engine can't work with less than 50% power.
            # Left-right:  -1.0..-0.5 fire left engine, +0.5..+1.0 fire right engine, -0.5..0.5 off
            self.action_space = spaces.Box(-1, +1, (2,), dtype=np.float32)
        else:
            # Nop, fire left engine, main engine, right engine
            self.action_space = spaces.Discrete(4)

        self.reset()

    def seed(self, seed=None):
        self.np_random, seed = seeding.np_random(seed)
        return [seed]

    def _destroy(self):
        if not self.moon: return
        self.world.contactListener = None
        self._clean_particles(True)
        self.world.DestroyBody(self.moon)
        self.moon = None
        self.world.DestroyBody(self.lander)
        self.lander = None
        self.world.DestroyBody(self.legs[0])
        self.world.DestroyBody(self.legs[1])

    def reset(self):
        self._destroy()
        self.world.contactListener_keepref = ContactDetector(self)
        self.world.contactListener = self.world.contactListener_keepref
        self.game_over = False
        self.prev_shaping = None

        W = VIEWPORT_W/SCALE
        H = VIEWPORT_H/SCALE

        # terrain
        CHUNKS = 11
        height = self.np_random.uniform(0, H/2, size=(CHUNKS+1,))
        chunk_x = [W/(CHUNKS-1)*i for i in range(CHUNKS)]
        rand_chunk = random.randint(0, CHUNKS-3)
        self.x_pos_middle_landing = chunk_x[rand_chunk + 1]
        self.helipad_x1 = chunk_x[rand_chunk]
        self.helipad_x2 = chunk_x[rand_chunk + 2]
        self.helipad_y = H/4
        height[rand_chunk] = self.helipad_y
        height[rand_chunk + 1] = self.helipad_y
        height[rand_chunk + 2] = self.helipad_y

        self.moon = self.world.CreateStaticBody(shapes=edgeShape(vertices=[(0, 0), (W, 0)]))
        self.sky_polys = []
        for i in range(CHUNKS-1):
            p1 = (chunk_x[i], height[i])
            p2 = (chunk_x[i+1], height[i+1])
            self.moon.CreateEdgeFixture(
                vertices=[p1,p2],
                density=0,
                friction=0.1)
            self.sky_polys.append([p1, p2, (p2[0], H), (p1[0], H)])

        self.moon.color1 = (0.0, 0.0, 0.0)
        self.moon.color2 = (0.0, 0.0, 0.0)

        initial_y = VIEWPORT_H/SCALE
        self.lander = self.world.CreateDynamicBody(
            position=(VIEWPORT_W/SCALE/2, initial_y),
            angle=0.0,
            fixtures = fixtureDef(
                shape=polygonShape(vertices=[(x/SCALE, y/SCALE) for x, y in LANDER_POLY]),
                density=5.0,
                friction=0.1,
                categoryBits=0x0010,
                maskBits=0x001,   # collide only with ground
                restitution=0.0)  # 0.99 bouncy
                )
        self.lander.color1 = (0.5, 0.4, 0.9)
        self.lander.color2 = (0.3, 0.3, 0.5)
        self.lander.ApplyForceToCenter( (
            self.np_random.uniform(-INITIAL_RANDOM, INITIAL_RANDOM),
            self.np_random.uniform(-INITIAL_RANDOM, INITIAL_RANDOM)
            ), True)

        self.legs = []
        for i in [-1, +1]:
            leg = self.world.CreateDynamicBody(
                position=(VIEWPORT_W/SCALE/2 - i*LEG_AWAY/SCALE, initial_y),
                angle=(i * 0.05),
                fixtures=fixtureDef(
                    shape=polygonShape(box=(LEG_W/SCALE, LEG_H/SCALE)),
                    density=1.0,
                    restitution=0.0,
                    categoryBits=0x0020,
                    maskBits=0x001)
                )
            leg.ground_contact = False
            leg.color1 = (0.5, 0.4, 0.9)
            leg.color2 = (0.3, 0.3, 0.5)
            rjd = revoluteJointDef(
                bodyA=self.lander,
                bodyB=leg,
                localAnchorA=(0, 0),
                localAnchorB=(i * LEG_AWAY/SCALE, LEG_DOWN/SCALE),
                enableMotor=True,
                enableLimit=True,
                maxMotorTorque=LEG_SPRING_TORQUE,
                motorSpeed=+0.3 * i  # low enough not to jump back into the sky
                )
            if i == -1:
                rjd.lowerAngle = +0.9 - 0.5  # The most esoteric numbers here, angled legs have freedom to travel within
                rjd.upperAngle = +0.9
            else:
                rjd.lowerAngle = -0.9
                rjd.upperAngle = -0.9 + 0.5
            leg.joint = self.world.CreateJoint(rjd)
            self.legs.append(leg)

        self.drawlist = [self.lander] + self.legs

        return self.step(np.array([0, 0]) if self.continuous else 0)[0]

    def _create_particle(self, mass, x, y, ttl):
        p = self.world.CreateDynamicBody(
            position = (x, y),
            angle=0.0,
            fixtures = fixtureDef(
                shape=circleShape(radius=2/SCALE, pos=(0, 0)),
                density=mass,
                friction=0.1,
                categoryBits=0x0100,
                maskBits=0x001,  # collide only with ground
                restitution=0.3)
                )
        p.ttl = ttl
        self.particles.append(p)
        self._clean_particles(False)
        return p

    def _clean_particles(self, all):
        while self.particles and (all or self.particles[0].ttl < 0):
            self.world.DestroyBody(self.particles.pop(0))

    def step(self, action):
        if self.continuous:
            action = np.clip(action, -1, +1).astype(np.float32)
        else:
            assert self.action_space.contains(action), "%r (%s) invalid " % (action, type(action))

        # Engines
        tip  = (math.sin(self.lander.angle), math.cos(self.lander.angle))
        side = (-tip[1], tip[0])
        dispersion = [self.np_random.uniform(-1.0, +1.0) / SCALE for _ in range(2)]

        m_power = 0.0
        if (self.continuous and action[0] > 0.0) or (not self.continuous and action == 2):
            # Main engine
            if self.continuous:
                m_power = (np.clip(action[0], 0.0,1.0) + 1.0)*0.5   # 0.5..1.0
                assert m_power >= 0.5 and m_power <= 1.0
            else:
                m_power = 1.0
            ox = (tip[0] * (4/SCALE + 2 * dispersion[0]) +
                side[0] * dispersion[1])  # 4 is move a bit downwards, +-2 for randomness
            oy = -tip[1] * (4/SCALE + 2 * dispersion[0]) - side[1] * dispersion[1]
            impulse_pos = (self.lander.position[0] + ox, self.lander.position[1] + oy)
            p = self._create_particle(3.5,  # 3.5 is here to make particle speed adequate
                                    impulse_pos[0],
                                    impulse_pos[1],
                                    m_power)  # particles are just a decoration
            p.ApplyLinearImpulse((ox * MAIN_ENGINE_POWER * m_power, oy * MAIN_ENGINE_POWER * m_power),
                                impulse_pos,
                                True)
            self.lander.ApplyLinearImpulse((-ox * MAIN_ENGINE_POWER * m_power, -oy * MAIN_ENGINE_POWER * m_power),
                                        impulse_pos,
                                        True)

        s_power = 0.0
        if (self.continuous and np.abs(action[1]) > 0.5) or (not self.continuous and action in [1, 3]):
            # Orientation engines
            if self.continuous:
                direction = np.sign(action[1])
                s_power = np.clip(np.abs(action[1]), 0.5, 1.0)
                assert s_power >= 0.5 and s_power <= 1.0
            else:
                direction = action-2
                s_power = 1.0
            ox = tip[0] * dispersion[0] + side[0] * (3 * dispersion[1] + direction * SIDE_ENGINE_AWAY/SCALE)
            oy = -tip[1] * dispersion[0] - side[1] * (3 * dispersion[1] + direction * SIDE_ENGINE_AWAY/SCALE)
            impulse_pos = (self.lander.position[0] + ox - tip[0] * 17/SCALE,
                        self.lander.position[1] + oy + tip[1] * SIDE_ENGINE_HEIGHT/SCALE)
            p = self._create_particle(0.7, impulse_pos[0], impulse_pos[1], s_power)
            p.ApplyLinearImpulse((ox * SIDE_ENGINE_POWER * s_power, oy * SIDE_ENGINE_POWER * s_power),
                                impulse_pos
                                , True)
            self.lander.ApplyLinearImpulse((-ox * SIDE_ENGINE_POWER * s_power, -oy * SIDE_ENGINE_POWER * s_power),
                                        impulse_pos,
                                        True)

        self.world.Step(1.0/FPS, 6*30, 2*30)

        pos = self.lander.position
        vel = self.lander.linearVelocity
        state = [
            (pos.x - VIEWPORT_W/SCALE/2) / (VIEWPORT_W/SCALE/2),
            (pos.y - (self.helipad_y+LEG_DOWN/SCALE)) / (VIEWPORT_H/SCALE/2),
            vel.x*(VIEWPORT_W/SCALE/2)/FPS,
            vel.y*(VIEWPORT_H/SCALE/2)/FPS,
            self.lander.angle,
            20.0*self.lander.angularVelocity/FPS,
            1.0 if self.legs[0].ground_contact else 0.0,
            1.0 if self.legs[1].ground_contact else 0.0,
            (self.x_pos_middle_landing - VIEWPORT_W/SCALE/2) / (VIEWPORT_W/SCALE/2)
            ]
        assert len(state) == 9

        reward = 0
        shaping = \
            - 100*np.sqrt((state[0]-state[8])*(state[0]-state[8]) + state[1]*state[1]) \
            - 100*np.sqrt(state[2]*state[2] + state[3]*state[3]) \
            - 100*abs(state[4]) + 10*state[6] + 10*state[7]  # And ten points for legs contact, the idea is if you
                                                            # lose contact again after landing, you get negative reward
        if self.prev_shaping is not None:
            reward = shaping - self.prev_shaping
        self.prev_shaping = shaping

        reward -= m_power*0.30  # less fuel spent is better, about -30 for heuristic landing
        reward -= s_power*0.03

        done = False
        if self.game_over or abs(state[0]) >= 1.0:
            done = True
            reward = -100
        if not self.lander.awake:
            done = True
            reward = +100
        return np.array(state, dtype=np.float32), reward, done, {}

    def render(self, mode='human'):
        from gym.envs.classic_control import rendering
        if self.viewer is None:
            self.viewer = rendering.Viewer(VIEWPORT_W, VIEWPORT_H)
            self.viewer.set_bounds(0, VIEWPORT_W/SCALE, 0, VIEWPORT_H/SCALE)

        for obj in self.particles:
            obj.ttl -= 0.15
            obj.color1 = (max(0.2, 0.2+obj.ttl), max(0.2, 0.5*obj.ttl), max(0.2, 0.5*obj.ttl))
            obj.color2 = (max(0.2, 0.2+obj.ttl), max(0.2, 0.5*obj.ttl), max(0.2, 0.5*obj.ttl))

        self._clean_particles(False)

        for p in self.sky_polys:
            self.viewer.draw_polygon(p, color=(0, 0, 0))

        for obj in self.particles + self.drawlist:
            for f in obj.fixtures:
                trans = f.body.transform
                if type(f.shape) is circleShape:
                    t = rendering.Transform(translation=trans*f.shape.pos)
                    self.viewer.draw_circle(f.shape.radius, 20, color=obj.color1).add_attr(t)
                    self.viewer.draw_circle(f.shape.radius, 20, color=obj.color2, filled=False, linewidth=2).add_attr(t)
                else:
                    path = [trans*v for v in f.shape.vertices]
                    self.viewer.draw_polygon(path, color=obj.color1)
                    path.append(path[0])
                    self.viewer.draw_polyline(path, color=obj.color2, linewidth=2)

        for x in [self.helipad_x1, self.helipad_x2]:
            flagy1 = self.helipad_y
            flagy2 = flagy1 + 50/SCALE
            self.viewer.draw_polyline([(x, flagy1), (x, flagy2)], color=(1, 1, 1))
            self.viewer.draw_polygon([(x, flagy2), (x, flagy2-10/SCALE), (x + 25/SCALE, flagy2 - 5/SCALE)],
                                    color=(0.8, 0.8, 0))

        return self.viewer.render(return_rgb_array=mode == 'rgb_array')

    def close(self):
        if self.viewer is not None:
            self.viewer.close()
            self.viewer = None

class RandomTargetLunarLander(LunarLander):
    continuous = True

def heuristic(env, s):
    """
    The heuristic for
    1. Testing
    2. Demonstration rollout.

    Args:
        env: The environment
        s (list): The state. Attributes:
                s[0] is the horizontal coordinate
                s[1] is the vertical coordinate
                s[2] is the horizontal speed
                s[3] is the vertical speed
                s[4] is the angle
                s[5] is the angular speed
                s[6] 1 if first leg has contact, else 0
                s[7] 1 if second leg has contact, else 0
                s[8] is the target coordinate
    returns:
        a: The heuristic to be fed into the step function defined above to determine the next step and reward.
    """

    angle_targ = s[0]*0.5 + s[2]*1.0         # angle should point towards center
    if angle_targ > 0.4: angle_targ = 0.4    # more than 0.4 radians (22 degrees) is bad
    if angle_targ < -0.4: angle_targ = -0.4
    hover_targ = 0.55*np.abs(s[0])           # target y should be proportional to horizontal offset

    angle_todo = (angle_targ - s[4]) * 0.5 - (s[5])*1.0
    hover_todo = (hover_targ - s[1])*0.5 - (s[3])*0.5

    if s[6] or s[7]:  # legs have contact
        angle_todo = 0
        hover_todo = -(s[3])*0.5  # override to reduce fall speed, that's all we need after contact

    if env.continuous:
        a = np.array([hover_todo*20 - 1, -angle_todo*20])
        a = np.clip(a, -1, +1)
    else:
        a = 0
        if hover_todo > np.abs(angle_todo) and hover_todo > 0.05: a = 2
        elif angle_todo < -0.05: a = 3
        elif angle_todo > +0.05: a = 1
    return a

def demo_heuristic_lander(env, seed=None, render=False):
    env.seed(seed)
    total_reward = 0
    steps = 0
    s = env.reset()
    while True:
        a = heuristic(env, s)
        s, r, done, info = env.step(a)
        total_reward += r

        if render:
            still_open = env.render()
            if still_open == False: break

        if steps % 20 == 0 or done:
            print("observations:", " ".join(["{:+0.2f}".format(x) for x in s]))
            print("step {} total_reward {:+0.2f}".format(steps, total_reward))
        steps += 1
        if done: break
    return total_reward

if __name__ == '__main__':
    demo_heuristic_lander(LunarLander(), render=True)

Here are the results:
Green is Normal Lunar Lander Continuous
Pink is the Random Target Lunar Lander Continuous

Quick question before going any deeper - how have you scaled the new flag position values for input to the NN - i.e. what range can they take "naturally" in the environment, and what values as input to the NN do you use to represent those natural values?

I have changed the title of this post to put what I think is your main question. Feel free to edit your post again if that's not the case.

Will post code shortly


text:
Is it normal getting noise values in the error history along training iteration?
I'm giving my first steps in really learning machine learning.
As an exercise in my online course, it was asked for me to code the Cost function of some neural network that should resolve the handwritten problem with digits between 1 to 10.
As most of you know, the cost function of NN is given by:
$$
J(\theta)=\frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K}\left[-y_{k}^{(i)} \log \left(\left(h_{\theta}\left(x^{(i)}\right)\right)_{k}\right)-\left(1-y_{k}^{(i)}\right) \log \left(1-\left(h_{\theta}\left(x^{(i)}\right)\right)_{k}\right)\right]
$$
so I tried to code it considering the following information:

where $\mathrm{h_{\theta}}(\mathrm{x^{i}})$ is computed as shown in Figure 2 and $K=10$ is the total number of possible labels. Note that $h_{\theta}\left(x^{(i)}\right)_{k}= a_{k}^{(3)}$ is the activation (output value) of the $k$ -th output unit Also, recall that whereas the original labels (In the variable $y$) were $1,2, \ldots, 10$, for the purpose of training a neural network, we need to recode the labels as vectors containing only values 0 or 1, so that
$$
y=\left[\begin{array}{c}
1 \\
0 \\
0 \\
\vdots \\
0
\end{array}\right],\left[\begin{array}{l}
0 \\
1 \\
0 \\
\vdots \\
0
\end{array}\right], \ldots . \text { or }\left[\begin{array}{c}
0 \\
0 \\
0 \\
\vdots \\
1
\end{array}\right]
$$
For example, if $x^{i}$ is an image of the digit $5,$ then the corresponding $y^{i n}$ (that you should use with the cost function) should be a $10-$ dimensional vector with $y_{5}=1,$ and the other elements equal to $0$. You should implement the feedforward computation that computes $\mathrm{h_{\theta}}(\mathrm{x^{i}})$ for every example $i$ and sum the cost overall examples. Your code should also work for a dataset of any size, with any number of labels (you can assume that there are always at least $K \geq 3$ labels)

plotting the graph, I got:

here's my error cost function code:
function [J grad] = nnCostFunction(nn_params, ...
                                   input_layer_size, ...
                                   hidden_layer_size, ...
                                   num_labels, ...
                                   X, y, lambda)

% Setup some useful variables
 m = size(X, 1)
% bias  = ones(m,1)';
Theta1 = [ ones(401,1)  reshape(nn_params(1:hidden_layer_size * (input_layer_size + 1)), ...
                 hidden_layer_size, (input_layer_size + 1))']';

Theta2 = reshape(nn_params((1 + (hidden_layer_size * (input_layer_size + 1))):end), ...
                 num_labels, (hidden_layer_size + 1));

J =0;
error_history = zeros(1,m);
y_coded = zeros(1,num_labels);
for i = 1:m
    y_coded(y) = 1;
    X_in = [1 X(i,:)]';
    hypotesis_array = sigmoid(Theta2*sigmoid(Theta1*X_in));

    for k =1:num_labels
        J = J  -(y_coded(k)*log10(hypotesis_array(k)) -  (1- y_coded(k))*log10(hypotesis_array(k)));
    end
    J =J/m;
    error_history(i) = J;

end
plot(1:5000, error_history);
ylabel("Error_Value");
xlabel("training iteration");

I did that considering that the weights were previously given.
Is it normal getting this noise error history value or I did something wrong with my code?


text:
How to get more accuracy of the logistic regression model?
I am working on a Baby Crying Detection model using logistic regression.
Out of $581$ audios, $222$ are of a baby crying. Each audio is of $5$ seconds.
what I have done is convert each audio into numbers. and those numbers go into a .csv file. so first I took $100$ samples from each audio, then $1000$ samples, and then all $110250$ samples into a .csv file, and at the end of each of them was a number 1 (crying) or 0 (not crying). Then I trained the model using logistic regression from that .csv file.
The Problem I m facing is that with $100$ samples the 64% accuracy on each audio,  while with 1000 samples and 110250 samples(Full dataset) it reaches to 66% accuracy only. How can I improve the accuracy of my model to upto 80% using logistic regression.
I can only use simple logistic regression because I have to deploy the model on Arduino.

How are you processing the raw audio samples "into numbers". For audio processing this is typically some variant of FFT power spectrums. MFCC has been popular for speech. Are you doing anything like that? Please explain more about the pre-processing before reaching the predictive model and [edit] those details into the question. In addition, is sampling triggered by anything (such as hitting a volume threshold)? Or is sampling done at fixed intervals?

Simply i use librosa library to get audio files into an array. then that array is pasted into a CSV file. I know about MFCC. lets say i pick 100 samples. I pick 100 samples at equal intervals, then those 100 samples per audio (100x581 table of total audios) go through logistic regression which gives output of 100 weights or coefficients. Then arduino takes 100 samples from live recording, these 100 samples and trained model 100 samples are multiplied one to one, then added, then sigmoid function is applied. if output value is greater than 0.5 it means baby is crying. my friend told this method

i cannot use MFCC because i need to deploy the model on arduino and all the calculation on arduino needs 100 weights, while MFCC outputs just one value per audio

Try Rectification
Improve the features available to your model, Remove some of the NOISE present in the data.

In audio data, a common way to do this is to smooth the data and then rectify it so that the total amount of sound energy over time is more distinguishable.

# Rectify the audio signal
audio_rectified = audio.apply(np.abs)

You can also calculate the absolute value of each time point. This is also called Rectification because you ensure that all time points are positive.

Smooth your data by taking the rolling mean in a window of say 50 samples

audio_rectified_smooth = audio_rectified.rolling(50).mean()

Calculating the envelope of each sound and smoothing it will eliminate much of the noise and you have a cleaner signal.

Calculate Spectrogram

Calculate a spectrogram of sound(i.e combining of windows Fourier transforms). This describes what spectral content (e.g., low and high pitches) are present in the sound over time. there is a lot more information in a spectrogram compared to a raw audio file. By computing the spectral features, you have a much better idea of what's going on.

this is similar to how we calculate rolling mean :

We calculate multiple Fourier transforms in a sliding window to see how it changes over time. For each time point, we take a window of time around it, calculate a Fourier transform for the window, then slide to the next window. The result is a description of the Fourier transform as it changes throughout the time-series called a short-time Fourier transform or STFT.

Choose a windows size and shape
At a timepoint, calculate the FFT for that window
Slide the window over by one
Aggregate the results

Calculating the STFT

We can calculate the STFT with librosa $\rightarrow$ import librosa as lr
There are several parameters we can tweak (such as window size)
For our purposes, we'll convert into decibels which normalizes the average values of all frequencies.
We can then visualize it with the specshow() function

This is how you can calculate the STFT
# Calculating the STFT
# Import the functions we'll use for the STFT
from librosa.core import stft, amplitude_to_db
from librosa.display import specshow

# Calculate our STFT
HOP_LENGTH = 2**4
SIZE_WINDOW = 2**7
audio_spec = stft(audio, hop_length=HOP_LENGTH, n_fft=SIZE_WINDOW)

# Convert into decibels for visualization
spec_db = amplitude_to_db(audio_spec)

# Visualize
specshow(spec_db, sr=sfreq, x_axis='time', y_axis='hz', hop_length=HOP_LENGTH)

Try Spectral feature engineering
you can also perform the Spectral feature engineering on your baby audio data

since each time series has a different spectral pattern.
We can calculate these spectral patterns by analyzing the spectrogram.
For example, spectral bandwidth and spectrum centroids describe where most of the energy is at each moment in time.

# Calculate the spectral centroid and bandwidth for the spectrogram
bandwidths = lr.feature.spectral_bandwidth(S=spec)[0]
centroids = lr.feature.spectral_centroid(S=spec)[0]

# Display these features on top of the spectrogram
ax = specshow(spec, x_axis='time', y_axis='hz', hop_length=HOP_LENGTH)
ax.plot(times_spec, centroids)
ax.fill_between(times_spec, centroids - bandwidths / 2,centroids + bandwidths / 2, alpha=0.5)

Now you can Combine spectral(spec $\rightarrow$ spectral dataframe) and temporal features in a classifier
centroids_all = []
bandwidths_all = []

for spec in spectrograms:
    bandwidths = lr.feature.spectral_bandwidth(S=lr.db_to_amplitude(spec))
    centroids = lr.feature.spectral_centroid(S=lr.db_to_amplitude(spec))
    # Calculate the mean spectral bandwidth
    bandwidths_all.append(np.mean(bandwidths))
    # Calculate the mean spectral centroid
    centroids_all.append(np.mean(centroids))
# Create our X matrix
X = np.column_stack([means, stds, maxs, tempo_mean, tempo_max, tempo_std, bandwidths_all, centroids_all])

for your logistic regression models

One of the ways to improve accuracy is by optimizing the prediction probability cutoff scores generated by your logit model.
You can Normalize all your features to the same scale before putting them in a machine learning model.
Look for class imbalance in your data.
You can Optimize on other Metrics also such as Log Loss and F1-Score.
Tune the hyperparameters of your model. In the case of LogisticRegression, parameter $C$ is a hyperparameter.

thanks alot for giving your time :) the audio files have nearly no noise in it as they are specially recorded sounds from a mobile application of baby crying. i have tried normalizing the values in the array from the range of -1 to 1, TO 0 to 1. Moreover i have tried using ANN which uses spectrograms method and it gave me an accuracy of 100% after 100 epoches. The problem is that i cannot use spectrograms or MFCCs, kindly read my comments in the above question so you will have an idea. Well my data is imbalanced since 222 out of 581 audios are of baby crying. Thanks alot once again! :)

@MuhammadWaqarAnwar FFT is an efficient algorithm, and libraries for it are available for the arduino - see e.g. https://www.youtube.com/watch?v=djGOOL2mqV8


text:
Does ChatGPT's limited understanding of novel inputs reveal limitations in language's ability to convey meaning?
As a language model, ChatGPT generate responses based on patterns they have learned from the data they were trained on. However, it may not always provide a perfect or complete answer to novel or unexpected inputs, indicating that it does not have a genuine understanding of the task or concept at hand.

The question then becomes whether inability of ChatGPT or other language based AI system to fully understand and respond to novel or unexpected inputs is a reflection of the inherent limitations of language as a medium for conveying meaning and understanding?

Do you have any examples in mind about your novel or unexpected inputs? Are they in the sense of purely syntactically technical/tricky or semantically philosophical?

"No, I don't have any specific examples in mind. I was asking about ChatGPT's general ability to understand novel inputs, regardless of whether they are purely syntactically technical or semantically philosophical in nature. I was wondering if ChatGPT's limited understanding of novel inputs reveals limitations in language's ability to convey meaning more broadly."

There are people e.g. Yann Le Cun, who believe that language is not enough to provide a LLM or other system a world model. World models / task models / environment  models are important in logic to codify what you know to be able to make novel logical outcomes.


text:
How is the noise in the forward process in Denoising Diffusion Probabilistic Models computed?
The inputs are decayed towards the origin using this formula within Denoising Diffusion Probabilistic Models (DDPMs):
$$q\left(\mathbf{x}_{1: T} \mid \mathbf{x}_0\right):=\prod_{t=1}^T q\left(\mathbf{x}_t \mid \mathbf{x}_{t-1}\right), \quad q\left(\mathbf{x}_t \mid \mathbf{x}_{t-1}\right):=\mathcal{N}\left(\mathbf{x}_t ; \sqrt{1-\beta_t} \mathbf{x}_{t-1}, \beta_t \mathbf{I}\right)$$
I however do not understand how the origin is determined. How is the noise in the forward process or diffusion process computed? In the original DDPM paper it is only stated that:

the forward process variances $β_t$ can be learned by reparameterization [33] or held constant as hyperparameters.

You can generate a noise $\epsilon_t$ by sampling a distribution (which you already explicitly stated as $\mathcal{N}(\mathbf{x}_t; \sqrt{1-\beta_t}\mathbf{x}_{t-1}, \beta_t\mathbf{I})$) during the sampling phase. The "origin" is the standard normal distribution $\mathcal{N}(\mathbf{0}, \mathbf{I})$, which is a constant instead of a variable ground truth to learn.
The variable ground truth to learn (using a U-Net) during the training phase is rather the noise $\epsilon_t$ added during the sampling phase. Reverse to the diffusion (sampling) process, the noise $\epsilon_t$, when subtracted from the noisier $\mathbf{x}_{t}$, can produce a less noisy $\mathbf{x}_{t+1}$. Therefore, given $\mathbf{x}_{t}$, the noise $\epsilon_t$ encodes certain information regarding the input data $\mathbf{x}_{T}$.

Can you maybe elaborate on that? I still do not understand how the origin is determined, is it the distribution of the input data point during the specific time point?

@Mariusmarten I edited the answer. Maybe you confuse origin with ground truth. Diffusion models are intuitively different from traditional neural networks due to the additional sampling phase.


text:
How to stop DQN Q function from increasing during learning?
Following the DQN algorithm with experience replay:
Store transition $\left(\phi_{t}, a_{t}, r_{t}, \phi_{t+1}\right)$ in $D$ Sample random minibatch of transitions $\left(\phi_{j}, a_{j}, r_{j}, \phi_{j+1}\right)$ from $D$ Set
$$y_{j}=\left\{\begin{array}{cc}r_{j} & \text { if episode terminates at j+1} \\ r_{j}+\gamma \max _{d^{\prime}} \hat{Q}\left(\phi_{j+1}, a^{\prime} ; \theta^{-}\right) & \text {otherwise }\end{array}\right.$$
Perform a gradient descent step on $\left(y_{j}-Q\left(\phi, a_{j} ; \theta\right)\right)^{2}$ with respect to the network parameters $\theta$.

We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$.

Assume I have positive but changing rewards. Meaning, $r>0$.

Thus, since the rewards are positive, by calculating the loss, I notice that almost always $Q(s)< Q(s+1)+r$.
Therefore, the network learns to always increase the $Q$ function , and eventually, the $Q$ function is higher in same states in later learning steps.
How can I stabilize the learning process?

Could you clarify whether you are working on an episodic or continuous problem?

didn't u use some negative rewards?

I changed the rewards to be negative and positive by substructing the mean reward.
It seems to improve the Q function boundries.

You can use discount factor gamma less then one.
You can use finite time horizon - only for states which are no farther away then T time steps reward propagate back
You can use sum of rewords averaged over time for Q

All of those are legitimate approaches.

@mimirror2image 1. My discounr factor is 0.9. This increasing process happens even with discount factor 0.1.
2. How to calculate it? The network should calculate it.
3.Sum of rewards will still be > 0, which returns to the same procedure of increasing q function

Therefore,the network learns to always increase the Q function , and eventually the Q function is higher in same states in later learning steps

If your value function keeps increasing in later steps that means that the network is still learning those Q-values, you shouldn't necessarily prevent that. Your Q-values won't increase forever even if the rewards are always positive. You basically have a regression problem here and when the value of $Q(s,a)$ becomes very close to the predicted value of  $r+Q(s',a)$ value of $Q(s,a)$ will stop increasing by itself.

Yes, but than the rewards are "swollen" in the Q function std. The Q function should'nt be more than 5*max_reward.

sorry, I didn't quite understand, are you saying that Q-value can't be more than 5 times maximum possible reward ? What makes you think that.

Im saying it should not be. Otherwise, the rewards will be swollen in the q function standard deviation

@BestR: there is no basis in theory for your statement about this limit to the action value - its value is supposed to be the discounted sum of future rewards, and that can be any arbitrary amount times individual rewards, depending on how you have set up the environment. Please could you explain more where this constraint that you want has come from, as it may help clarify your question?

I'm still a bit confused, why exactly are you calculating std of Q-values, it's not part of the DQN algorithm or are you doing that for your own statistics ? Also, what do you mean by rewards get "swollen" in  Q-value std, why exactly are you using rewards to calculate std of Q-values. Are you maybe changing your rewards based on the value of std for Q-values ?

The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions affirmatively. In particular, we first show that the recent DQN algorithm, which combines Q-learning with a deep neural network, suffers from substantial overestimations in some games in the Atari 2600 domain. We then show that the idea behind the Double Q-learning algorithm, which was introduced in a tabular setting, can be generalized to work with large-scale function approximation. We propose a specific adaptation to the DQN algorithm and show that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games.
https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/viewPaper/12389


text:
How can Viv generate new code based on some user's query?
I have been looking into Viv, an artificial intelligent agent in development. Here is a demonstration of Viv (by Dag Kittlaus).
Based on what I understand, this AI can generate new code and execute it based on a query from the user.
What I am curious to know is how this AI can learn to generate code based on some query. What kind of machine learning algorithms are involved in this process?
One thing I considered is breaking down a dataset of programs by step. For example, here is the code to take an average of 5 terms

Add all 5 terms together
Divide by 5

Then I would train an algorithm to convert text to code. That is as far as I have figured out. However, I haven't tried anything because I'm not sure where to start.
Does anybody have any ideas on how Viv is implemented?

I was looking into genetic algorithms and things of that sort when I came across this video: Genetic Algorithms for Automated Source Code Evolution: a C++11 tutorial by David Miller, who seems to know what he's doing. I watched like 20 minutes of it. Interesting stuff and he goes step by step with everything.

I wouldn't be surprised if it's based on NLP and statistical models that consider existing codes that are associated with comments/documents. So some paraphrasing models for the comments to cover a wide rage of tests, and some correction models after getting the initial draft of the code. This is my guess.

4 years and 6 months have past since your unanswered question.
That is an eternity in terms of Machine Learning and things have evolved a lot. So I will answer about the present, not the past.
Today, there are some code generating models, like GitHub Copilot and OpenAi Codex which are based on NLG (Natural Language Generation). The principle is very simple:

Make a huge (billions of parameters) model to predict the next "word" (token) in a sequence of text.
Train that model with all available data you can find, and it will understand our language and all kinds of concepts, from philosophy to medicine (that's a GPT).
Then you fine-tune it to perform well in programming code, specially in a database filled with code comments.

And now you have a highly capable model for code prediction (generation) from a comment (imperative natural language query).

There are also other AI strategies for code prediction, like https://www.tabnine.com/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Why is it a problem if the outputs of an activation function are not zero-centered?
In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that the gradient of our loss w.r.t. the weights $\frac{\partial L}{\partial w}$ which is equal to $\frac{\partial L}{\partial \sigma}\frac{\partial \sigma}{\partial w}$ will always be either negative or positive and we'll have a problem updating our weights as she shows in this slide, we won't be able to move in the direction of the vector $(1,-1)$. I don't understand why since she only talks about one component of our gradient and not the whole vector. if the components of the gradient of our loss will have different signs which will allow us to adjust to different directions I'm I wrong? But the thing that I don't understand is how this property generalizes to non zero-centered functions and non-zero centered data?

Yes, if the activation function of the network is not zero centered, $y = f(x^{T}w)$ is always positive or always negative. Thus, the output of a layer is always being moved to either the positive values or the negative values. As a result, the weight vector needs more updates to be trained properly, and the number of epochs needed for the network to get trained also increases. This is why the zero centered property is important, though it is NOT necessary.
Zero-centered activation functions ensure that the mean activation value is around zero. This property is important in deep learning because it has been empirically shown that models operating on normalized data––whether it be inputs or latent activations––enjoy faster convergence.
Unfortunately, zero-centered activation functions like tanh saturate at their asymptotes –– the gradients within this region get vanishingly smaller over time, leading to a weak training signal.
ReLU avoids this problem but it is not zero-centered. Therefore all-positive or all-negative activation functions whether sigmoid or ReLU can be difficult for gradient-based optimization. So, To solve this problem deep learning practitioners have invented a myriad of Normalization layers (batch norm, layer norm, weight norm, etc.). we can normalize the data in advance to be zero-centered as in batch/layer normalization.
Reference:
A Survey on Activation Functions and their relation with Xavier and He Normal Initialization

Thank you for your answer. Can you provide more details please ? I don't understand how if the $y=f(x^{T}w)$ function isn't zero-centered then the outputs will always be either positive or negative. If we can imagine that $f$ has a symetrical graph (like a gaussian function) shifted towards the positives for example, then it'll all depends on the "width" am I wrong ? A more concrete example: $y=tanh(z-2)$. Then if $x^{T}w=[-4,2,4]$ we'll have $tanh(x^{T}w)=[-1,0,1]$. Do we always expect that $x^{T}w$ will be very close to zero so that we aren't in the later case?

like For the sigmoid function, the gradient saturates as $σ(x)→1$ or $σ(x)→0$ . We can simply tell this by the slope of the function as these points are approached, and also compute the derivative. We know that $σ′(x)=σ(x)(1−σ(x))$  , so $σ′(x)→0$ at these extreme values.

Therefore, if inputs are not zero-centered and with sigmoid , saturation will be more likely than if our inputs were zero-centered. but early on in learning, we generally want larger gradients, since our error is high. This can drastically slow down $\eta$. Since zero-centering the data will bring the inputs of the sigmoid away from where they can saturate, zero-centering could increase the gradient signal during the early stages of learning, providing for faster learning.

you know that for sigmoid the bound in the range of (0,1). Hence it always produces a non-negative value as output.

This maybe somewhat out of context but let me ask it, "For negative values, Relu activation gives zero as output. Is it be a problem be for negative values weights of that layer will not update just like vanishing gradient of Tanh activation?

@Faizy thank you for your answer. I think the term zero-centered may be a little confusing because we may have zero-centered data but that have big magnitude, imagine the set of points ${ -10, -5, -3, 3, 5, 10}$. Then we may say that this set is zero-centered, and it'll still give us saturation. But I understand what you mean. Thank you for all the explanation.

@SwaksharDeb I don't think the problem comes from the weights alone since you have $tanh(x^{T}w)$. As long as $x^{T}w$ is negative that neuron won't be updated since the gradient will be zero. I believe this happens for two reasons: 1) When you initialize the weights, you're unfortunate to have $x^{T}w$ negative and you get no update. 2) You have good initialization but you update the weights in such a way that $x^{T}w$ become negative and you're neuron just dies. But since we're only able to work on our weights I guess you're right.


text:
Prooving the convergence rate of estimators (machine Learning)
I want to estimate a quantity and have two choices for estimators (they both sample from the same distribution). I suspect one of them has a higher variance and thus a slower convergence rate. I want to mathematically prove this, but I don't know where to start. Is there a standard reference in statistics that I can consult for example proofs of convergence rates in ML/statistics?

The Rate of Convergence of an Estimator will depend on a number of factors including:

the type of estimator used,
the data set used for training,
and the tuning parameters of the estimator.

However, in general, one can expect that an estimator will converge at a rate that is at least as fast as the inverse of the number of training examples used.
...I suspect one of them has a higher variance and thus a slower convergence rate...
Yes, if two estimators are sampling from the same distribution, the estimator with higher variance will converge more slowly than the estimator with lower variance &, this can be important when you are choosing between different algorithms to use for training your models, but There is no general rule; the choice depends on the specific situation, the one with the faster convergence rate will generally be more accurate that obvious.

"the rate of convergence provide useful insights when using iterative methods for calculating numerical approximations. If the order of convergence is higher, then typically fewer iterations are necessary to yield a useful approximation. Strictly speaking, however, the asymptotic behavior of a sequence does not give conclusive information about any finite part of the sequence."

There are a few different ways to approach this. One way would be to use the law of large numbers(~4). This states that, as the number of samples increases, the average of the sample values will converge to the true value of the quantity being estimated.
Another way would be to use the Central Limit Theorem(~5). This states that, as the number of samples increases, the distribution of the sample values will become more and more like a normal distribution. The mean of this distribution will be the true value of the quantity being estimated.
Here are some references may be relevant:

Computing and Estimating the Rate of Convergence
Order of convergence - berkeley.edu
A Course in Large Sample Theory by Thomas S. Ferguson ~ pdf


text:
Handcraft RNN with attention to extract central element
I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely certain how to design it. The goal of this question is to understand both mechanisms better.

So for example I have (10,20,30) or (10,20,30,40,50) given as input sequence.
At input 30 the RNN should output 20 at position 50 -> 30 and so forth.

My idea for the RNNs hidden state is to just increase it by 1. The hidden state h would just be a scalar.
e.g. (10,20,30) produces the states (1,2,3)
But now I am stuck as attention should work with the input and the hidden state. What I would need as output would be scored (0,1,0) * (10,20,30) = 20.
The scoring function I come up with would be s(h, number, i) = 1 if h/2 == i else 0.
But there I am using the index as an additional parameter / positional encoding and wondering if I can do it without it.

What could be other approaches to handcraft an RNN with attention to extracting the half-position element of a sequence?

One can view the hidden states just like stated in the question as made up from a linear increasing counter and the input.
The query will be the current position n and key the position count in the hidden state t:
a(t, n) = I(t == n/2) = w_t

where I is the indicator function.
The output can be seen as the sum over all w_t * x_t = x_t.
To be strict the value=x_t can be stored in the hidden states which would be
h_t = (x_t, t)


text:
When labelled data is not available, what are some common unsupervised learning algorithms for pattern recognition that can be used?
In pattern recognition systems, when no labeled data is available, what are some common unsupervised learning algorithms for pattern recognition, that can be used?

There are some unsupervised learning algorithms that can be used for pattern recognition (i.e. the discovery of patterns in data). The most notable one is probably k-means, which is a clustering algorithm. In k-means, you cluster your unlabeled data into groups (or clusters) based on the distance (or similarity) between them. When a new data point arrives, you'll associate it with the most similar cluster. In this sense, you are performing pattern recognition in an unsupervised way.
Here's an excerpt from the famous book Pattern Recognition and Machine Learning (2006) by C. Bishop

In other pattern recognition problems, the training data consists of a set of input vectors $x$ without any corresponding target values. The goal in such unsupervised learning problems may be to discover groups of similar examples within the data, where it is called clustering, or to determine the distribution of data within the input space, known as density estimation, or to project the data from a high-dimensional space down to two or three dimensions for the purpose of visualization.

So, there are other problems, apart from the problem of clustering, that you may want to solve with unsupervised learning algorithms for the purpose of pattern recognition, such as density estimation (see e.g. mixture models) or dimensionality reduction (see e.g. PCA).

Thank You. This is helpful. Couldn't up vote your answer as I don't have  >15 reputation.

@StevenM You're welcome! Anyway, although this information is available on the web, it isn't often clear what people mean by "pattern recognition", so I understand your confusion. Some people just say that it's the "task of finding patterns in the data", but my advice is that you always keep in mind your context!


text:
Do I have to downsample the input and upsample the output of the neural network when implementing the NICE algorithm?
Consider that my input is an RGB image. The size of my image is $N\times N$. I'm trying to implement NICE algorithm presented by Dinh. The bijective function $f: \mathbb{R}^d \to \mathbb{R}^d$ maps $X$ to $Z$. So I have $p_Z(Z)=p_X(X)$.
What I can't understand is that $N$ is much bigger than $d$. Does this mean that I should downsample the inputs? Does the resulting loss function change if I add a downsampling layer at the beginning of the neural net and also add an upsampling layer at the end of the net?


text:
How to normalize rewards in DQN?
I want to use a Deep Q-Network for a specific problem. My immediate rewards ($r_t = 0$) are all zeros. But my terminal reward is a large positive value $(r_T=100$). How could I normalize rewards to stabilize the training?  I think clipping rewards to be in range $[0,1]$ makes training harder because it just forces most values to be near zero.

why do you need to scale the reward here? the problem I would be concerned with is that almost all of your rewards are non-zero. sparse reward environments can be difficult to solve. perhaps you should look at the Hindsight Experience Replay algorithm as a starting point for overcoming sparse rewards.

Thank you. I now understand what I should do. I need to Define a bigger state which contains smaller state and gradually decrease it's volume so at the end of training it is equal to smaller state.


text:
What assumptions are made when positing the emergence of superintelligence?
Many experts seem to think that artificial general intelligence, or AGI, (on the level of humans) is possible and likely to emerge in the near-ish future. Some make the further step to say that superintelligence (much above the level of AGI) will appear soon after, through mechanisms like recursive self-improvement from AGI (from a survey).
However, other sources say that such superintelligence is unlikely or impossible (example, example).
What assumptions do those who believe in superintelligence make? The emergence of superintelligence has generally been regarded as something low-probability but possible (e.g. here). However, I can't seem to find an in-depth analysis of what assumptions are made when positing the emergence of superintelligence. What specific assumptions do those who believe in the emergence of superintelligence make that are unlikely, and what have those who believe in the guaranteed emergence of superintelligence gotten wrong?
If the emergence of superintelligence is to be seen as a low-probability event in the future (on par with asteroid strikes, etc.), which seems to be the dominant view and is the most plausible, what assumptions exactly makes it low-probability?

Yes, now it clearer! Maybe you could even add my example "e.g. what are the assumptions that people make when they say that recursive self-improvement is possible and will lead to the creation of SI", although that may restrict your question and people could focus on just that, while you're looking for general assumptions.

I am not an expert on the topic, but I will provide some information that could be useful or helpful.
I think that the first and maybe trivial assumption that people make when they say that AGI or SI can emerge is that general intelligence (whatever the definition is) is computable, i.e. there's a Turing machine or, in general, a mathematical model of computation that can simulate an algorithm that possibly represents general intelligence. I don't know how likely the mind of a human (or any other general intelligence) is computable (or to what extent), but there's the so-called computational theory of mind (CTM) that goes into this direction.
An assumption related to the emergence of SI is that recursive self-improvement is physically possible. I also don't know exactly how likely or unlikely recursive self-improvement is, but my limited knowledge of thermodynamics (and physics) suggests that it will not be possible (at least, at the pace or the way people claim or want to suggest). I am not saying that we will not see more progress in the next years (we will), but this doesn't mean that we will be able to create an AGI. People often overestimate themselves and their intelligence.
A third assumption is related to the current achievements in the artificial intelligence field. Many people claim that there has been a lot of progress in recent years and that there's no reason to believe that this will not continue. However, history tells us that the predictions of AI scientists about the future of the AI field were often wrong. There have been at least 2 AI winters because of these wrong predictions and unmet expectations.
Many people also don't exclude the possibility of the emergence of AGI or SI only because they have not yet been proven wrong.
Another assumption is that there won't be any catastrophe (like COVID-19  can potentially be) that will significantly slow down the general scientific progress.

Thank you for your answer! I have some questions: (1) why does it matter if intelligence is computable? Why can't AGI be based on a neural network fed massive amounts of data, or a very advanced semantic reasoner? (2) I agree with your other points, but I'm wondering if you know of any sources that discuss the speed at which recursive self-improvement could happen? For example, it's conceivable that improvement takes longer and longer each iteration (even if the machine is smarter, it has too much computation to do, etc.). Any sources on this?

@user35673 (1) That's the point. If you have a neural network that can be used to represent GI, then GI is computable. However, the converse may not be true, i.e. if GI is computable, it doesn't mean that there's a NN that can compute/represent it. Anyway, how do we know that GI is computable? And what is the definition of GI? These questions need to be answered before attempting to implement GI. There are some definitions of GI. For example, have a look at AIXI (which is actually incomputable, although there are computable approximations!).

@user35673 (2) Honestly, no. Maybe have a look at Bostrom's book "Superintelligence: Paths, Dangers, Strategies". I had read like the first 100 pages of that book, which goes into this direction, but I don't guarantee that you will find exactly what you're looking for there. Maybe have a look at this http://people.idsia.ch/~juergen/history.html.

I see, thank you. So would the following be a fair summary of the current state?: AGI is possible, but we have no clue on when or whether it will exist (because of problems like the computability of human-level intelligence). Even if AGI does emerge, it is unlikely it will achieve superintelligence because of the reasons outlined here. And, finally, if superintelligence is achieved, we do not know if it matters too much (e.g. it takes too much computing power, or is simply not that useful).

If this is an accurate summary, I can certainly see why people think SI is extremely unlikely. To expand on my last point ("simply not that useful"), it is conceivable to me how even the smartest agent could not solve global warming by itself. Or, even, that it is given the instruction to by humans. Presumably, if humans develop this kind of SI, they would be unbelievably careful with its usage.

@user35673 Roughly, yes.


text:
In NEAT, is it a good idea to give the same ID to node genes created from the same connection gene?
Do I have to prevent nodes created from the same connection gene to have different IDs/innovation number? In this example, the node 6 is created from the connection going from node 3 to node 4:

In the case where that specific node was already globally created, is it useful to give it the same ID for crossover? Because the goal of NEAT is to do meaningful crossover by doing historical marking. The paper from Kenneth O. Stanley says at page 108:

[...] by keeping a list of the innovations that occurred in the current generation, it
  is possible to ensure that when the same structure arises more than once through independent mutations in the same generation, each identical mutation is assigned the
  same innovation number.

Why don't we do that for node genes too?

In NEAT, the innovation of a node does not affect the evolution directly. Only the connection genes and their innovation will matter. So you can simply have whole numbers as IDs under each Genome / Network.
--EDIT-- (Complete reasoning)
In the original paper, it is clearly stated that the nodes from the better genome is taken during crossover and then only the connections are cross-over'ed (in some method) and hence the innovation numbers for the connections. NEAT is connection-centric and does not care much about evolving nodes.
Adding to that from basic neural networks theory, the nodes will never matter in a neural network because all the calculation and the learning happens in the connections. Think about a regular feed-forward network. You only care about the weight matrix which is a property of the connections instead of the nodes though both are present. Similarly in a NEAT generated network, the nodes will not matter, as all the learning takes place in the way these nodes are connected and the weights of the network.
Further, the nodes list can easily be derived from the connection list and hence marking the connections, are enough.


text:
How to implement a LSTM for multilabel classification problem?
I would like to develop an LSTM because I have a variable input matrix. I am zero-padding to a specific length of 800.
However, I am not sure of how to classify a certain situation when each input matrix has multiple labels inside, i.e. 0, 1 and 2. Do I need to use multi-label classification?

Data shape
(250,800,4)

x_train(150,800,4)
y_train(150,800,1)
x_test(100,800,4)
y_test(100,800,1)

Building LSTM
model = Sequential()
model.add(LSTM(100, input_shape=))
model.add(Dropout(0.5))
model.add(Dense(100, activation='relu'))
model.add(Dense(800, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

I am not sure of how to build the LSTM for training and test. If I train my model with a 3D shape. It will mean that for real time predictions I should also have a 3D shape, but the idea is to have a 2D matrix as an input.


text:
How to train a LSTM with multidimensional data
I am trying to train a LSTM, but I have some problems regarding the data representation and feeding it into the model.
My data is a numpy array of three dimensions: One sample consist of a 2D matrix of size (600,5). 600(timesteps) and 5(features). However, I have 160 samples or files that represent the behavior of a user in multiple days. Altogether, my data has a dimension of (160,600,5).
The label set is an array of 600 elements which describes certain patterns of each 2D matrix. The shape of the output should be (600,1).
My question is how can I train the LSTM to the corresponding label set? What would be the best approach to handle this problem? The idea is that the output should be an array of (600,1) with 3 label inside.
Multiple_outputs {0,1,2}
      Output:    0000000001111111110000022222220000000000000
                 -------------600 samples ------------------

Input: (1, 600, 5)
Output: (600, 1)
Training: (160,600,5)

I look forward for some ideas!
dataset(160,600,5)

X_train, X_test, y_train, y_test = train_test_split(dataset[:,:,0:4], dataset[:,:,4:5],test_size = 0.30)

model = Sequential()
model.add(InputLayer(batch_input_shape = (92,600,5 )))
model.add(Embedding(600, 128))
#model.add(Bidirectional(LSTM(256, return_sequences=True)))
model.add(TimeDistributed(Dense(2)))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer=Adam(0.001),
              metrics=['accuracy'])

model.summary()

model.fit(X_train,y_train, batch_size=92, epochs=40, validation_split=0.2)

it is called Over-parameterization - "It refers to the scenario where the number of parameters of the model exceed the size of the training dataset or a similar threshold. Many neural networks are trained in an over-parameterized regime." - there's nothing extra-ordinary in your case for DL

I don't see any special characteristic in the problem you're posing. Any LSTM can handle multidimensional inputs (i.e. multiple features). You just need to prepare your data such as they will have shape [batch_size, time_steps, n_features], which is the format required by all main DL libraries (pytorch, keras and tensorflow).
I linked below 2 tutorials that shows how to implement an LSTM for part of speech tagging in Keras and Pytorch. This task is conceptually identical to what you want to achieve: use 2D inputs (i.e. embeddings) to predict the class (i.e. the pos tags) of each element of a sequence (i.e. every single word). For example:
# Input sample, 4 steps, encoded in 2D embeddings matrix
['we', 'had', 'crispy', 'fries']

# Output predictions, 4 labels
['PRP', 'VBD', 'JJ', 'NNS']

In your case the predicted label would be 0, 1 or 2 and you would have to encode your data in a matrix of shape [n_batch, 600, 5]. The only thing you definetly want to pay attention, since you're using temporal data, is to not shuffle your data at all before the training.
Keras tutorial: Build a POS tagger with an LSTM using Keras
Pytorch tutorial: LSTM’s in Pytorch

I get the following error -->                                                                                ValueError: Error when checking target: expected activation_6 to have 4 dimensions, but got array with shape (80, 600, 1)

could you add your code to the question? It's hard to help you debugging it without seeing it .

Sure. The 2D matrix is of size (600, 5) 600 rows that represent a value every second and each row shows specific characteristic at that point length, width, shape, color, material.  Each matrix is taken every 2 hrs. I will have per day 12 samples. I collected so far just 160 samples of the same length (600,5).  My training data is the concatenation of all these 2D arrays.


text:
What does "shape information" mean in terms of GAN(generative adversarial networks)?
A paper says

However, annotations used as inputs to C-GAN are typically based only on shape information, which can result in undesirable intensity distributions in the resulting artificially-created images.

What does "shape information" mean here?
I am aware of basic concept of GAN(generative adversarial networks), though I don't understand what "shape information" refers to.

I am aware the image above is an illustration of Image Segmentation. Would I think of the any one of the segmented area (red, green, blue) as a shape for a GAN?
Could someone please give a hint? Thanks in advance.


text:
Before GAN, what are the commonly used techniques for image-to-image translation?
As per a post, image-to-image translation is a type of CV problem.
I guess I understand the concept of image-to-image translation.

I am aware that GANs(generative adversarial networks) are good at this kind of problems.
I just wondered what the commonly used techniques are for this kind of problems Before GAN?
Could someone please give a hint? Thanks in advance.

Image to Image translation is the task of transferring an image's characteristics from one domain and representing it in another. GANs have provided an end to end method to do this task.
Prior to Gans, these tasks were done individually, by using classic image processing techniques mainly.
Techniques such as image denoising, or finding edges in photos, or using web results to join various images were used.
As mentioned by P Isola et. al., in the CycleGAN paper, Hertzmann et al. in their paper, Image Analogies, have employed a non-parametric texture model on a single input-output training image pair.
In Image Quilting for Texture Synthesis and Transfer, the authors used existing patches of images and stitched them together.
In Data-driven Hallucination of Different Times of Day from a Single Outdoor Photo, here the authors compare the input image with an available dataset of time-lapse videos similar to the input and find the frame at the time of input image and frame for a target time. They use local affine transformation to change the scene of an input image into target image.
Recent works have focused on using a dataset of paired input-output examples to learn the translation function using CNNs for semantic segmentation,
Fully convolutional networks for semantic segmentation


text:
Stochastic gradient descent does not behave as expected, even with different activation functions
I have been working on my own AI for a while now, trying to implemented SGD with momentum from scratch in python. After looking around and studying all the maths behind it, i finally managed to implement SGD in a neural network that i trained to recognize the classic MNIST digits dataset.
As activation function i always used sigmoid for both hidden and output neurons, and everything seems to work more or less ok, but now i wanted to step it up a bit and try to let SGD operate with different activations, so i added 2 other  functions to my code: relu and tanh.
The behaviours that i expected based on articles, documentation and "tutorials" found online were:
tanh: Should be slightly better than sigmoid
relu: should be much better than sigmoid and tanh
(By better i mean faster or at least higher accuracy the the end, or a mix of both)
Using tanh it looks like it's much slower converging to a minimum compared to sigmoid
Using relu...well, the results were very, VERY horrible
Here's the outputs with the different activations (Learning rate: 0.1, Epochs: 5, MiniBatch size: 10, Momentum: 0.9)
Sigmoid training

[Sigmoid for hidden layers, sigmoid for output layer]
Epoch: 1/5 (14.3271 s): Loss: 0.0685, Accuracy: 0.6231, Learning rate: 0.10000
Epoch: 2/5 (14.0060 s): Loss: 0.0503, Accuracy: 0.6281, Learning rate: 0.10000
Epoch: 3/5 (14.0081 s): Loss: 0.0482, Accuracy: 0.6382, Learning rate: 0.10000
Epoch: 4/5 (13.8516 s): Loss: 0.0471, Accuracy: 0.7085, Learning rate: 0.10000
Epoch: 5/5 (13.9411 s): Loss: 0.0374, Accuracy: 0.7990, Learning rate: 0.10000

Tanh training

[Tanh for hidden layers, sigmoid for output layer]
Epoch: 1/5 (13.7553 s): Loss: 0.3708, Accuracy: 0.4171, Learning rate: 0.10000
Epoch: 2/5 (13.7666 s): Loss: 0.2580, Accuracy: 0.4623, Learning rate: 0.10000
Epoch: 3/5 (13.5550 s): Loss: 0.2289, Accuracy: 0.4824, Learning rate: 0.10000
Epoch: 4/5 (13.7311 s): Loss: 0.2211, Accuracy: 0.5729, Learning rate: 0.10000
Epoch: 5/5 (13.6996 s): Loss: 0.2142, Accuracy: 0.5779, Learning rate: 0.10000

Relu training

[Relu for hidden layers, sigmoid for output layer]
Epoch: 1/5 (14.2100 s): Loss: 0.7725, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 2/5 (14.6218 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 3/5 (14.2116 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 4/5 (14.1657 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 5/5 (14.1427 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000

Another run with relu

Epoch: 1/5 (14.7391 s): Loss: 15.4055, Accuracy: 0.1658, Learning rate: 0.10000
Epoch: 2/5 (14.8203 s): Loss: 59.2707, Accuracy: 0.1709, Learning rate: 0.10000
Epoch: 3/5 (15.3785 s): Loss: 166.1310, Accuracy: 0.1407, Learning rate: 0.10000
Epoch: 4/5 (14.9285 s): Loss: 109.9386, Accuracy: 0.1859, Learning rate: 0.10000
Epoch: 5/5 (15.1280 s): Loss: 158.9268, Accuracy: 0.1859, Learning rate: 0.10000

For these examples the epochs are just 5 but incrementing the epochs the results dont change, tanh and relu for me perform worse than sigmoid.
Here is my python code reference for SGD:
SGD with momentum
This method was created to accept different activation functions to dynamically use them when creating the neural network object
The activation functions and their derivatives:
Activation functions and derivatives
The loss function i used is the mean squared error:

def mean_squared(output, expected_result):
    return numpy.sum((output - expected_result) ** 2) / expected_result.shape[0]

def mean_squared_derivative(output, expected_result):
    return output - expected_result

Is there some concept i am missing? Am i using the activation functions the wrong way? I really cannot find the answer to this even after searching for a long time.
I feel like the problem is somewhere in the backpropagation but i can't find it.
Any kind of help would be greatly appriciated
PS: I hope i posted this in the right place, i am pretty new to asking questions here, so if there is any problem i will move the question somewhere else
Edit:
I tried to implement this with tensorflow, using relu for hidden layers and sigmoid for output. The results i get with this implementation are the same as the ones i mentioned in my question, so unless i am doing something wrong in both situations i am left to think i cannot use relu with sigmoid, which makes sense cause relu can have very high values while sigmoid pushes them down between 0 and 1, therefore most of the times giving values very close to 1.
Code reference:
TensorFlow implementation

Did you try a lower learning rate for relu ?

I tried any kind of learning rate, from 0.000001 to 10, also with schedules, but nothing improves

Could you post the pseudocode of your backpropagation algorithm?
I recommend you start off as simple as possible (this includes your cost f(x), I would simply use Yexpected-Youtput) and see if it works and then continue adding things. If it's your first time with neural networks, I recommend you check this link out and you could also try practising the algorithms on a programming language like Octave/Matlab (it can be very efficient speed wise). Also check this question out (link). At the bottom there is a code example for the XOR problem. Please post the pseudocode of your code instead of just dumping it there.  Finally, don't just copy paste algorithms into your code, you need to understand them.

Hi, thanks for the reply. All the code inside my python project was written by me, i had to study the maths and everything behind it, so it took a lot of time, but everything worked well (with sigmoid). Apparently something is missing. I did start out with simple things and just then i passed to the mnist classification task. My problem now is that i dont get the expected results using other activations other than sigmoid, but with that, is working pretty neat so i was wondering if it was something specific to my code that was causing the error. I will edit my question with the pseudo code

I have only tried using the sigmoid and the step function so I can't really help you with the other ones.  But you can check this link: https://en.wikipedia.org/wiki/Activation_function .

So, i tried using Yexpected-Youtput (as the cost derivative to calculate gradients) instead of Youtput-Yexpected. The results become as bad as relu with all activation functions

As far as I know you don't need to find the derivative of the cost function. Only the activation function one. Check this link out if you are still unsure about backpropagation. I think that is your issue. http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html. You probably didn't convert the maths to the algorithms well enough and left something out

From what i understand, the cost function is used to find the cost of the network, but to optimize the whole thing in backpropagation you need the slope of the cost function, that tells you what direction and how much you should modify your weights. The mean squared cost function that i mentioned is the mean of the sum of  (Youtput - Yexpected) ^ 2 applied to each output neuron, which derivative is 2*(Youtput - Yexpected) or just (Youtput - Yexpected) since 2 in this context becomes irrelevant because it is a constant. Unless this is false, i don't think the problem is in the cost function.

I'm sorry but if this is a problem with your code I can't really help you out. I don't know python. But anyway maybe you could plot a graph with your results after, I don't know, 500 epochs ...


text:
How I can identify holes in a 3D CAD file?
How I can identify holes in a 3D CAD file?  I want to identify different types of holes, counterbored or countersunk holes. My program lets me extract, for example, the faces and adjacency of the faces.
I am talking about the Siemens NX for example.
The different types of holes you can see there:
https://www.google.com/url?sa=i&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_VwwfDrdggc&psig=AOvVaw1kIYdOt2qxazSYFXwHuXpb&ust=1586329824363000&source=images&cd=vfe&ved=0CAIQjRxqFwoTCOjd1qHh1egCFQAAAAAdAAAAABAJ

Hi and welcome to AI SE! Maybe you could provide a picture of the specific holes that you want to identify. Also, what program are you talking about?