text:
Concrete Example for Q Learning
I am not sure if I understood the q learning algorithms correctly.
Therefore I would give a concrete example and ask if someone can tell me how to update the q value correctly.
First I initialized a Neural Network with random weights. It shall henceforth evaluate the Q Value for all possible actions(4) given a State S.
Then the following happens. The agent is playing and is exploring.
For 3 steps the Q Values evaluated were:
(0,-1,-5,0), (0,-1,0,0), (0,-.6,0,0)
The reward given was: 0,0,1
The action took were: (1.,1.,1.)
In the random walk example (same reward given), it was: (1.,2.,3.)
So what are the new Q - Values, assuming a discount factor of 0.99 and the learning rate 0.1?
The States for Simplicity are only one number: 1,1.3,2.4 Where 2.4 is the state who ends the game...
The same example holds for exploiting. Is the algorithm the same here?
Here you see my last implementation:
    public void rlearn(ArrayList<Tuple> tupels, double learningrate, double discountfactor) {

    //newQ = sum of all rewards you have got through
    for(int i = tupels.size()-1; i > 0; i--) {
        MLData in = new BasicMLData(45);
        MLData out = new BasicMLData(5);

        //Add State as in
        int index = 0;
        for(double w : tupels.get(i).statefirst.elements) {
            in.add(index++, w);
        }

        //Now start updating Q - Values
        double qnew = 0;
        if(i <= tupels.size()-2){
            qnew = tupels.get(i).rewardafter + discountfactor*qMax(tupels.get(i+1));
        } else {
            qnew = tupels.get(i).rewardafter;
        }

        tupels.get(i).qactions.elements[tupels.get(i).actionTaken] = qnew;
        //Add Q Values as out
        index = 0;
        for(double w : tupels.get(i).qactions.elements) {
            out.add(index++, w);
        }
         bigset.add(in, out);
    }
}

Edit: This is the qMax - function:
    private double qMax(Tuple tuple) {
    double max = Double.MIN_VALUE;
    for(double w : tuple.qactions.elements) {
        if(w > max) {
            max = w;
        }
    }
    return max;
}

See https://ai.stackexchange.com/questions/7298/reinforcement-learning-in-asteroid-game/8028#8028

Your example needs to give the action taken on each step that generated those sampled rewards. You should include at least one step where the non-maximising action was taken. For a full explanation, you should give the example data in the form of state_label, predicted_rewards, action_taken, actual_reward, next_state_label, end_flag - these don't all need to be in vector/numeric form, although it would help if the Q values and rewards are (as you have already done), plus the action id needs to be numeric in order to find what the predicted Q value was

Your right, I edited the Acgtions taken, What are endflag and nextstatelabel?

state_label and next_state_label identify the states in the trajectory - it is implied in your question but not stated that your neural network estimates $(q(s, a_0), q(s, a_1), q(s, a_2), q(s, a_3) )$, and we need to know $s$. It is $q(s,a)$ that you revise, using $\text{max}{a'} q(s',a')$ to improve the estimate, so you need to identify $s$ (state_label), $s'$ (next_state_label) and $a$. The end_flag is boolean - whether the transition ends an episode - that is critical information on how you learn Q values, because $\text{max}{a'} q(s',a')$ is by definition $0$ in that case

I understood the end_flag. But I think I do not understand the state label. For what do you need the states again? Just to recalculate the error of the net. (see above)

The ids of the states are needed to explain how the formula works. They are used both for calculating the new target value, and for showing which Q value is being updated. References to different states are used in different parts of the same step, so it is important to make it clear which state is being used and why.

If your examples are sequential from the same trajectory (looks like they are), then you will end up with repeats, so first time step might have state_label $s_a$ and  next_state_label $s_b$ then the second time step might have  state_label $s_b$ and  next_state_label $s_c$ etc - or you could make up a state vector for each one (because that's what you'd have for input to the NN). I am asking because I want you to add this information in the way that you understand it, so that the answer can explain things to you in your own terms.

I have added the stated. Can you now explain it to me?

Let us continue this discussion in chat.

Most Deep Q-learning implementations I have read are based on Deep Q-Networks (DQN).  In DQN, the q-value network maps an input state to a vector of q-values, one for each action:
$$
Q(s, \mathbf{w}) \to \mathbf{v}
$$
where $s$ is the input state from the environment, $\mathbf{w}$ are the parameters of the neural network, and $\mathbf{v}$ is a vector of q-values, where $v_i$ is the estimated q-value of the ith action. In the Sutton and Barto book, the q-value function is written as $Q(s, a, \mathbf{w})$, which corresponds to the network output for action $a$.
Unlike tabular Q-learning, Deep Q-learning updates the parameters of the the neural network according to the gradients of the loss function with respect to the parameters. DQN uses the loss function
$$
L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2
$$
where $\gamma$ is the discount rate, $a$ is the selected action (either greedily or randomly for an $epsilon$-greedy behavior policy), $s'$ is the next state, $a'$ is the argmax action for the next state, and $\mathbf{w^-}$ is an older version of the network weights $\mathbf{w}$ that is used to help stabilize training.
In deep Q-learning, training directly updates parameters, not q-values. Parameters are updated by taking a small step in the direction of the gradient of the loss function
$$
\mathbf{w} \gets \mathbf{w} + \alpha [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})] \nabla_w Q(s, a, \mathbf{w})
$$
where $\alpha$ is the learning rate.
In frameworks like tensorflow or pytorch the derivative is calculated automatically by giving the loss function and model parameters directly to an optimizer class which uses some variation of mini-batch gradient descent. In eagerly executed tensorflow updating the parameters for a mini-batch might look something like
batch = buffer.sample(batch_size)
observations, actions, rewards, next_obervations = batch

with tf.GradientTape() as tape:
    qvalues = model(observations, training=True)
    next_qvalues = target_model(next_obervations)
    # r + max_{a'} Q(s', a') for the batch
    target_qvalues = rewards + gamma * tf.reduce_max(next_qvalues, axis=-1)
    # Q(s, a) for the batch
    selected_qvalues = tf.reduce_sum(tf.one_hot(actions, depth=qvalues.shape[-1]) * qvalues, axis=-1)
    loss = tf.reduce_mean((target_qvalues - selected_qvalues)**2)

grads = tape.gradient(loss, model.variables)
optimizer.apply_gradients(zip(grads, model.variables))

Though I am not familiar with the Encog neural network framework you are using, based on the example Brain.java file from your Github repo and Chapter 5 of the Encog User Manual and the Encog neural network examples on Github it looks like weights are updated as follows:

A training set is constructed from pairs of input and target output.
A Propagation instance, train, is constructed with a network and training set. Different subclasses of Propagation use different loss functions to update the network parameters.
The method train.iterate() is called to run the network on the inputs, calculate the loss between the network outputs and target outputs, and update the weights according to the loss.

For DQN, a training set is constructed from a random sample from the experience replay buffer to help stabilize training. A training set could also be the trajectory of an episode, which is what the tupels argument in the example code of the question appears to be.
The input would be the statefirst member of each element of tupels. Since the network produces a vector of q-values, the target output must also be a vector of q-values.
The target output element for the selected action is $r + \gamma max_{a'} Q(s', a', \mathbf{w^-})$,  In the example code of the question, this is
double qnew = 0;
if(i <= tupels.size()-2){
    qnew = tupels.get(i).rewardafter + discountfactor*qMax(tupels.get(i+1));
} else {
    qnew = tupels.get(i).rewardafter;
}
tupels.get(i).qactions.elements[tupels.get(i).actionTaken] = qnew

The target output elements for actions that were not selected should be $Q(s, b, \mathbf{w})$, where $b$ is one of the non-selected actions.  This should have the effect of ignoring the q-values of non-selected actions by making the network output equal to the target output.
So what are the new Q - Values, assuming a discount factor of 0.99 and the learning rate 0.1?
Assuming you mean target outputs by the new Q - Values, and given the trajectory of actions, (1, 1, 1), and q-value vectors from the question, the concrete target outputs are (0, 0 + 0.99 * 0, -5, 0), (0, 0 + 0.99 * 0, 0, 0), and (0, 1 + 0, 0, 0).

$$w←w+α[(r+γmaxa′Q(s′,a′,w−))−Q(s,a,w)]∇wQ(s,a,w)$$
Does this mean I have to hold always an older version of the network?
I do not understand this. I understand what the loss function does. It is the normal loss function for most BP-algorithms. But how to calculate:
$$maxa′Q(s′,a′,w−))$$

Yes, the DQN method uses an older version of the model which gets replaced every N time steps by the current model. According to the paper this improves training.

You could calculate $max_{a'} Q(s', a', w^-)$ by running the state s' through an older copy of the model (i.e. the target network) and choosing the max of the outputs.

Unfortunately, this still does not work. Is there anything else to consider. I must have done an obvious mistake.


text:
Can an ML model sort a random sequence of numbers from 1 to $ 2^{2^{512}} $ in our universe in infinite time?
I am pondering on the question in the title. As a human being, somehow I can sort a random sequence of numbers from 1 to $ 2^{2^{512}} $  in our universe in infinite time (But I am not sure.). Can an ML model do that in our universe if it is provided with infinite time? There is no restriction on how the learning algorithm is supposed to learn how to sort. (Be careful, even $ 2^{512} $ is bigger than the number of atoms in the universe. Therefore you will have limited memory.)

Are you basically asking whether a ML algorithm can learn to sort? What's the point of the range if you give the algorithm an infinite amount of time, can you elaborate a bit on that?

@SpiderRico The point is that the number is very huge which cannot be represented by bits. I try to compare human cognition and ML models. It seems (not sure) that the computer cannot sort such big numbers because, after a point, the numbers cannot be represented even if you use all atoms in the universe to represent numbers. Can a human solve the problem without time limitation?

So you assume a memory limit then? Isn't then the question essentially boils down to whether we can store such a number in memory or not? I just can't see the ML part of the question here.

The question is related to machine learning because I'm asking if an ML algorithm could theoretically learn how to sort any sequence of numbers (i.e. if it could learn some sorting algorithm).

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Inaccurate masks with Mask-RCNN: Stairs effect and sudden stops
I've been using matterport's Mask R-CNN to train on a custom dataset. However, there seem to be some parameters that i failed to correctly define because on practically all of the images, the bottom or top of the object's mask is cut off:

As you can see, the bounding box is fine since it covers the whole blade, but the mask seems to suddenly stop in a horizontal line on the bottom.
On another hand, there is a stair-like effect on masks of larger and curvier objects such as this one (in addition to the bottom and top cut-offs):

The original images are downscaled to IMAGE_MIN_DIM = IMAGE_MAX_DIM = 1024 using the "square" mode.
USE_MINI_MASK is set to true with MINI_MASK_SHAPE = (512, 512) (somehow if i set it off, RAM gets filled and training chrashes).
RPN_ANCHOR_SCALES = (64, 128, 256, 512, 1024) since the objects occupy a large space of the image.

It doesn't feel like the problem comes from the amount of training. These two predictions come from 6 epochs of 7000 steps per epoch (took around 17 hours). And the problem appears from early stage and persists along all the epochs.
I posted the same question on stack overflow, and an answer pointed out that this issue is common when using mask r-cnn. It also suggested to look at PointRend, an implementation to mask r-cnn that addresses this issue.
Nevertheless I feel like I could still optimize my model and use the full potential of mask r-cnn before looking for an alternative.
Any idea on what changes to make ?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How does the regression layer in the localization network of a spatial transformer work?
I am trying to understand the spatial transformer network mentioned in this paper https://papers.nips.cc/paper/5854-spatial-transformer-networks.pdf. I am clear about the last two stages of the spatial transformer i.e. the grid generator and sampler. However I am unable to understand the localization network which outputs the parameters of the transformation that is applied to the input image. So here are my doubts.

Is the network trained on various affine/projective transforms of the input or only the standard input with a standard pose?
If the answer to question 1 is no, then how does the regression layer correctly regress the values of the transformation applied to the image? In other words how does the regression layer know what transformation parameters are required when it has never seen those inputs before?

Thanks in advance.

Localization network is not trained separately on special transform of input. It's just a part of feed forward network which is trained as a whole, with normal backpropagation.
It's just part of the network which affect the loss function. As in any backpropagation loss function propagate back gradient, which backpropagate through final part of the network, backpropagate through differentiable sampler(that is non-trivial part, which use transfromation produced by localization subnetwork) and after that backpropagate into localization part
Whole approach of spatial transformers could be in doubt now. There were some anecdotal evidences that it was not working on less trivial tasks (private communications, was not working for me either)

Thanks for the reply. However my doubt still stands. How does the regression layer output the exact transform parameters in a previously unseen image?  It seems magic to me. Am I missing something about regression which the localization network uses?

It's called "regression layer" because it produce not discrete set of probabilities but continuous parameters. It's just common fully connected layer. Nothing special about it. And it seems whole method work only on transformation of simple flat images like road signs. It's not so good for affine approximation, not a magical solution


text:
Does an advanced Dialogue state tracking eliminate the need of intent classifier and slot filling models in dialogue systems/ chatbots?
I am learning to create a dialogue system. The various parts of such a system are Intent classifier, slot filling, Dialogue state tracking (DST), dialogue policy optimization and NLG.
While reading this paper on DST, I found out that a discriminative  sequence model of DST can identify goal constraints, fill slots and maintain state of the conversation.
Does this mean that now I dont need to create an intent classifier and slot filling models separately as the tasks are already being done by this DST? Or I am misunderstanding both the things and they are separate?


text:
How to design the reward for an action which is the only legal action at some state
I am working on a RL project,but got stuck at one point: The task is continuous (Non-episodic). Following some suggestion from Sutton's RL book, I am using a value function approximation method with average reward (differential return instead of discount return). For some state (represented by some features), only one action is legal. I am not sure how to design reward for such action. Is it ok to just assign the reward in the previous step? Could anyone tell me the best way to decide the reward for the only legal action. Thank you!
UPDATE:
To give more details, I added one simplified example:
Let me explain this by a simplified example: the state space consists of a job queue with fix size and a single server. The queue state is represented by the duration of jobs and the server state is represented by the time left to finish the current running job.  When the queue is not full and the server is idle, the agent can SCHEDULE a job to server for execution and see a state transition(taking next job into queue) or the agent can TAKE NEXT JOB into queue. But when the job queue is full and server is still running a job, the agent can do nothing except take a BLOCKING action and witness a state transit (time left to finish running job gets decreased by one unit time). The BLOCKING action is the only action that the agent can take in that state.

@DukeZhou Thanks, please see the update in question


text:
Do we have to consider the feasability of an action when defining the reward function of a MDP?
Do we have to consider if (s is given) an action a can lead to s' when defining a reward function?
To be more specific:
Let's say I have a 1D Map like: |A|B|C|D|
To define a reward function, I simply defined a matrix for every action, where the columns and rows represent the states (A-D) and the entries represent the reward. But I made it even simpler. As reaching a specific state gives a reward of 1 I just assigned the reward to a column (C).
$$
\begin{matrix} -1&0&1&0\\ -1&0&1&0\\-1&0&1&0\\-1&0&1&0 \end{matrix}
$$
However, lets say the matrix is specified for the action "going right". Now there are entries reading: D ==> going right ==> C getting reward of 1. This is actually not possible. However, I thought the transition function would handle this issue since I will define there what is possible and what is not. But anyway it is said that for a horizon of one the immidiate reward is given and the transition function isn't even considered. This leads to arbitrary result. So do I have to consider the "physics" of my world?

Regarding " But anyway it is said that for a horizon of one the immidiate reward is given and the transition function isn't even considered" - could you reference where you have read this. It is entirely wrong, but perhaps it is taken out of context or a misunderstanding

Your issue is related to how you are representing your rewards, and not anything to worry about for the MDP.
You have chosen to use a matrix to represent your reward function, which maps $(s,s')$ to $r$. If some transition $s \rightarrow s'$ doesn't happen, then it doesn't matter at all what you put there for the reward value for $(s,s')$. It is only because you have decided to use a matrix for this that you even need to think about it.
A complete reward function would map $s, a, s'$ to a scalar value, and can also include a random factor. You don't need to use all the factors - a MDP can use any or all of them, and all the MDP theory is still correct. In your case, it looks like arriving in a "goal state" $s' = C$ is what triggers a +1 reward. So instead of a matrix you could define a function:
$$r(s,a,s') = \begin{cases}
    -1,& \text{if } s' = A\\
    1,& \text{if } s' = C\\
    0,              & \text{otherwise}
\end{cases} $$
This is equally valid in the MDP as your matrix, and avoids the issue of storing data about impossible transitions.

I thought the transition function would handle this issue since I will define there what is possible and what is not.

Yes that should be the case. If your transition function does not allow for certain state changes, then there is no need to handle them in any particular way in the reward function.

But anyway it is said that for a horizon of one the imm[e]diate reward is given and the transition function isn't even considered

I am not sure where you have read this, but it does not relate to your representation issue or resolving "impossible" state transitions.
If you are using a model-based method, such as Policy Iteration or Value Iteration, then you do use the transition function and reward function - more or less directly in the form of the Bellman equations. In this case, the transition function will assign a weight of 0 to the impossible transitions, so the reward value doe not matter (and efficient code would likely skip even looking up or calculating the reward in that case).
If you are using a model-free method, such as Q learning, then the agent does not use the transition function or the reward function directly. They are part of the environment that it is learning. However, any code for the simulated environment has to implement the physics of your world, and that includes using the transition function to resolve what happens when the agent takes an action. In that case, the simulated model of the environment would prevent the agent ever experiencing impossible transitions, so there is never any need to calculate the reward for them or allow for them as edge cases in a reward function (you might still choose to check validity and/or return something as defensive programming of the reward function, but it is not required for the MDP and reinforcement learning to work).


text:
Are there any pathfinding algorithms that take customized rules into account when determining the shortest path?
I need a pathfinding algorithm that considers the history of visited nodes and varies its path depending on some rules (like already visited). Are there good approaches serving this purpose?
To be more specific: Let's say I have a graph representing a map. I want to find a Route from B to D: Once I am in D following B->C->D, I want to calculate a new path, let's say to A. The shortest path would be D->C->B->A. But I want a path with unvisited nodes even if it's longer than the shortest possible path. The new path should be the shortest among all possible paths according to the rules.
Another example is the game "snake". Seeing the grid as a graph I cannot visit already visited nodes as fas as the corpus (of the snake) is in that node (or I have visited the node t time steps ago)
Maybe the problem is too specific and I have to implement some basic pathfinding algorithm in a wider algorithm.

You could just find any implementation of the A* algorithm and modify it by adding a 'visited' flag to each node. Then exclude moves to nodes that have this flag set.

@OliverMason I was thinking of something similiar. Like having a hashed value that lists all visited nodes as values, but I was hoping for a well known approach that can be used with more rules like "if this kind of node was visited twice, dont visit it again". And if possible not to use customized solutions but rather go with something known. I was thinking of using a signal interpreted petri net and even PDDL but both are not suitable for my purpose.

Yes, this is easily solved using the A* algorithm. Once your agent has visited a particular node, increase the cost of that node to infinity and recalculate the path.


text:
What are AI use cases for communication service providers?
I am currently looking for AI use cases for Telco. What are the different AI use cases for Telcos/communication service providers?

AI could hold the key in automating and optimizing networks. On the subscriber side, ML and AI will assist telecom operators in profiling the subscribers. This will be achieved by analyzing network activity, conversion rate of offers and data usage trends.
Below are a few use cases and how they will transform the telecommunication sector. (Source H2o.ai blog https://www.h2o.ai/telecom/ )
Old generation telecom technologies.

Reactive Maintenance
Network optimization with human intervention
Centralized intelligence
Security attack repair
Backlogged customer tickets

Future generation AI based telecom technologies.

Predictive Maintenance
Self-optimizing network
Optimal network quality
Intelligence at the edge
Security attack prediction
Improved customer experience through customer service chat bots.
Speech and voice services for customer which allows users to explore media content by spoken word rather than remote control.
Predictive maintenance which is the ability to fix problems with telecom hardware such as cell towers, power lines e.t.c before they happen by detecting signals that usually lead to failure.

AI can also be used by Telcos to:
(1) Improve Usage & Retention efforts: Make relevant up-sell and cross-sell offers to the right users at the right time.
(2) Make segmentation more granular: make bespoke recommendations based on a user’s behavioral patterns and content preferences and assessing which call & data packages best suit different customer segments thereby increasing sales success rates

Well AI can be used in Major Areas in Telecom
1/ Network Optimization
    Where the network is trained to come up with the required Parameter tunning to solve a particular issue, for example in 4G system we have user licenses for cell level that how many users this cell will support. Every time the number of users increases on that cell the Network optimizer sends a work order to the Backoffice to increase the number of users with temporary license that is in pool. For an AI system it can analyze the utilization and increase the licenses or it can tune the parameters to shift traffic on the neighbour cell, it can learn from the optimizers behaviour and actions, this is a single example

2/ Customer Intelligence
     Based on the customer behavior, his social media engagements and responses, usage patterns, past usage history and current context  the AI System can generate personalized offers for the user that can add value to the overall customer Journey and make him and engaged happy customer.

3/ Network expansions and Dynamic capacity:
     Mostly  network operators have licenses per resource contracts with vendors, the excess capacity is charged. In most cases for example in summers some particular areas need more additional capacity but once the summers are gone that particular area is the lowest utilized area. An AI system can learn based on the usage patterns when a particular area need higher capacity and when low capacity and can increase and decrease the capacity resulting in reduced costs.

Above are some of the use cases which can further be added with alot of Customer Engagement and Customer Experience Management uses cases
Regards
Muhammad Azfar
Customer Journey and Business Analyst
Customer Experience Management

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What is the fundamental difference between neural networks for classifying and generating data
What is the fundamental difference between NN for classifying data and generating data?
Most examples show how neural networks can be used to classify data. Like is it an image of a dog or a cat. However, there are applications where NN are used to create images and even write short stories.

The formal name for this difference is "generative" vs "discriminative" models.
By default, a supervised learning process using a simple feed-forward neural network and a set of training data with expected answers will produce a discriminative model. It is hard to use such a model to generate content directly.
Differences between discriminative and generative models tend to be at the architecture level or deeper, although a few generative techniques are simple variants or re-purposing of discriminative networks. There is no single "fundamental" difference in terms of NN design. About the only common theme is that generative models are more complex and harder to work with than discriminative ones.
Images

Probably the most popular currently for image generation are Generative Adversarial Networks or GANs. These train a Generator to create an image from random inputs, alongside a Discriminator that tries to spot fakes compared to real images. Training them together results in a generator network that gets progressively better at creating fake data.

Also suitable for image generation are Variational Autoencoders (VAEs) which learn essentially by compressing a set of images down into a small representation, and as a result become able to "uncompress" similar representations.

There are also VAEGANs, which combine VAEs and GANs

There are other models which generate examples from a dataset. For example, Restricted Boltzmann Machines (RBMs) are similar to neural networks, but the neurons fire randomly, and RBMs require a different training process to NNs.

GANs, VAEs, VAEGANs, RBMs can be used to generate data in any simple non-sequential datasets, they are not restricted to just images, but recent work with GANs for example has excelled there.

For a different kind of generator, you can look at how Deep Dream works. The interesting thing here is that it is a modification of a feed forward network that has been trained by supervised learning. Essentially to run Deep Dream, you take an existing image and change it so that it maximises some internal part(s) of the existing neural network, "training" the image as if it were a set of weights.

The difference between Deep Dream and other generative techniques is that it does not generate content that could be in the training set - it is no good for creating "realistic" images.

Text

Recurrent Neural Networks, and specifically Long Short-Term Memory (LSTM) networks can be used to model word sequences by training them to predict the next word. Because the resulting model is probabilistic (e.g. chance of next work being "Hello" is ~0.001%) then a simple method to make this generative is to sample from the predictions randomly, and then feeding them back into the sequence to see what it will predict next. There is a great blog post about this technique by Andrei Karpathy.

Audio and Music

DeepMind's WaveNet architecture is similar to a CNN predicting the next samples of audio from a given input. It can be used in speech or music generation.

LSTMs are a popular choice here, as well as for language models.

One important caveat. Nearly all these models are hard to understand, work with and train, compared to simpler supervised techniques. Probably the easiest to get to grips with if you want to understand them well enough to implement your own versions, are Deep Dream, VAEs and LSTMs.
It is also worth noting that there are many other ways to generate content than neural networks. For instance, sound and music generation has a long history completely independent of recent AI developments.

I would say that the sentence "The formal name for this difference is "generative" vs "discriminative" models." is a little misleading. Generative and discriminative models are not necessarily NNs (and given the context of the question). I would rephrase it to say that, in general, there two families of algorithms that can be used for classification: generative and discriminative models. Maybe you meant something else.

@nbro: Except you would not use generative models for classification. I'll try to re-phrase the first sentence.


text:
Ensemble models - XGboost
I am building 2 models using XGboost, one with x number of parameters and the other with y number of parameters of the data set.
It is a classification problem.
A yes-yes, no-no case is easy, but what should I do when one model predicts a yes and the other model predicts a no ?
Model A with x parameters has an accuracy of 82% and model B with y parameters has accuracy of 79%.

You haven't given any information on how well these two models perform.  Show their ROC curves, confusion matrices, accuracies, etc.  Once you summarize how the two models perform you can pick the better of the two and ignore the other one.

@siddharth I approved the edit, but can I ask, are siddharth and Siddharth Srivastava the same person?  (If so, please consider merging the two accounts:)

Without any additional information, lean towards the vote of the best performing classifier when it comes to ties.
However, as others have stated already, it is best to analyze the performances in more detail (e.g. confusion matrices).
For instance, it could be that model B almost always classifies class X correctly (hardly any false positives). In that case, you could lean towards the prediction of model B if it predicts class X and model A does not. In other words; you could weigh the votes of the models based on how well they did in similar, previous predictions.

Given only the fact that model A has a higher accuracy than model B you should just use model A.  More information on the performance of the two classifiers should be provided for a better answer.

But what is the margin of error?  imho I don't see a lot of difference between 82% and 79%...

We don't have any information to determine a 'margin of error'.  Without any other performance data I would say 82% is 3% more than 79% which is enough to be the winner.  With more detailed information from Siddarth we could make a better analysis.  Accuracy is not a lot to make a determination especially when the application is not known.  I have seen high accuracy classifiers get beaten by much lower accuracy classifiers.


text:
Face recognition model loss not decreasing
I wrote a script to do train a Siamese Network style model for face recognition on LFW dataset but the training loss doesnt decrease at all. Probably there's a bug in my implementation. Could you please point it out.
Right now my code does:

Each epoch has 0.5M triplets all generated in an online way from data (since the exhaustive number of triplets is too big).
Triplet sampling method: We have a dictionary of {class_id: list of file paths with that class id}. We then create a list of classes which we can use for positive class (some classes have only 1 image so cant be used as positive class). At any iteration we randomly sample a positive class from this refined list and a negative class from the original list. We randomly sample 2 images from positive (as Anchor or A and Positive as P) and 1 from negative (Negative or N). A,P,N form our triplet.
Model used is ResNet with the ultimate (512,1000) softmax layer is replaced with (512,128) Dense layer (no activation). To avoid overfitting, only the last Dense and layer4 are kept trainable and rest are frozen.
During training we find triplets which are semi-hard in a batch (Loss between 0 and margin) and use only those to do backprop (they mention this in the FaceNet paper)

from torchvision import models, transforms
from torch.utils.data import Dataset, DataLoader
import torch, torch.nn as nn, torch.optim as optim
from torch.utils.tensorboard import SummaryWriter
import os, glob
import numpy as np
from PIL import Image

image_size = 224
batch_size = 512
margin = 0.5
learning_rate = 1e-3
num_epochs = 1000

model = models.resnet18(pretrained=True)
model.fc = nn.Linear(model.fc.in_features, 128, bias=False)

for param in model.parameters():
  param.requires_grad = False
for param in model.fc.parameters():
  param.requires_grad = True
for param in model.layer4.parameters():
  param.requires_grad = True

optimizer = optim.Adam(params=list(model.fc.parameters())+list(model.layer4.parameters()), lr=learning_rate, weight_decay=0.05)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = nn.DataParallel(model).to(device)
writer = SummaryWriter(log_dir="logs/")

class TripletDataset(Dataset):
  def __init__(self, rootdir, transform):
    super().__init__()
    self.rootdir = rootdir
    self.classes = os.listdir(self.rootdir)
    self.file_paths = {c: glob.glob(os.path.join(rootdir, c, "*.jpg")) for c in self.classes}
    self.positive_classes = [c for c in self.classes if len(self.file_paths[c])>=2]
    self.transform = transform

  def __getitem__(self, index=None):
    class_pos, class_neg = None, None
    while class_pos == class_neg:
      class_pos = np.random.choice(a=self.positive_classes, size=1)[0]
      class_neg = np.random.choice(a=self.classes, size=1)[0]

    fp_a, fp_p = np.random.choice(a=self.file_paths[class_pos], size=2, replace=False)
    fp_n = np.random.choice(a=self.file_paths[class_neg], size=1)[0]

    return {
        "fp_a": fp_a,
        "fp_p": fp_p,
        "fp_n": fp_n,
        "A": self.transform(Image.open(fp_a)),
        "P": self.transform(Image.open(fp_p)),
        "N": self.transform(Image.open(fp_n)),
            }

  def __len__(self):
    return 500000

def triplet_loss(a, p, n, margin=margin):
    d_ap = (a-p).norm(p='fro', dim=1)
    d_an = (a-n).norm(p='fro', dim=1)
    loss = torch.clamp(d_ap-d_an+margin, min=0)
    return loss, d_ap.mean(), d_an.mean()

transform = transforms.Compose([
        transforms.RandomResizedCrop(image_size),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.596, 0.436, 0.586], [0.2066, 0.240, 0.186])
        ])
train_dataset = TripletDataset("lfw", transform)
nw = 4 if torch.cuda.is_available() else 0
train_dataloader = DataLoader(train_dataset, batch_size=batch_size, num_workers=0, shuffle=True)

num_batches = len(train_dataloader)
model.train()
running_loss = 0

for epoch in range(num_epochs):
    for batch_id, dictionary in enumerate(train_dataloader):
        a, p, n = dictionary["A"], dictionary["P"], dictionary["N"]
        a, p, n = a.to(device), p.to(device), n.to(device)
        emb_a, emb_p, emb_n = model(a), model(p), model(n)
        losses, d_ap, d_an = triplet_loss(a=emb_a, p=emb_p, n=emb_n)

        semi_hard_triplets = torch.where((losses>0) & (losses<margin))
        losses = losses[semi_hard_triplets]
        loss = losses.mean()
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

        print("Epoch {} Batch {}/{} Loss = {} Avg AP dist = {} Avg AN dist = {}".format(epoch, batch_id, num_batches, loss.item(), d_ap.item(), d_an.item()), flush=True)
        writer.add_scalar("Loss/Train", loss.item(), epoch*num_batches+batch_id)
        writer.add_scalars("AP_AN_Distances", {"AP": d_ap.item(), "AN": d_an.item()}, epoch*num_batches+batch_id)

    print("Epoch {} Avg Loss {}".format(epoch, running_loss/num_batches), flush=True)
    writer.add_scalar("Epoch_Loss", running_loss/num_batches, epoch)
    torch.save(model.state_dict(), "facenet_epoch_{}.pth".format(epoch))

Loss graphs: https://tensorboard.dev/experiment/8TgzPTjuRCOFkFV5lr5etQ/
Please let me know if you need some other information to help you help me.


text:
Can AlphaFold predict proteins with metals well?
There are certain proteins that contain metal components, known as metalloproteins. Commonly, the metal is at the active site which needs the most prediction precision. Typically, there is only one (or a few) metals in a protein, which contains far more other atoms. So, the structural data that we could be used to train AlphaFold will contain far less information about the metal elements. Not to mention most proteins don't have metals at all (it is estimated that only 1/2-1/4 of all proteins contain metals [1]).
Given that maybe there is not enough structural data about protein local structure around metal atoms (e.g. Fe, Zn, Mg, etc.), then AlphaFold cannot predict local structure around metals well. Is that right?
I also think that the more complex electron shell of metal also makes the data less useful, since its bounding pattern is more flexible than carbon, etc.

I've edited your post to add more details based on your comments. Can you explain this sentence "Commonly, the metal is at the active site which needs the most prediction precision"? Why do you say that it needs "the most prediction precision"? Maybe you could look at the exact proteins used to train the AlphaFold in the CASP competition. Maybe there's some info about the type of proteins used to train AlphaFold.

@nbro The prediction precision for the active-site will be much more important than for other parts of the protein, because the active site is where the most "action" takes place.

Let me address first some of the things you wrote in your question:

There are certain proteins that contain metal components, known as
metalloproteins.

Natural proteins do not ever contain metal components as far as we know. Natural proteins are composed of natural amino acids which only contain H,C,O,N,S. Selenocysteine contains Se (also a non-metal!) but it's a proteinogenic amino acid, which means it's a precursor to proteins but doesn't typically show up in the protein itself. From the Wikipedia page that you gave us in your question: "Metalloprotein is a generic term for a protein that contains a metal ion cofactor" and "A cofactor is a non-protein chemical compound or metallic ion".
But your question still deserves an answer, because even though it would incorrect to call a co-factor "part" of the protein, they can still affect the folding and the overall shape. But let's first address one last part of your question:

Given that maybe there is not enough structural data about protein
local structure around metal atoms (e.g. Fe/Zn/Mg), then AlphaFold
cannot predict local structure around metals well. Is that right?

The first sentence of the Wikipedia article that you linked in your question, says "For instance, at least 1000 human proteins (out of ~20,000) contain zinc-binding protein domains [3] although there may be up to 3000 human zinc metalloproteins [4]." Therefore, while metalloproteins might not be the majority of all proteins, there's a decent enough number of them that are of relevance to the human body, and therefore constructing training databases that contain enough metalloproteins (or even 100% metalloproteins, if desired) is not difficult.
I mentioned a bit elsewhere that AlphaFold was used to predict protein structures in the CASP competition, for which you can see for yourself that many/most of the proteins for which contestants (such as DeepMind) need to predict the structure, come from studies of proteins of relevance to humans because the CASP structures typically come from X-ray crystallography studies, which are typically done on proteins of relevance to humans.
You can also see for yourself not only the "target list" that I showed above, but also the results of the competition which will show how well AlphaFold performed in CASP13 (2018) and CASP14 (2020) for metalloproteins.
Finally:

I also think that the more complex electron shell of metal also makes
the data less useful, since its bounding pattern is more flexible than
carbon, etc.

It is true that metals are typically harder to model than C,H,O,N,S, and even Se, if doing ab initio calculations on the metals or metail-containing complexes. However, the purpose of machine learning in protein folding studies is to skip ab initio, statistical-dynamical and/or molecular-dynamical calculations of the relevant structures and simply use training data to predict the protein structures. That being said, there needs to be enough training data available (as you correctly pointed out) to learn what happens near the metal co-factors: The answer to this is that there are indeed enough metalloproteins to sufficiently populate a training set, but they won't contain enough of the specific metals involved in every metalloprotein. For example, lots of data will be available for proteins containing Fe since Fe is in hemoglobin (for example) which is essential to the functioning of red blood cells to absorb oxygen, but the protein vanabins contains vanadium which is much rarer and therefore training data involving it will be much less available. You're also correct that metal elements can form more bonds than typical elements found in organic compounds.
So it depends on the metal in the relevant co-factor. Fe-based co-factors will have quite a lot of training data available, as well as Mg-based ones, Zn-based ones, and a lot of other ones which contain the "more common" metals. For proteins like vanabins which contains vanadium, you are quite correct that training data will be limited, but also keep in mind that vanabins is a very rare protein found in sea squirts and we already know more about its structure (through X-ray crystallography, which means we don't need machine learning for it) than we even know about what it even does. The chances of other vanadium-containing co-factors in metalloproteins being very significant is too low to justify working on protein folding algorithms specifically for them.

The part "There are certain proteins that contain metal components, known as metalloproteins." was written by me to provide some introduction, so, please, do not address that in this answer, but please propose an edit to the original post to fix that description (if you really think it's wrong). I wrote "parts" to provide a generic/general description, given that the OP had not provided any description initially.

Well you shouldn't be injecting scientifically incorrect edits into other people's posts if you don't know enough about the topic. I just spent a couple hours on the answer and got myself late for something, so I was about to leave and will have to make those edits later.

Where did I inject scientifically wrong info? The first sentence of the first section of that article says "It is estimated that approximately half of all proteins contain a metal.". I wrote "components" and "parts" and I don't know why you think that's wrong as a generic (of course, not very precise) description (but this site is not about biological concepts, so you should expect some imprecision). See also https://en.wikipedia.org/wiki/Rubredoxin, for example. I also spent a lot of time trying to clarify the original question, which was not answerable at all in the beginning.

So, not sure where your conclusion "Natural proteins do not ever contain metal components as far as we know" comes from. For instance, here we have "Metalloproteins are commonplace in biological systems.". It's true that I am not expert in this specific topic, but I studied biology in high school a few years ago, so my knowledge of biological concepts is not completely zero. I have an idea of what an atom, ion, protein, amino acid, etc., are.

“Typically, there is only one (or a few) metals in a protein, which contains far more other atoms. So, the structural data that we could be used to train AlphaFold will contain far less information about the metal elements” That's the main point of the question, but you didn't address this one

IMO cofactor has two types, first is a protein or small molecule that use normal weak interaction to bind to the main protein. The second is metal like, which use metal-sepecific coordination bound to bind to the main protein. Although all called cofactor, but they are not very same. IMO the metal cofactor is better to be considered as a must-have structural component of the protein, not a "factor" that modify a protein's properties when bound

About the link of competition result you provide: (1) The question ask about the precision around the metal, I'm not sure whether the competition result is average precision over the whole protein. (2) One reason of asking the question is the result is too domain specific, not easy for outsiders like me to understand the metrics and methods


text:
Are calculus and differential geometry required for building neural networks?
I've been studying geometry and linear algebra for months with the goal to build neural networks. But now I'm reading that perceptrons require fitting curves, and curves are not expressed as linear functions. So, I might need to study differential geometry and calculus for building good fitting curves in perceptrons.
I already know how to code and was hoping to get my hands dirty by coding a few neural networks. But should I study calculus and differential geometry before coding?
From this video, I understand that the least squares approximation can be used to fit a curve through a set of points, so maybe linear algebra is enough for building good neural networks?

What level are you hoping to build neural networks at? You only need a brief overview of calculus and statistics if you want to use a toolkit like Keras or PyTorch. If you want to understand all the details, and build everything yourself in C++ or similar, then your need more.

Building it from scratch. Not using any external library. I've already studied statistics and probability. Ok so I need Calculus.

As a side note, "perceptrons" and "neural networks" may not be the same thing. People usually use the term perceptron to refer to a very simple neural network that has no hidden layer. Maybe you meant the term "multi-layer perceptron" (MLP).

@nbro FYI: I asked a new question based on your comment.

Neural networks are essentially just repeated matrix multiplications and applications of an activation function, so you really don't need a great deal of linear algebra to construct a simple neural network — if you understand how to multiply matrices, that's probably sufficient.
The harder bit is the training process which is typically done through backpropagation. You need a bit of calculus, but differential geometry is overkill. There are some interesting topics in differential geometry for machine learning, but it's far beyond what is needed to implement backpropagation.
To understand backpropagation you just need to know about the gradient of a function and what this means intuitively; you should also have a good knowledge of the chain rule. That's really all you need, and any course on "multivariable calculus" or something similar would give you more than enough to get started.
Of course, it never hurts to know more, but fortunately neural networks are simple
enough that you don't need to struggle for years before you can implement a basic neural network; try to get started as soon as you have the basics, and learn the rest as you go.

To give some practical advice, it is important to understand parts of calculus. This is mainly because Backpropagation is a leaky abstraction in modern libraries. In a nutshell, there is a lot  which can go wrong (exploding or vanishing gradient for example) and you will need knowledge about gradient descent to handle it.
I highly recommend Andrej Karpathys Lecture on it. He gives a easy to understand and intuitive explanation.


text:
How to normalize for perceptual loss when training neural net from scratch?
Let's say we are training a new neural network from scratch. I calculate the mean and standard deviation of my dataset (assume I am training a fully convolutional neural net and my dataset is images) and I standardise each channel of all images based on that mean and standard deviation. My output will be another image.
I want to use for example VGG for perceptual loss (VGG's weights will be frozen). Perceptual loss is when you input your prediction to a pretrained network to extract features from it. Then you do the same for the ground truth and the L2 distance between the features from ground truth and features from prediction is called perceptual loss.
As far as I know, I am supposed to standardise my data based on the mean and standard deviation VGG was trained with (since I am using VGG for inference essentially), which is different than the mean and standard deviation of my dataset. What is the correct way to do this? Should I undo the standardization of my dataset by multiplying standard deviation and adding the original mean to the output of my network, and then restandardise using VGG's statistics to calculate the loss? Or should I continue without restandardising?

I am a bit confused with the part of "training a new neural network from scratch" and then the part of "VGG's weights will be frozen" because the answer changes since they are different cases. It can also happen, from what I read, that you are using VGG as backbone of another network for transfer learning but it is not specified either. Anyway, I will try my best.
The cases I see:

Training a network from scratch: no, you don't need to undo your normalization. In fact, when training from scratch you can normalize however you want.
Training a network from scratch starting from pretrained weights: no, you don't need to change your normalization. Although it would help to faster convergence because the weights are fitted to other kind of normalization. But as the network trains it will fit to your new normalization without any problem.
Training a network that uses VGG as backbone with frozen weights: no, you don't need to change your normalization although it would help. In spite of having your backbone fitted for a specific kind of normalization your network head is still an universal approximator that would fit to any other kind of normalization. However if you use the same normalization you are easing the network's job (that's why we normalize in the first place)

When you really need to force same normalization is in inference. When you want to infer you really need to do the same normalization in preprocessing or you won't have any meaningful results.

Sorry for not making it more clear. I am talking about using VGG as a loss function (aka perceptual loss). This is when you put the output of a network you are training through a pretrained VGG and get some features out of it from an intermediate layer. Then you do the same for your ground truth. Then your loss can be the L2 distance between the extracted features of your prediction and the extracted features of the ground truth

I have modified the question to make it more clear

Then yes, you are using VGG for inference (which in this case is a mean of computing the perceptual loss). You need to standardize your data according to however your VGG net was trained


text:
How is it that AI can become biased, and what are the proposals to mitigate this?
This is not meant to be negative or a joke but rather looking for a productive solution on AI development, engineering and its impact on human life:
Lately with my Google searches, the AI model keeps auto filling the ending of my searches with:
“...in Vietnamese”
And
“...in a Vietnamese home”
The issue is I have never searched for that but because of my last name the model is creating this context.
The other issue is that I’m a halfy and my dad is actually third generation, I grew up mainstream American and don’t even speak Vietnamese. I’m not even sure what a Vietnamese home means.
My buddy in a similar situation of South Asian and noticed the same exact thing more so with YouTube recommended videos.
We already have enough issues in the US with racism, projections of who others expect us to be based on any number of things, stereotyping and putting people in boxes to limit them - I truly believe AI is adding to the problem, not helping.
How can we fix this. Moreover, how can we use AI to bring out peoples true self, talents and empower and free them them to create their life how they like ?
There is huge potential here to harness AI in ways that can bring us more freedom, joy and beauty so people can be the whole of themselves and with who they really are. Then meet peoples needs, wishes, dreams and hope. Given them shoulders to stand on to create their reality, not live someone else's projection of themselves.

@DuttA: Creators of NLP and face recognition algorithms do recognise unwanted bias creeping into their models. These models reflect the biases of the media that they consume, statistically. For instance, if you do word "math" you end up finding the female analog to a male doctor is a nurse (and many other examples where NLP model subtly demotes female roles). There are things that can be done about it at the model-building end, that may also apply here.

@DuttaA: You can remove unwanted bias systematically from a statistical model if you also have a statistical model of the bias. For instance any NLP system that associates non-gendered words (like "doctor") in skewed ways to gender pronouns can have a loss function adjusted to prevent this effect, assuming it is unwanted for the purpose of the model. This can be done, and have seen in a training course - may write an answer about this

@duttaa: I understand that but it is clearly connecting the dots incorrectly to the reality I live in.. and in a constricting and narrow unhelpful way. I don't speak Vietnamese, i can maybe understand a handful of words at best. and I don't even understand what a Vietnamese home entails.

@duttaa: what does "what people of your heritage search" have to do with what I need to search and which of my two major heritages (or 15 based on my 21 & me) ??

Lately with my Google searches, the AI model keeps auto filling the ending of my searches with:
“...in Vietnamese”

I can see how this would be annoying.
I don't think Google's auto-complete algorithm and training data is publicly available. Also it changes frequently as they work to improve the service. As such, it is hard to tell what exactly is leading it to come up with this less-than-useful suggestion.
Your suspicion that it has something to do with Google's service detecting your heritage seems plausible.
The whole thing is based around statistical inference. At no point does any machine "know" what Vietnamese - or in fact any of the words in your query  - actually means. This is a weakness of pretty much all core NLP work in AI, and is called the grounding problem. It is why, for instance, that samples of computer generated text produce such surreal and comic material. The rules of grammar are followed, but semantics and longer term coherence are a mess.
Commercial chatbot systems work around this with a lot of bespoke coding around some subject area, such as booking tickets, shopping etc. These smaller domains are possible for human developers to "police", connecting them back to reality, and avoiding the open-ended nature of the whole of human language. Search engine text autocomplete however, cannot realistically use this approach.
Your best bets are probably:

Wait it out. The service will improve. Whatever language use statistics are at work here are likely change over time. Your own normal use of the system without using the suggestions will be part of that data stream of corrections.

Send a complaint to Google. Someone, somewhere in Google will care about these results, and view them as errors to be fixed.

Neither of these approaches guarantee results in any time frame sadly.

We already have enough issues in the US with racism, projections of who others expect us to be based on any number of things, stereotyping and putting people in boxes to limit them - I truly believe AI is adding to the problem, not helping.

You are not alone in having these worries. The statistics-driven nature of machine learning algorithms and use of "big data" to train them means that machines are exposing bias and prejudice that are long buried in our language. These biases are picked up by machinery then used by companies that don't necessarily want to reflect those attitudes.
A similar example occurs in natural language processing models with word embeddings. A very interesting feature of LSTM neural networks that learn statistical language models is that you can look at word embeddings, mathematical representations of words, and do "word math":
$$W(king) - W(man) + W(woman) \approx W(queen)$$
$$W(he) - W(male) + W(female) \approx W(she)$$
This is very cool, and implies that the learned embeddings really are capturing semantics up to some depth. However, the same model can produce results like this:
$$W(doctor) - W(male) + W(female) \approx W(nurse)$$
This doesn't reflect modern sensibilities of gender equality. There is obviously a deep set reason for this, as it has appeared from non-prejudiced statistical analysis of billions of words of text from all sorts of sources. Regardless of this though, engineers responsible for these systems would prefer that their models did not have these flaws.

How can we fix this. Moreover, how can we use AI to bring out peoples true self, talents and empower and free them them to create their life how they like ?

Primarily by recognising that statistical ML and AI doesn't inherently have prejudice or any agenda at all. It is reflecting back ugliness already in the world. The root problem is to fix people (beyond scope of this answer, if I had solid ideas about this I would not be working in software engineering, but in something more people-focussed).
However, we can remove some of the unwanted bias from AI systems. Broadly the steps toward this go:

Recognise that a particular AI system has captured and is using unwanted gender, racial, religious etc bias.

Reach a consensus about how an unbiased model should behave. It must still be useful for purpose.

Add the desired model behaviour into the training and assessment routines of the AI.

For instance in your case, there are possibly some users of Google's system who would prefer to read articles in Vietnamese, or have English translated into Vietnamese, and are finding it awkward that the default assumption is that everything should be presented in English. These users don't necessarily need to use the search text for this, but presumably are for some reason. A reasonable approach is to figure out how their needs could be met without spamming "in Vietnamese" on the end of every autocomplete suggestion, and perhaps in general move suggestions to localise searches by cultural differences out of autocomplete into a different part of the system.
For the case of gender bias in NLP systems, Andrew Ng's Coursera course on RNNs shows how this can be achieved using the embeddings themselves. Essentially it can be done by identifying a bias direction from a set of words (e.g. "he/she", "male/female"), and removing deviations in that direction for most other words, preserving it only for words where it is inherently OK to reflect the differences (such as "king" and "queen" for gender bias).
Each case of unwanted bias though needs to be discovered by people and oversight of this as a political and social issue, not primarily a technical one.

@DuttaA: Yes I suspect they do have that level of compute power and more, and routinely re-train their main models. However, the decision becomes a more complex cost/benefit analysis, and does not only involve time and electricity, but development work. The specific case of gender bias adjustment does not require re-training as such but is a post-processing step. Actual fixes will vary a lot.

"Statistical ML doesn't have any agenda": check. "Statistical ML doesn't have any prejudice other than that in the data": nope. There are a few papers which prove that, except for very specific cases and even if the data have been correctly collected, a classifier cannot be at the same time calibrated and fair, and if fair under a certain fairness metric, it will be unfair under another one. See https://papers.nips.cc/paper/7151-on-fairness-and-calibration.pdf for one example.

@DeltaIV: Fairness and prejudice are not always opposites. The paper is strongly related to this question, but not talking about quite the same thing as my answer when I refer systemic bias in NLP systems. I will try to incorporate it into the answer though . . .

@neil-slater: amazing answer, thank you. The root issue of fixing humans is eye opening. Sure the models have no agenda but either do bullets. This means there is huge potential here to harness AI in ways that can bring us more freedom, joy and beauty so people can be the whole of themselves and with who they really are. Then meet peoples needs, wishes, dreams and hope. Given them shoulders to stand on to create their reality, not live someone else's projection of themselves.

Another fallacy that appears common to most search engines is that anything a person searches on is an aspect of their own identity. I once searched on walk-in tubs for a very elderly relative, and was followed all over the web by ads for aids for the infirm elderly. Users who recognize that Google uses their searches to build their profile can alter their searches accordingly. It's also fun to mess with Google's model. Try searching on "dragon images" and see how fast Google and advertisers decide you are a teenage female. Have fun with it. Do your best to turn Google's model of you into self-contradictory garbage.

Yeah good point, wtf? Is there an AI app that can just send out random queries throughout the day?

The key I think is teaching the algorythm by providing better data. The only thing an AI can use is the data available for itself. Figuring out whatever it can is not bias, as it's based on objective facts.
If it knows 98% of Nguyens are interested in X, knowing nothing else about you personally, showing you X might be good. If you consistently click on downvote/not interested, etc. buttons on the site, your personal data will override the default, and you won't see X anymore.
As a user you could give better reviews for better results, and as a developer you can give better ways to get this: by logging what you click on, search, and showing "not interested/interested/upvote/downvote/like" etc. buttons.
Note that I'm using youtube from different, unlinked machines/browsers, and I get different suggestions from all of these, probably because I've trained the AI with different data.
You can also use services with less intrusive data collection, e.g. duckduckgo, bitchute, etc.

thanks, good points. would love to see the developers add more feedbac/training hooks for users.

What are the proposals to mitigate [the AI biases]?

I'll answer based on this recent survey, which focuses on Large Language Models (LLMs): Bias and Fairness in Large Language Models: A Survey:
There exists many different types of biases:

There exist many bias mitigation strategies:

Example of Pre-Processing Mitigation Technique:

Example of In-Training Mitigation Technique:

Equalizing Objective Functions for Bias Mitigation:

Example of Intra-Processing Mitigation Technique:

Example Post-processing Mitigation Techniques:

One can refer to the survey for more details.

This is a useful update, but so many screenshots of text


text:
Is there any research work that shows that we should explicitly mark the word boundaries for 1D CNNs?
I'm doing character embedding for NLP tasks using one-dimensional convolutional neural networks (see Chiu and Nichols (2016) for the motivation). I haven't found any empirical evidence of whether or not marking the word boundaries makes a difference. As an example, a 1-D CNN with kernel size 2 would take "the" as input and use {"th", "he"} in its filters. But if I explicitly marked the boundaries it would give me {"t", "th", "he", "h"}.
Is there a go-to paper or project that definitively answers this question?

The idea of "character embedding" comes from Chiu and Nichols (2016). Character embeddings from the CNN are fed into a word-level LSTM.


text:
DQN exploration strategy for large grid-world environment
My task involves a large grid-world type of environment (grid size may be $30\times30$, $50\times50$, $100\times100$, at the largest $200\times200$). Each element in this grid either contains a 0 or a 1, which are randomly initialized in each episode.  My goal is to train an agent, which starts in a random position on the grid, and navigate to every cell with the value 1, and set it to 0. (Note that in general, the grid is mostly 0s, with sparse 1s).
I am trying to train a DQN model with 5 actions to accomplish this task:

Move up
Move right
Move down
Move left
Clear (sets current element to 0)

The "state" that I give the model is the current grid ($N\times N$ tensor). I provide the agent's current location through the concatenation of a flattened one-hot ($1\times(N^2)$) tensor to the output of my convolutional feature vector (before the FC layers).
However, I find that the epsilon-greedy exploration policy does not lead to sufficient exploration. Also, early in the training (when the model is essentially choosing random actions anyway), the pseudo-random action combinations end up "canceling out", and my agent does not move far enough away from the starting location to discover that there is a cell with value 1 in a different quadrant of the grid, for example. I am getting a converging policy on a $5\times5$ grid w/ a non-convolutional MLP model, so I think that my implementation is sound.

How I might encourage exploration that will not always "cancel out" to only explore a very local region to my starting location?
Is this approach a good way to accomplish this task (assuming I want to use RL)?
I would think that attempting to work with a "continuous" action space (model outputs 2 values: vertical and horizontal indices of grid cells that contain 1s) would be more difficult to achieve convergence. Is it wise to always try to use discrete action spaces?

What does your Q-network look like?
If you're using Convolutional Neural Networks (CNNs), I would advise against using pooling layers since the spatial translation data matters in this application (i.e. a cell with '1' top right of the agent is not the same as a cell with '1' below).

1) How I might encourage exploration that will not always "cancel out" to only explore a very local region to my starting location?

This is referred to as exploration vs exploitation dilemma.
One way to add tackle this is by adding extra logic to the way you're choosing random actions. To elaborate, the chosen random action shouldn't be the same as the opposite of the previous action: if the agent went LEFT, the next random shouldn't be RIGHT as that is not exploring.
Another method is to use different action selection strategies such as Boltzmann Policy rather than epsilon-greedy. Click here for a quick crash course on the mainstream strategies by Arthur Juliani.

2) Is this approach a good way to accomplish this task (assuming I want to use RL)?

RL is well suited for this application. Try different parameters for your network. I'd start with a small network and add more layers/neurons as necessary if it does not converge. However, keep in mind that it might take a while (hours or days) to converge so don't get hasty.

3) I would think that attempting to work with a "continuous" action space (model outputs indexes of "1" elements) would be more difficult to achieve convergence. Is it wise to always try to use discrete action spaces?

I'm not sure I fully understand your question. Mind elaborating? What do you mean by the indexes of "1" elements that the model outputs?

Thanks for the suggestions. To answer your last question, it seems that another potential way to tackle this problem would be to ditch to "moving agent" approach entirely. Instead, I could use a "continuous" action space by having my network output 2 values: a vertical index and horizontal index. For example, if the network outputted 38 and 71, the value at grid[38][71] would be cleared to 0. Of course, I then could not use Q-learning at all, and use actor critic instead.


text:
How can an ANN efficiently predict multiple numbers with fixed sum (in other words, proportions)?
I need a neural network (or any other solution) to predict 3 values which sum equals a fixed number (100). This will help me calculate proportions. Which is the most efficient way to do this?
The learn data only contains extreme situations where each row contains one and only one output value set to 100.
The data to predict is expected to contain more nuances in the output values.
All my attempts lead to very low accuracy as the predicted output sum is almost never a 100. Even when I try to normalize the predicted output, the predictions show very poor accuracy.
Should I try to organize the data with 2 angles instead and deduct the 3rd angle as the remainder in a circle? How to normalize those 2 angles and how to make sure their sum will not exceed the maximum value making the 3rd angle negative?
Illustration of learn data extract (4 input columns and 3 output columns).
0    1    2    3    100  0    0
4    5    6    7    0    100  0
8    9    0    1    0    0    100

Illustration of desired output predictions where each line sums as 100:
7    83   10
39   12   49
68   24   8
28   72   0
86   6    8
32   49   19
0    0    100

I think Normalization is the route— what approach did you try?

I tried several normalization approaches. One of them consisted in dividing each output value by the sum of the 3 and multiply by 100. Does this make sense?

Yes. That should be fine. What’s your loss?

Thanks for the confirmation. I didn't even try to calculate the loss, the numbers were completely off. Maybe my issue is not with the normalization then? Maybe I should rethink the whole learning process?

If you don’t have a loss— how did you train?

I use a Jupyter MLPRegressor with a StandardScaler as explained here. I used various maximum number of iterations (1000-10000) and hidden layer sizes (50,50 or 200,200,200). I used different activation functions and stick to logistic. I recently got much better predictions by removing several less relevant input columns. Sorry, I didn't realize the loss is necessarily calculated. I get now 'MLPRegressor' has no attribute 'loss_curve_' error when I try to plot. I'll look for a solution for that.

The error I had regarding the loss was due to a wrong copy/paste. I get a loss of 19 (starting around 1600) with 10000 iterations, layer sizes (200,200,200) and a relu activation function. Predictions are not perfect but it is a lot better.

@mshlis , my results are not very convincing. I wonder whether a multi output activation function to take into account this proportion constraint in the last neural network layer could help. It seems that the "softmax" activation function could be an answer to this. Though, I'm not sure I can easily use such a solution with jupyter.


text:
Train a recurrent neural network by concatenating time series. Is it safe?
As the title says, I want to train a Jordan network (i.e. a particular kind of recurrent neural network) using a certain number of time series.
Let's say that $x_1, x_2, \ldots x_N$ are $N$ input time series (i.e. $x_i = [x_{i,1}, x_{i,2}, \ldots, x_{i,T}]$, where $T$ is the length of the time series) and $y_1, y_2, \ldots y_N$ (i.e. $x_i = [y_{i,1}, y_{i,2}, \ldots, y_{i,T}]$) are the corresponding target time series.
More specifically, the target time series are just sequences of "$0$s", which may end with sequences of "$1$"s. Here I show you some example:
$$y_i = [0 ~ 0 ~ 0 \ldots 0 ~ 0 ~ 1 ~ 1 ~ 1 \ldots 1 ~1 ], $$
$$y_i = [0 ~ 0 ~ 0 \ldots 0 ~ 0]. $$
This means that I want that my machine "learn to raise" under some situations related to the corresponding inputs $x_i$. Indeed, the objective of my network is to "raise" an alarm if "something" happens.
At the moment, my training strategy is the following. I create a new time series which corresponds the concatenation of all the available $x_i$ and $y_i$. Let's call the concatenated series $X$ and $Y$. Then I use $X$ and $Y$ to train a network.
Here is my problem. If I concatenate, then I also teach to my machine to "drop", since I can have situation like this:
$$Y = [ \ldots 1 ~ 1 ~ 0 ~ 0 \ldots].$$
Is this really a problem? Are there other "training strategies" to be employed so that I avoid this kind of unwanted behaviors?

My first thought when reading this was that the issue would be with the Xs instead. By concatenating the Xs, you are artificially including a transition from x_{i,T} to x_{j, 1}. If this transition does actually happen in your data, you might get some interference so its worth exploring your data for that. Otherwise, I think that this approach is fine.

@JadenTravnik you are right, thanks. Anyway, I'm trying to train a neural network since it is very difficult to assess if there is a real transition in the data $x$. Indeed, data $x$ are multidimensional. If there was a "graphical evidence", then maybe machine learning would have been unnecessary.

Why a downvote? I think it is stupid to downvote without commenting.


text:
Nature-inspired artificial intelligent methods for Blockchain?
I am working on the blockchain technology and I am not very familiar with the AI concept.
The proposal of this web page: (http://www.euraxess.lu/jobs/349354) opens a discussion about use of nature inspired artificial intelligent methods for traceability chain decision in blockchain technology as an alternative to current consensus mechanisms. It continues as follows:

"It has been demonstrated that using traceability chain is a more
  effective method. In traceability chain, since the mechanism has to
  trace related information among participant’s nodes across the entire
  chain, the extraction and recognition of the data features plays a
  crucial role in improving the efficiency of the process."

However, it does not give any example to demonstrate an instance of this approach. So, I searched in google.scholar and any ordinary web pages to find only an instance similar to this approach since it has mentioned: "It has been demonstrated that using traceability chain is a more effective method." (Please read the web page)
Is someone here familiar with this approach? And if yes, Is there any article/example to explain a more about this approach of consensus in blockchain? And what does exactly mean "traceability chain"? And also ingeneral, can we call this approach as a consensus? The text is not really clear to me.
Please not that I have no idea about this proposal and just I'd like to know if it's practicable? or it's buzzwords?
Also, may this approach related to the approach that has been mentioned in this answer (Swarm Intelligence): https://ai.stackexchange.com/a/1315/19910 Or it is a different concept?
Thanks for your help


text:
How Swarm Intelligence can empower Blockchain?
Are there examples of applications in blockchain consensus using swarm intelligence, as opposed to classical consensus mechanisms like PoW or PBFT?
Please note that recent classical consensuses, including lottery-based such as PoW in which the winner of lottery creates the new block, or voting-based such as PBFT or Paxos in which the entities achieve a consensus through a voting process; both approaches have problems of efficiency, latency, scalability, performance etc. And emerging a new alternative approach seems necessary. In this way, can nature-inspired algorithms (such as evolutionary algorithms (EA), particle swarm optimization (PSO), ant colony optimization (ACO) etc) be employed as an alternative to classical consensus algorithms?

I've been working on Proof of Reputation and variation on the "trust" mechanism in regards to the consensus problem. In trust/merit/reputation consensus swarm algorithms seem to be applicable to classification schemes and used similarly to Bayesian techniques, SVM, or linear regression. By running the chain transactions against some threshold there is some possibility that best fit membership can be found on the basis of trust or reputation. The research is unfinished, but promising.


text:
Classical Internet routing vs. Swarm routing (such as Ant routing)?
Is it possible to mention the drawbacks/advantages of Swarm routing (such as Ant routing etc) in comparison with classical routing algorithms in communication networks in a general view?
In other words, what will we gain if we replace a classical routing algorithm with a swarm routing based algorithm?
Can we compare these two type of routing algorithms in a general view to mention their to count the drawbacks/advantages?
The main purpose of this question is to define applications of each of those routing approaches. Which one is more decentralized? And which one has more efficient performance?
Here is my personal opinion (I am not sure about it) :
classical internet routing in more centralized than swarm routing (such as ACO based routing) which does not use any routing table and router to avoid moving towards centralization where those routing tables and routers can be manipulated (as a point of failure). Instead, classical internet routing may be faster than swarm based routing. Briefly, classical may be more centralized, but faster and on the other side, swarm based is more decentralized but may be slower. Am I wrong?
Please note that when I say "decentralized", I mean a network like "ad hoc networks" that do not rely on routers or access points (as a point of failure) to avoid moving towards centralization. In case of using routers or access point, some kind of centralization is inherent. With this definition of decentralization, it seems Swarm based routing such as Ant routing would be more decentralized. However, I am not sure about it and that's my main question.


text:
PyTorch and Opacus for Differential Privacy
When testing an example code from the TensorFlow website using Jupyter Notebook, which is available at the following link: [LINK_1], I encountered an error. You can find my question about that error at the link below: [LINK_2].
As a result, I decided to write equivalent implementations for the same functionality using PyTorch with Opacus and PySyft. However, I unfortunately encountered another error.
Below is the code for implementing the same functionality of the example code from the TensorFlow website, but using PyTorch with Opacus and PySyft, along with the error message.
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
from opacus import PrivacyEngine

# Define a simple model
class SimpleCNN(nn.Module):
    def __init__(self):
        super(SimpleCNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, kernel_size=3)
        self.fc1 = nn.Linear(32*26*26, 10)

    def forward(self, x):
        x = torch.relu(self.conv1(x))
        x = x.view(-1, 32*26*26)
        x = self.fc1(x)
        return torch.log_softmax(x, dim=1)

# Data loaders
transform = transforms.Compose([transforms.ToTensor()])
train_dataset = datasets.MNIST('.', train=True, download=True, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)

# Initialize model, optimizer, and loss function
model = SimpleCNN()
optimizer = optim.SGD(model.parameters(), lr=0.01)
criterion = nn.NLLLoss()

# Initialize PrivacyEngine
privacy_engine = PrivacyEngine(
    model,
    batch_size=64,
    sample_size=len(train_loader.dataset),
    epochs=1,
    max_grad_norm=1.0,
)

privacy_engine.attach(optimizer)

# Training loop
model.train()
for epoch in range(1):
    for data, target in train_loader:
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

# Print privacy statistics
epsilon, best_alpha = optimizer.privacy_engine.get_privacy_spent(1e-5)
print(f"Epsilon: {epsilon}, Delta: 1e-5")

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[1], line 32
     29 criterion = nn.NLLLoss()
     31 # Initialize PrivacyEngine
---> 32 privacy_engine = PrivacyEngine(
     33     model,
     34     batch_size=64,
     35     sample_size=len(train_loader.dataset),
     36     epochs=1,
     37     max_grad_norm=1.0,
     38 )
     40 privacy_engine.attach(optimizer)
     42 # Training loop

TypeError: PrivacyEngine.__init__() got an unexpected keyword argument 'batch_size'


text:
Reinforcement Learning to Grouped Scheduling Optimisation Problem
I am not sure the name of this kind of problem, but anyway, the situation is as below.
Assign teachers into Groups and consider on each of their workload, availability etc.
There are some other soft/hard constraint (equality/inequality) like

Each group should have at least 2 teachers
Everyone in the group have similar workload
Total workload in the group is below a certain value
All are in different expertise

and more...
I am trying to build a sub-optimal solution to solve this problem. Linear/non-linear programming seems not working for grouping problems. I am thinking of genetic algorithm or reinforcement learning.
Can this problem solve by using RL or DRL?
I am trying to define the groups as state, and actions include "assignToGroup" and "removeFromGroup".
And any kind of idea or suggestion of how to solve this problem?
Many thanks

It is often possible, by describing certain parts of a problem as state, others as actions, and some measurement you want to optimise as a reward, to frame a problem as a reinforcement learning task. However, this is not always helpful.
I suspect your problem is in the "RL is not that helpful" category. You could give it a try, and maybe expect to find working solutions but take an unreasonable amount of time to find optimal solutions.
What you appear to have here is a problem in combinatorial optimisation. There are many applicable algorithms, a good selection are described in the free book Clever Algorithms: Nature-Inspired Programming Recipes - which is best will be determined by the specific details of your constraints and the size of the problem.
If many constraints are soft and can be assigned a cost, you may be able to use genetic algorithms (with genome being group assignments in order) or simulated annealing for example.


text:
Regarding L0 sparsification of DNNs proposed by Louizos, Kingma and Welling
I am reading the paper on $\ell_0$ regularization of DNNs by Louizos, Welling and Kingma (2017) (Link to arxiv).
In Section 2.1 the authors define the cost function as follows: $$ \mathcal{R}\left( \tilde{\theta}, \pi \right) = \mathbb{E}_{q(z|\pi)}\left[ \frac{1}{N} \left(\sum_{i=1}^N \mathcal{L}\left(h\left( x_i, \tilde{\theta}\circ Z\right), y_i \right) \right)\right] + \lambda\sum_{i=1}^{|\tilde{\theta}|}\pi_i. $$
In the above display, $\tilde{\theta}$ are the weights, $Z$ is a random vector of the same dimension as $\tilde{\theta}$ consisting of independent Bernoulli components $q(Z_i|\pi) \sim Bernoulli (\pi_i)$, and $\circ$ is the element-wise product.
The authors then state the following:

the first term is problematic for $\pi$ due to the discrete nature of $Z$, which does not allow for efficient gradient based optimization.

I am not sure I understand this. Denoting the first term by $\mathcal{R}_1 = \sum_{i=1}^N \frac{1}{N}R_i$ ($R_i$ defined below), and using the notation $\pi_z = \prod \pi_i^{z_i} (1-\pi_i)^{1-z_i}$ and $\mathcal{Z}$ for the set of all possible values of $Z$, we should have
$$
R_i := \mathbb{E}_{q(z|\pi)}\left[ \mathcal{L}\left( h\left(x_i, \tilde{\theta}\circ z\right), y_i \right) \right] = \sum_{z \in \mathcal{Z}}\pi_z\mathcal{L}\left( h\left(x_i, \tilde{\theta}\circ z\right), y_i \right)
$$
So, it seems to me that the gradient of $R_i$ with respect to $\pi_j$ can be obtained as
$$
\frac{d R_i}{d\pi_j} = \sum_{z \in \mathcal{Z}}\mathcal{L}\left( h\left(x_i, \tilde{\theta}\circ z\right), y_i \right) \frac{d\pi_z}{d\pi_j}
$$
and
$\frac{d\pi_z}{d\pi_j} = \frac{\pi_z}{\pi_j}$ if $z_j=1$ and $-\frac{\pi_z}{1-\pi_j}$ if $z_j=0$. So, it appears that we can obtain the derivative of the first term with respect to $\pi_j$ as well.
My question is the following:

If my above calculation is correct, then the derivatives $\frac{d\mathcal{R}_1}{d\pi_j}$ can be computed, and we can perform SGD on the cost function $\mathcal{R}(\tilde{\theta}, \pi)$. But the authors claim that it cannot be obtained and hence they introduce the `hard concrete' distribution etc. to construct a differentiable cost function.


text:
How to Mask an image using Numpy/OpenCV?
I am detecting wheels with a deep learning algorithm. The algorithm gives me the coordinates of those rectangles. I want to keep data that is in the rectangles of the image. I created rectangles as a mask of the area I want to keep.
Here is the output of my system
I read my image
im = cv2.imread(filename)

I created the rectangles with:
height,width,depth = im.shape
cv2.rectangle(img,(384,0),(510,128),(0,255,0),3)
cv2.rectangle(rectangle,(width/2,height/2),200,1,thickness=-1)

How can I mask out the data outside of the rectangle from the original image? and keep those rectangles?
Edited: I wrote this code and it only gives me one wheel. How can I have multiple masks and get all the wheels?
  mask = np.zeros(shape=frame.shape, dtype="uint8")

# Draw a bounding box.
# Draw a white, filled rectangle on the mask image
cv.rectangle(img=mask,
             pt1=(left, top), pt2=(right, bottom),
             color=(255, 255, 255),
             thickness=-1)

# Apply the mask and display the result
maskedImg = cv.bitwise_and(src1=frame, src2=mask)

cv.namedWindow(winname="masked image", flags=cv.WINDOW_NORMAL)
cv.imshow(winname="masked image", mat=maskedImg)

You can use cv2.bitwise_and and pass rectangle as a mask.
im = cv2.imread(filename)
height,width,depth = im.shape
cv2.rectangle(img,(384,0),(510,128),(0,255,0),3)
cv2.rectangle(rectangle,(width/2,height/2),200,1,thickness=-1)

masked_data = cv2.bitwise_and(im, im, mask=rectangle)

cv2.imshow("masked_data", masked_data)
cv2.waitKey(0)


text:
Scrabble game using machine learning
I've been thinking if machine learning can be used to play the game Scrabble. My knowledge is limited in the ML field, thus I've seeking some pointers :)
I want to know how could I possibly build a model that picks a move from all the given valid moves of the current game state, and then plays the move and wait for the delayed reward. The actions here aren't static actions, they are basically selecting move to maximize the final score.
Is there any way to encode the valid moves and then use a model to pick those moves?
I've also considered the genetic approach, but I think if I can represent my move with a set of features (score, consonantVowels ration, rack leave score, #blank tiles after the move, ...etc), training a neural network like this could take a long time.
Another training related question, is it feasible to run the training on a GPU given that I will be waiting for a response (the new game state) from the opponent (e.g. Quackle) after every action?
Thank you :)

Why do you want to use machine learning for this problem ? Isn't a simple dictionary more efficient ?
I understand you want to use machine learning for delayed reward, but when I think about Scrabble, I don't really see it as a problem : since you will have random letter the next step, there is no point to "keep your letters". When I play Scrabble, I just try to maximize the score for the current turn, and there is very little point of trying to maximize score for next turns.

I think there're some features that could describe each move, more than just its board score. Some of those features are the rack leave score, the number of blank tiles it uses compared to the score it's going to give me, the number of hot spots it opens, ... etc. Therefore I need to somehow tune the parameters for these features.

According to comments below, this reply seems to be incorrect. See the comments for more links and discussion.
Link to Maven: https://en.wikipedia.org/wiki/Maven_(Scrabble)

The term "solved" has a very specific meaning in the context of game AI, which definitely doesn't apply here. Even though it's already quite old, as far as I'm aware the PhD thesis about Maven is probably indeed still one of the more important source on AI for Scrabble. The full thesis can still be downloaded from this link: https://project.dke.maastrichtuniversity.nl/games/files/phd/Shepard_thesis.pdf

From what I've seen, your resources are pointing to Maven which doesn't involve reinforcement learning!

Scrabble is not close to solved. No AI is favored against the best humans. See here for expert analysis.


text:
Reduce receptive field size of CNN while keeping its capacity?
I have a convolutional encoder (a CNN) consisting of DenseBlocks and a total of 50 layers (cf. FC-DenseNet103). The receptive field of the encoder (after last layer) is 660 according to Tensorflow function compute_receptive_field_from_graph_def(..)) whereas the input image is 64x64 pixels. Obviously the receptive field is way too big.
How can the receptive field be reduced to say 46 but the capacity of the encoder be more or less kept at the same level? By capacity I simply mean the number of parameters of the model. The capacity requirement is justified due to the complex dataset to be processed.
Using less layers or smaller kernels reduces the receptive field size but also the capacity. Should I then just increase the number of filters in the remaining layers in order to keep the capacity?

One way to keep the capacity while reducing the receptive field size is to add 1x1 conv layers instead of 3x3 (I did so within the DenseBlocks, there the first layer is a 3x3 conv and now followed by 4 times a 1x1 conv layer instead of the original 3x3 convs (which increase the receptive field)). In doing that, the number of parameters can be kept at a similar level. While 1x1 convolutions are good to add non-linearity, they are not useful to learn spatial information where neighboring pixels/values correlate. This might turn out to be a problem.
Therefore, experiments will have to show if this idea of adding 1x1 conv layers to keep the capacity proves useful or not in learning spatial features on a natural image data set.


text:
How to preprocess a modified dataset so that a fitted CNN makes correct predictions on an un-modified version of the dataset?
for a school project I have been given a dataset containing images of plants and weeds. The goal is to detect when there is a weed in the pictures. The training and validation sets have already been created by our teachers, however they probably didn't have enough images for both so they "photoshopped" some weeds in some of the training pictures.
Here are examples of images with the weed label in the training set:

In some cases, the "photoshopped" weed is hard to detect, and no shape resembling a weed is clearly visible like in this picture (weed at the very bottom, near the middle):

And here is an example of an image with the weed label in the validation set:

How would I go about preprocessing the training set so that a CNN trained on it would perform well on the validation set? I was thinking of applying a low-pass filter to the rough edges of the photoshopped images so that the network doesn't act as an edge detector, but it doesn't seem very robust. Should I manually select the best images from the training set? Thank you!