text:
Why doesnt my Neural Network work?
I Build this NN in c++. I reviewed it since 3 days. I checked every line 100 times, but I cant find my error.
If someone can please help me find the Bugs:
1. The output is garbage
2. The weights go from 2e^79 down to -1.8e^80 after approximatly 400 iterations.
mat flip(mat m) {
    mat out(m.n_cols,m.n_rows);

    for (int i = 0; i < m.n_rows; ++i)
        for (int j = 0; j < m.n_cols; ++j)
            out(j, i) = m(i, j);

    return out;
}

Layer::Layer(int nodes) :
    rand_engine(time(0))
{

    y = mat (nodes, 1);
    net = mat(nodes, 1);
    e = mat(nodes, 1);
}

Layer::Layer(int nodes, int next_nodes) :
    Layer(nodes)
{

    this->next_l = next_l;

    auto random = bind(uniform_real_distribution<double>{-1, 1}, rand_engine);

    w = mat(next_nodes,nodes);

    for (int i = 0; i < w.n_rows; ++i) {

        for (int j = 0; j < w.n_cols; ++j) {
            w(i,j) = random();
        }
    }
}

Layer::Layer(int nodes, Layer* next_l) :
    Layer(nodes,next_l->y.n_rows)
{
    this->next_l = next_l;
}

void Layer::feed_forward()
{
    next_l->net = w*y;

    for (int i = 0; i < next_l->y.n_rows;++i)
        next_l->y[i] = sig(next_l->net[i]);

}

void Layer::backprop()
{

    for (double d : w)
        cout << d << "\t";

    e = flip(w)*next_l->e;

    for (int i = 0; i < e.n_rows; ++i) {
        e[i] *= net[i] * (1 - net[i]);
        cout << e[i] << '\t';
    }

    w += l_rate*(next_l->e*flip(y));

}

void Layer::backprop_last(mat t)
{
    for (int i = 0; i < e.n_rows; ++i) {
        e[i] = net[i] * (1 - net[i])*(t[i] - y[i]);
        cout << e[i] << '\t';
    }

}

void Layer::feed_forward(Layer* next_l)
{
    this->next_l = next_l;

    feed_forward();

}

double Layer::sig(double x)
{
    return 1 / (1 + exp(-x));
}

Network::Network(vector<int> top):
    top(top)
{

    network = new Layer*[top.size()];
    network[top.size() - 1] = new Layer(top.back());

    for (int i = top.size()-2; i > -1; --i)
        network[i] = new Layer(top[i], network[i + 1]);

}

Network::~Network()
{
    delete[] network;
}

void Network::forward()
{
    for(int i = 0; i < top.front();++i)
        network[0]->y[i] = input[i];

    for (int i = 0; i < top.size() - 1; ++i)
        network[i]->feed_forward();
}

void Network::forward(vector<double> input)
{
    set_input(input);
    forward();
}

void Network::backprop()
{

    network[top.size() - 1]->backprop_last(t_vals);

    for (int i = top.size() - 2; i > -1; --i) {
        network[i]->backprop();
    }

}

void Network::backprop(vector<double> t_vals)
{
    set_t_vals(t_vals);
    backprop();
}

I know its a bunch of code but im really desprate since I cant find whats wrong. I tested it with a simple XOR.
Edit:
Heres my Main code:
    #include "Network.h"
#include <iomanip>

using namespace std;

vector<vector<double>> input = { {0,0},{0,1},{1,1},{1,0} };

vector<vector<double>> true_vals = { {0},{1},{0},{1} };

int main() {

    ifstream f("out.txt", fstream::out);
    f.clear();

    cout << fixed;
    cout << setprecision(5);

    Network net({2,5,1});

    vector<double> in,t,out;

    auto buf = cout.rdbuf();

    for (int i = 0; i < 1000; ++i) {
        cout.rdbuf(f.rdbuf());

        in = input[i % 4];

        net.forward(in);

        out = net.get_output();

        t = true_vals[i % 4];

        net.backprop(t);
        cout << '\n';
        cout.rdbuf(buf);
        if ((i %101))continue;

        cout << "it: " << i << '\n';

        cout << "in:\t";
        for (double d : in)
            cout << d << ' ';

        cout << '\n';

        cout << "out:\t";

        for (double d : out)
            cout << d << ' ';

        cout << '\n';

        cout << "true:\t";

        for (double d : t)
            cout << d << ' ';

        cout << '\n';

        double err = net.get_error();

        cout <<"err:\t"<< err << '\n' << '\n';

    }

    cout.rdbuf(NULL);
    f.close();
    return system("pause");
}

What is this code for? Problem statement?

@kiner_shah that's the code I made. And I want to know if anyone knows why it doesn't work.

What is that code for? XOR function or AND function or something else? And this is not your entire code! Post your whole code with main() function

@kiner_shah it's a simple feed forward and backprop network. Input output and true values are doubles

@kiner_shah I added the main code

Ok now I get it, it's an XOR Problem!

This question is off-topic here, sorry.

This type of question is more suitable for Cross Validated or Data Science.

A little search on Google answers your question.
XOR input space is not linearly separable. It means that you cannot separate the input points in a 2D space into 1 area and 0 area by simply drawing a line between them. It requires at least 2 lines to separate the XOR input space and consequently 2 output nodes (used as classifiers rather than regression). You can easily find its details in google. Search "XOR problem in Neural Net".
You can manually implement the desired Neural Network with two output Nodes acting as classifiers as follows :

Where A & B are two output Nodes which act as classifiers (by forming AA' and BB' decision lines respectively during training by Backpropagation). The interpretation of the Outputs of the nodes is given in the table where Net column represents the Overall output to be interpreted.
I showed the above manual implementation just to give you the idea of how classification is done behind the scenes.
Here is the actual Automatic implementation :

Here , all the task is performed by the Neural Net behind the scenes and you get the desired output from the output node in the topmost layer

Since you said the 4 required outputs are used as classifiers, why do I need 4 classifiers for a binary output and how to I interpret it.

Now I'm confused. On the link the network only  has 1 output node. You stated it needs 2. Please explain what I'm getting wrong.

Ok... Ignore it. I am editing my answer to make you understand.

Ok thank you very much. I think I understand it now.


text:
Is it possible to manage the evolution of super-intelligent AI?
Post singularity AI will surpass human intelligence. The evolution of AI can take any direction, some of which may not be preferable for humans. Is it possible to manage the evolution of super-intelligent AI? If yes, how? One way I can think of is following. Instead of having a mobile AI like humanoid, we can keep it immobile, like a box, like current super computers. It can be used to solve problems of maths, theoretical science etc.

Assuming super-intelligence is possible, the answer is probably yes and no.
Yes in Kurzweil-like scenarios, where super-intelligence is an extension of human beings by technology (we are already in to some extent). Then control follows, as super-intelligence depends on us. It would extend our capabilities, such as speed of processing, extent of processing, etc. Even then control is debatable, as a remote-controlled killing machine would be part of a super-intelligent organism, partially human "controlled", partially autonomous.
No in "Future of Life Institute"-like scenarios, where super-intelligence is independent from humans. The thinking is simple: What can we hope to do facing someone way more intelligent? The usual parallel is to compare this scenario with the arrival of the "developed" conquistadors in early America. Gunpowder vs. mere raw strength and arrows.

You mean Kurzweil? what is "FIL-like scenarios"?

let me counter argue in the case of non human like super-intelligence. of course, they will be made by humans. for such intelligence to do any harm, they must need resources i.e. power,materials etc. can't we control them by controlling these resources? can we think of any situation where they won't need resources to harm? but then again,we may not be able think of such situation because we are low intelligence! :)

@AmitMaurya That's the whole story. We may reach a point where we cannot wrap our head around what is happening, either because of complexity, or mere too-high pace of change. It seems you have follow-up questions, and those would be terrific new entries on the site.

Competition always gives better result. If machines will try to improve themselves, we as human beings will definitely try to improve ourself.

Yes, it is possible.
When humans were working on the first nuclear bomb, some field experts of the time thought that when the reaction went super-critical, it would not stop, and would devour the earth. It was a plausible possibility given our understand of nuclear energy at the time, and we didn't know for sure until we did it.
Some scientists synthesize black-hole like environments in laboratories. Some experts think that if a certain point is accidentally crossed due to ignorance or negligence, we may devour our planet with a self made black hole.
The situation is the same with AI. Until we actually create a super-intelligent AI, we cannot say with certainty whether it will be controlled or controllable until it happens. Until that time comes the answer to your question is yes, it's possible, but that does not mean it will or will not happen that way.

Without going into more detail at the moment (b/c I'm time strapped), I strongly urge you to research the Control Problem.
My own personal view is that humans are more problematic than machines.  Machines are at least rational.
To be more specific, I believe human "management" (read as "mis-management") of powerful AI is potentially more of a problem than super-intelligent AI left to it's own devices.
Humans are known to abuse power, and history is filled with such examples.  Machines, at least, have  a clean slate in this regard.

thanks. problem is machines are created by humans.problematic humans may create problematic machines. even if machines are good initially,the way they will evolve can be problematic.

@AmitMaurya Could be problematic, but I see that as a factor of humans imposing poor ideas based on shoddy understanding.  Details here: http://ai.stackexchange.com/a/2878/1671

u r very correct when u say "Humans are known to abuse power, and history is filled with such examples".  power corrupts. so what will happen with powerful AIs? "leaving it on its  own" may not be a  good idea.

@AmitMaurya Yes, but humans have emotions that cloud logic and lead to poor decisions. Machines are not so burdened. The only apocalyptic scenarios regarding AI that are completely sound, imho, are the ones regarding lack intelligence (grey goo) or insufficient intelligence ("Hal 2000", "I Robot", etc.) Hal was smart but he couldn't contextualize or reconcile contradictions. .

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
Is a binary attribute type the same as binomial attribute type?
I am not sure if I can use the words binomial and binary and boolean as synonyms to describe a data attribute of a data set which has two values (yes or no). Are there any differences in the meaning on a deeper level?
Moreover, if I have an attribute with three possible values (yes, no, unknown), this would be an attribute of type polynominal. What further names are also available for this type of attribute? Are they termed as "symbolic"?
I am interested in the realtion between the following attribute type: binary, boolean, binominal, polynominal (and alternative describtions) and nominal.

@SmallChess's answer is a good start, but there are some additional parts to the question.
binary variables or binary data consist of data with the values 0 or 1, and no other values. We usually don't talk about "binary distributions", because it's only data, variables, or outcomes that can be binary. A distribution might produce binary data, but is not itself binary because its parameters typically take on real-values.
A binomial distribution is a distribution that produces binary data. In particular, it is a random process that produces the value 1 with probability $$p$$, and the value 0 with probability $$1-p$$. Notice that although it makes binary data, it is not itself a kind of data, and is in fact charactorized by a non-binary number (p).
Boolean data takes on the values true or false. Often, but not always, these are stored as 0's and 1's. The distinction is that boolean data may not be stored numerically. There might also be different expectations about how Boolean data should be processed (for instance, $$true + true = true$$, but $1 + 1 = 2$.
I am not aware of the term polynomial being applied to data. However, multinomial distributions are probability distributions that produce 0 with probability $p_0$, 1 with probability $p_1$, 2 with probability $p_2$, and so on, producing $p_n$ with probability $1 - \sum_{i=0}^{n-1} p_i$ for $n$ different numbers. Like binomial distributions, multinomial distributions are characterized by a set of real-valued numbers, and are distinct from the kind of data they generate.
Categorical data takes on values from a set of categories. The example you give (yes, no, maybe) is not strictly multinomial data, but could be generated from a multinomial distribution by mapping the values 0, 1 and 2 onto yes, no and maybe. Note again that categorical data might be non-numeric. Operations like adding might be non-sensical.
Cardinal data isn't something you asked about, but arises when data can be nicely ordered. For example, playing cards are easily mapped to the numbers 1-13, and can have reasonable semantic meaning when represented this way (e.g. A + 2 = 3, and 1 + 2 = 3).
Nominal Data is just literal numbers that mean exactly what they purport to mean. For example, if you store the number of cans of beer a customer purchased, that would be nominal data.

Thanks for this great overview! The tool rapidminer uses the terms binominal and polynominal as attribute types. Do you think these are the "right" terms for attribute type? For instance, the attribute type "Integer" is used to describe natural numbers as well as their additive inverses (in the tool rapidminer). From this point of view, I think it would be also to correct to name the type of an attribute with two values binary or boolean. Do you agree?

@user3352632 I'm not familiar with rapidminer, but "polynomial" is definitely a non-standard term. If someone told me they had "polynomial data", my guess would be that they had data generated from a function like y=x^3 + 2*x^2 + x, not that they had data with 3 or 4 values. If you do a google search for the term, that's what you find also. As mentioned in the answer, the terms "binary" and "boolean" have precise meanings too, and are not completely interchangeable. If you have two values that are non-numeric, and that aren't true/false, you actually have categorical data!

Binomial is a distribution characterised by $p$, the probability of success for an independent trial. Each sample you get from the distribution is a binary variable, 0 or 1.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Which loss / activation function with 2 classes that do not occur often and do not sum to one?
I have a neural network that predicts 2 classes of a time series (bottom and top). Currenlty my Y labels are size 2: [1 0] for bottom and [0 1] for top. The NN has 2 output nodes.
Of course not every instance is a bottom or top. (Overwhelmingly) most of the time we have no bottom and no top.
Should I change my network (and Y labels) to have 3 outputs?
And what would be appropriate as a loss function. I guess with 3 outputs I could use entropy or focal loss. With 2 outputs I don't think entropy is correct.
Currently I use PyTorch (lightening), and I have:
self.lstm = nn.LSTM(input_size=n_features,
                            hidden_size=hidden_size,
                            num_layers=num_layers,
                            dropout=dropout,
                            batch_first=True)
self.linear = nn.Linear(hidden_size, 2)

def forward(self, x):
        lstm_out, _ = self.lstm(x)
        y_pred = self.linear(lstm_out[:,-1])
        return y_pred

criterion = nn.CrossEntropyLoss()

This is multi-label classification, which means you have two binary classification problems, one for each of your classes. This is different than multi-class classification.
For this use binary cross-entropy loss, with binary encoded labels (0/1), not one-hot encoded labels, and put a sigmoid activation at the output with two neurons (one for each class).

Thank you. To make this work optimally, is there any issue that both classes occur very scarcely in the training set? I.e. very unbalanced? Would something like FocalLoss be a solution? I can create a third 'normal' class as class 1 and 2 are mutually exclusive. Trying to find what would perform best.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Which features and algorithm could optimize this air-conditioner problem?
Imagine we have 2 air conditioner systems (AA) and 2 "free cooling" systems which mix external and internal air (FC) in a closed box which always tends to warm up. For each system, we have to find turn on and off temperatures (for some hysteresis, let's say between the range 20-40 each one) to optimize the energy consumption.
As we don't know the relation between these parameters and the energy consumption (and we don't intend to know them), we treat the problem as a black-box function.
Till now, the problem would be solvable via a bayesian optimizer (eg. with gaussian process acquisition function).
But there is a problem: the best configuration may change between seasons, and even days! A simple bayesian optimizer maybe could deal with these changes limiting the data it takes into account by, for example, the last 15-30 days. But this would deal with the change AFTER the consumption increased.
So, the idea is introduce some contextual variables which would help the system prevent these changes (eg. the external and internal temperature, and the vectors of variation of external and/or internal temperature, the weather prediction, whatever).
Also, some of these variables we can take into account might be internal of the system, which means while these influence the best configuration, the actual configuration also influences these variables! and this becomes a reinforcement learning problem.
1) Is there a way (documented or experimental) to know which variables (both internal or external) influences the optimal configuration of these AA/FC systems?
2) Based on the first question, which would be the best approach?
2.1.) No features. This might be considered a multiarmed bandit problem for continuous reward. (FIX POSTERIOR TO SCENARIO CHANGE, IF THERE IS A SCENARIO CHAGNE)
2.2.) Only external features to predict the scenario change. This might be considered a contextual multiarmed bandit-problem. (FORESEE THE SCENARIO CHANGE)
2.3.) Consider only system-internal features. This can be considered a reinforcement-learning problem. (FIX IMMEDIATLY THE SCENARIO CHANGE)
2.4.) Consider both external and internal features. This can be considered a reinforcement-learning problem where some of the states are not influenced by the configuration. (FORESEE THE SCENARIO CHANGE, AND IF SOMETHING FAILS, FIX IMMEDIATLY).

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Which algorithm to use to solve this optimization problem?

I have items called 'Resources' from 1 to 7.
I have to use them in different actions identified from 1 to 10.
I can do a maximum of 4 actions each time. This is called 'Operation'.
The use of a resource has a cost of 1 per each 'Operation' even if it is used 4 times.
The following table indicates the resources needed to do the related actions:

|        |            Resources             |
|--------|----------------------------------|
| Action |  1 |  2 |  3 |  4 |  5 |  6 |  7 |
|--------|----------------------------------|
|     1  |  1 |  0 |  1 |  1 |  0 |  0 |  0 |
|     2  |  1 |  1 |  0 |  0 |  1 |  0 |  0 |
|     3  |  1 |  0 |  1 |  0 |  0 |  1 |  0 |
|     4  |  0 |  1 |  0 |  0 |  0 |  0 |  0 |
|     5  |  1 |  0 |  1 |  1 |  0 |  1 |  0 |
|     6  |  1 |  1 |  1 |  0 |  0 |  0 |  0 |
|     7  |  0 |  1 |  0 |  0 |  0 |  0 |  0 |
|     8  |  1 |  0 |  1 |  0 |  1 |  0 |  0 |
|     9  |  0 |  1 |  0 |  1 |  0 |  0 |  0 |
|    10  |  1 |  1 |  1 |  0 |  0 |  0 |  1 |

The objective is to group all the 'Actions' in 'Operations' that minimize the total cost. For example, a group composed by actions {3, 7, 9} needs the resources {1, 2, 3, 4, 6} and therefore has a cost of 5, but a group composed by actions {4, 7, 9} needs the resources {2, 4} and therefore has a cost of 2.
It is needed to get done all the actions the most economically.
Which algorithm can solve this problem?

Is the objective to perform each action at least once? ( in one of the groups)

Exactly one time all the actions

If the problem is the size of the table you showed or not much bigger the best way is probably brute forcing it. What is the real size of the problem?

30 actions with 10 resources grouped in a max of 6. Only the group of 6 are 4.000.000.000 combinations, but there are also other size combinations... I couldn't calculate it but probable huge

It is not that big, 30 combinations 6 is 600k, then you do the same for 30 combination 5 etc it's around 1M total. Then you just do planning, I think it is pretty feasable, I can try to write something to help you out. You don't need to count in the resources in the combinations, you just verify it during planning and count up the total cost.

I have finally abandoned the idea of doing it with an exact method and I have passed to the heuristic. I have mixed multi-boot, local search and certain random movements. Apparently this is called Greedy Randomized Adaptative Search Procedures (GRASP)
Hypothesis: the best solution is reached filling the 'operations' with the maximum of 'Actions' per 'Operation'. Other combinations are more expensive

create a random solution
[[1 2 3 4] [5 6 7 8] [9 10]]
calculate the cost of each 'Operation'
[[6] [6] [5]] == 17
Study the posibility of minimize the lower 'Operation' permuting some 'Action' with the other 'Operations'.
Actions 4 & 7 permute with 9 & 10
[[1 2 3 9] [5 6 10 8] [4 7]]
Calculate the new cost
[[6] [7] [1]] == 14
Repeat steps 2 to 4 until it is not possible to minimize the lower, then the second lower, then the third etc
You will find soon a Local Minimum. If no more moves are allowed but the Local Minimum is not the Global Minimum, permute two random 'actions' between two random 'operations' and repeat the steps 2º to 5º.

With this method it is possible to find very fast a solution adjusted to the minimum global cost with a complexity of O(n·log(n)), where n is the amount of 'actions'
I've tested it with a random sample of 60 'actions' and 26 'resources' grouped in 'operations' of 6 'actions'. It toke arround 5 minutes to get a really good solution arround 40% better that the initial one (*), and 30 minutes after it only got better solution in about 1%
(*) the initial solution was not actually a random solution. Instead of that I used 'Ant Colony' algorithm with 15 ants to get a better initial solution and reduce the amount of iterations, but that is another history

A possible approach would be starting by grouping the actions into all possible groups.
For 30 actions grouped in a maximum of 6 this would mean:
$$
C^{30}_6 + C^{30}_5 + C^{30}_4 + C^{30}_3 + C^{30}_2 + C^{30}_1 = 768 211
$$
Operations.
Then, I would define the constraints to simplify the problem, as in a Constraint Satisfaction Problem (CSP). The main constraint that you have is that one action can only be represented in one Operation. So, if you choose 1 of the 768ks operations to start with, the second option will be restricted by the condition.
Then I would do some sort of planning and sum the costs until the end doing a depth-first search. Whenever I find a total cost better the previous best, you would update the best value. (Remember that the search ends when between all the operations used, all the actions were explored once.)
While doing the search the costs are added up on the go. You can precompute a vector with the same size as the number of operations where each value corresponds to the total cost of the operation, and then sum it as you are going down the path.
You should also add some heuristic to prevent checking the same operations in different orders. For example doing operating [2,3,4] followed by [7,10], where each number represents an action, is the same as doing: [7,10] followed by [2,3,4].


text:
How to force LLM (like OpenAI ChatGPT) to output a variable list of values?
What prompt (or other technique) should I use with an LLM so that

The result is guaranteed to be reliably parseable as a list of values (e.g. a Python list of strings)
LLM would understand that a variable list of values is expected during generation

EDIT: Chosen solution
What I personally ended up using is BAML. Works with all LLMs and guarantees correct output.
https://github.com/BoundaryML/baml
pip install baml-py
Example:
https://www.promptfiddle.com/BAML-Examples-7xLvl

Related https://ai.stackexchange.com/questions/42127/how-to-prompt-open-ai-for-a-variable-list-length?rq=1

This question is off-topic here. This is about specific tool or tools and how to use them in practice. It has little to do with any scientific theory because there aren't real scientific theories about prompt engineering. Just rules of thumb, which often don't work. This is not science but opinions. Still, this question may be useful, but it's off-topic here. It would be on-topic at https://genai.stackexchange.com/ I think.

I agree with @Neil Slater -- it's never guaranteed because inherently the LLM are a stochastic model, not deterministic.
Given the need for JSON output, OpenAI heeded to the call of the community and made it easy to increase your likelihood of generating a JSON output even with the cheaper gpt-3.5-turbo model.
Here's a code snippet from their documentation:
from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
  model="gpt-3.5-turbo-0125",
  response_format={ "type": "json_object" },
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to output JSON."},
    {"role": "user", "content": "Who won the world series in 2020?"}
  ]
)
print(response.choices[0].message.content)

A tip I would offer is to create a simple test in which you pass a set of typical inputs and have that occur over 20 iterations. Then you examine the accuracy of the output.

There are no guarantees that an LLM used directly will output with any specific constraints.
You can add to prompts, fine-tuning or other customisation to request a specific output, and that often helps, but does not guarantee output will conform to what you requested. At best you can get a certain level of reliablility e.g. 90% accurate or 99% accurate. How accurate you can get depends on the complexity of the task you have stated in the prompt.
I have found that ChatGPT v4 (and I assume others at the same level) will output code snippets quite reliably, with either Python or JSON structure, and using markdown formatting, when this is specified in setting up an Assistant, a GPT or in a system prompt.
A simplified example of a system prompt that might do this:

You are a sentiment analysis tool for user reviews on a shopping site. For each user prompt, decide whether it is "negative", "neutral" or "positive". Then output a JSON snippet with key of "sentiment" and the decision as the value.

Example user input:

This television had a scratch on the screen when it was delivered.

Example LLM output:

A scratched screen is a sign of fault that may have occurred in manufacture or during delivery. Although the customer did not express any emotions in their post, they are likely to be unhappy with this defect.
{
 "sentiment": "negative"
}

In addition, it usually helps with accuracy of processing if you allow the chat session to add preamble to this, as per my example. I.e. do not demand, in the prompt, to only receive a final processed result. That is because the LLM does not "think" about the text output, it simply outputs text. The coherency of the text can be used to emulate some forms of processing.
This is a common misunderstanding when using services like ChatGPT, that many people seem to assume that posting a question causes the AI to "think" then return an answer. It doesn't do that. So if you want any kind of logical analysis of your input, it can help to have the LLM work through the issue - a bit like thinking out loud - before it outputs structured data.
To get your structured data:

In the web UI, code snippets with markdown will have a copy icon, and you can click it to get a copy of the code block to paste into something else.
Via the API, you will need to write code that can look for the markdown block, and extract it and parse. Regular expressions are good for this, and available in most programming languages.

In both cases, you need to allow for the times when the LLM doesn't output what you want. You can retry, fail with an error, or whatever is appropriate for your use case.

Note that this answer was posted before ChatGPT o1 models that do "think" before answering. Under the hood in those models, something very similar to the chain-of-thought suggested in this answer is going on. I.e. the bot is generating some reasoning/explaining text, but hiding it

I'd say it's possible, although it depends on the model you're using.
If you're using llama.cpp (or its python bindings), you can use their grammars support to constrain the output

GBNF (GGML BNF) is a format for defining formal grammars to constrain model outputs in llama.cpp. For example, you can use it to force the model to generate valid JSON, or speak only in emojis.

For a more generic approach, and if you're using Python, take a look at guidance.

guidance is a programming paradigm that offers superior control and efficiency compared to conventional prompting and chaining. It allows users to constrain generation (e.g. with regex and CFGs) as well as to interleave control (conditional, loops) and generation seamlessly.

Here's a snippet of what a guidance configuration would look like (borrowed from their docs):
@guidance
def character_maker(lm, id, description, valid_weapons):
    lm += f"""\
    The following is a character profile for an RPG game in JSON format.
    ```json
    {{
        "id": "{id}",
        "description": "{description}",
        "name": "{gen('name', stop='"')}",
        "age": {gen('age', regex='[0-9]+', stop=',')},
        "armor": "{select(options=['leather', 'chainmail', 'plate'], name='armor')}",
        "weapon": "{select(options=valid_weapons, name='weapon')}",
        "class": "{gen('class', stop='"')}",
        "mantra": "{gen('mantra', stop='"')}",
        "strength": {gen('strength', regex='[0-9]+', stop=',')},
        "items": ["{gen('item', list_append=True, stop='"')}", "{gen('item', list_append=True, stop='"')}", "{gen('item', list_append=True, stop='"')}"]
    }}```"""
    return lm

lm = llama2 + character_maker(1, 'A nimble fighter', ['axe', 'sword', 'bow'])


text:
Convert LSTM univariate Autoencoder to multivariate Autoencoder
I have the following code snippet which takes in a single column of value i.e. 1 feature. How do I modify the LSTM model such that it accepts 3 features?
import numpy as np
from keras.models import Sequential
from keras.layers import LSTM, Input, Dropout
from keras.layers import Dense
from keras.layers import RepeatVector
from keras.layers import TimeDistributed
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from keras.models import Model
import seaborn as sns

dataframe = pd.read_csv('GE.csv')
dataframe.head()

df = dataframe[['Date', 'EnergyInWatts']]
df['Date'] = pd.to_datetime(df['Date'])
sns.lineplot(x=df['Date'], y=df['EnergyInWatts'])

#train, test = df.loc[df['Date'] <= '2003-12-31'], df.loc[df['Date'] > '2003-12-31']
train = df.loc[df['Date'] <= '2003-12-31']
test = df.loc[df['Date'] > '2003-12-31']

scaler = StandardScaler()

scaler = scaler.fit(train[['EnergyInWatts']])

train['EnergyInWatts'] = scaler.transform(train[['EnergyInWatts']])
test['EnergyInWatts'] = scaler.transform(test[['EnergyInWatts']])

seq_size = 30

def to_sequences(x, y, seq_size=1):
    x_values = []
    y_values = []

    for i in range(len(x)-seq_size):
        #print(i)
        x_values.append(x.iloc[i:(i+seq_size)].values)
        y_values.append(y.iloc[i+seq_size])

    return np.array(x_values), np.array(y_values)

trainX, trainY = to_sequences(train[['EnergyInWatts']], train['EnergyInWatts'], seq_size)
testX, testY = to_sequences(test[['EnergyInWatts']], test['EnergyInWatts'], seq_size)

model = Sequential()
model.add(LSTM(128, input_shape=(trainX.shape[1], trainX.shape[2])))
model.add(Dropout(rate=0.2))
model.add(RepeatVector(trainX.shape[1]))
model.add(LSTM(128, return_sequences=True))
model.add(Dropout(rate=0.2))
model.add(TimeDistributed(Dense(trainX.shape[2])))
model.compile(optimizer='adam', loss='mae')
model.summary()

history = model.fit(trainX, trainY, epochs=10, batch_size=32, validation_split=0.1, verbose=1)
```

If each of your three features is a scalar then my first attempt would be to combine them into a vector for each step in the sequence. So instead of LSTM(128, input_shape=(30,1)) for a length-30 univariate sequence you would say LSTM(128, input_shape=(30,3)) for a multivariate (3) sequence. Similarly your output would become TimeDistributed(Dense(3, activation='linear')).
If your input features are each a vector then you should consider switching from the sequential API to the functional API. You would need three separate Input objects followed by a Concatenate layer before the LSTM layer. You would have three Dense layers to be your corresponding outputs. I tried adapting your code:
in0 = Input(shape=(trainX0.shape[1], trainX0.shape[2]))
in1 = Input(shape=(trainX1.shape[1], trainX1.shape[2]))
in2 = Input(shape=(trainX2.shape[1], trainX2.shape[2]))
concat = Concatenate()([in0, in1, in2])
lstm_enc = LSTM(128)(concat)
lstm_enc_drop = Dropout(rate=0.2)(lstm_enc)
rep = RepeatVector(trainX0.shape[1])(lstm_enc_drop)
lstm_dec = LSTM(128, return_sequences=True)(rep)
lstm_dec_drop = Dropout(rate=0.2)(lstm_dec)
out0 = TimeDistributed(Dense(trainX0.shape[2], activation='linear'))(lstm_dec_drop)
out1 = TimeDistributed(Dense(trainX1.shape[2], activation='linear'))(lstm_dec_drop)
out2 = TimeDistributed(Dense(trainX2.shape[2], activation='linear'))(lstm_dec_drop)
model = Model(inputs=[in0, in1, in2], outputs=[out0, out1, out2])
model.compile(optimizer='adam', loss='mae')
model.summary()

My 3 inputs would be 3 columns of time series values e.g. 0.31314 0.15891 0.41713. Would the output of concatenate be a single vector i.e. [0.31314 0.15891 0.41713] or a single value?

Concatenate would produce a 3-dimension vector. The output tensor from the Concatenate layer will be of form (batch, steps, features) so your model summary should report an output tensor shape of (None, 30, 3) given your 30 time steps and 3 features.
I think that using multiple Inputs with Concatenate would be functionally equivalent to changing your input shape to (30, 3) and providing it with the 30x3 input data array. That way you wouldn't need to worry about changing from Sequential to Functional.


text:
Is a very powerful oracle sufficient to trigger the AI singularity?
Lets say we have a oracle $S$ that, given any function $F$ and desired output $y$,  can find an input $x$ that causes $F$ to output $y$ if it exists, or otherwise returns nil. I.e.:
$$S(F, y) = x \implies F(x) = y$$
$$S(F, y) = nil \implies !\exists  x \hspace{10px}s.t.\hspace{10px} F(x) = y$$
And $S$ takes $1$ millisecond to run (plus the amount of time it takes to read the input and write the output), regardless of $F$ or $y$. $F$ is allowed to include calls to $S$ in itself.
Clearly with this we can solve any NP-Complete problem in constant time (plus the amount of time it takes to read the input and write the output), and in fact we can go further and efficiently solve any optimization problem:
def IsMin(Cost, MeetsConstraints, x):
  def HasSmaller(y):
    return MeetsConstraints(x) and Cost(y) < Cost(x) and y != x
  return MeetsConstraints(x) and S(HasSmaller, True) == nil

def FindMin(Cost, MeetsConstraints):
  def Helper(x):
    return IsMin(Cost, MeetsConstraints, x)
  return S(Helper, True)

Which means we can do something like:
def FindSmallestRecurrentNeuralNetworkThatPerfectlyFitsData(Data):
  def MeetsConstraints(x):
    return IsRecurrentNeuralNetwork(x) and Error(x, Data) == 0
  return FindMin(NumParamaters, MeetsConstraints)

And something similar for any other kind of model (random forest, random ensemble of functions, etc.). We can even solve the halting problem with this, which probably means that there is some proof similar to the halting problem proof that shows such an oracle could not exist. Lets assume this exists anyway, as a thought experiment.
But I'm not sure how to take it from here to something that achieves endless self improvement. What exactly the "singularity" even means I suppose is tricky to define formally, but I'm interested in any simple definitions, even if they don't quite capture it.
A sidenote, here is one more function we can do:
IsEquivalent(G, H):
    def Helper(x):
      return G(x) != H(x)
    return P(Helper, True) == nil

"Clearly with this we can solve any NP-Complete problem". Note that, in principle, you can  already solve any NP-complete problem, but you need exponential time to do so, unless someone proves $P = NP$: in that case, there would a polynomial-time solution for each NP-complete problem.

In general, in this question, you're talking about something you don't even know well. So, your assumptions and conclusions will likely not lead anyone anywhere. I would encourage you to read the book "Introduction to the Theory of Computation" by Sipser before trying to understand if the singularity is possible.

AFAIK, the singularity is not even a well defined or mathematically formalised problem. You may want to attempt to do that. But I guess you will need to work hard.

@nbro sorry, I added an edit to clarify what I meant (with this oracle, we would do it very efficiently). I'm not sure where you are getting your assumptions that I don't know these things well: I have read through all of Sipser (it is a good recommendation) and understand complexity classes and such. I'm even a published author in multiple computer science academic conferences.
I know such an oracle (probably) isn't possible, but I don't see why such a thought experiment isn't interesting regardless? There are all kinds of questions we study in computer science about oracles.

@nbro "the singularity is not even a well defined or mathematically formalised problem" I agree. Another way to phrase my question could be "what kind of self-improving behavior could we get with such an oracle, and how close does this behavior match common intuitions of what the 'singularity' might be?"

I think a better question on this website would be: "Can we define the AI singularity mathematically"? You may want to have a look at AIXI. Anyway, I think that the oracle (as you define it) would just provide speed of learning.

Done, thanks for the suggestion, but I feel this question is different enough that I won't remove it.

Regarding AIXI, you may want to have a look at https://ai.stackexchange.com/a/10377/2444.


text:
Can augmented reality be a training system for computer vision?
Is augmented reality a training system for computer vision? As in, Augmented systems use their data to help train computer vision algorithms, or is augmented reality computer vision itself?

Let me re-frame your question before answering:

Can augmented reality be a training system for computer vision?

Yes, why not! Also, I'm pretty sure the big companies who have their aug. reality products out there are secretly optimizing their CV algorithms(although we'll have no proof).
A very similar(but a different) example would be how Comma.ai's app pays users for recording their driving, and then uses it to train their own self-driving cars.
So, if that can be done, I don't see why augmented reality cannot be used to train CV algos. In fact, it's a much more richer, localized dataset.

Noted and corrected. Thank you.
I thought it would make sense, since it is quite rich data. From what i understand about AR, it is analyzing its field of view which is also what CV will do.. right?
This Comma.ai is new to me, is it similar to what Andy Rubin is doing with dash cams? Or at least plans to do with them? Giving them out for free with state of the art CV to build a network of connected dash cam data, real time.

@James Yeah, something like that. Comma.ai's app records your driving videos. You need to install the cam on your windshield, and it keeps recording, and that footage is used in their self-driving cars example.


text:
Can BERT be used for sentence generating tasks?
I am a new learner in NLP. I am interested in the sentence generating task. As far as I am concerned, one state-of-the-art method is the CharRNN, which uses RNN to generate a sequence of words.
However, BERT has come out several weeks ago and is very powerful. Therefore, I am wondering whether this task can also be done with the help of BERT? I am a new learner in this field, and thank you for any advice!

Can OpenAI GPT be used for this? I believe OpenAI GPT has different architecture and is used for text generation

I believe CharRNN is definitely not SOTA, due to the limited context length, from working at the scale of characters. Instead, there is work on subwords, and byte-pair encodings

For newbies, NO.
Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its bidirectional nature.

For advanced researchers, YES.
You can start with a sentence of all [MASK] tokens, and generate words one by one in arbitrary order (instead of the common left-to-right chain decomposition). Though the text generation quality is hard to control.
Here's the technical report BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model, its errata and the source code.

In summary:

If you would like to do some research in the area of decoding with
BERT, there is a huge space to explore
If you would like to generate
high quality texts, personally I recommend you to check GPT-2.

Wouldn’t it be easier for the model, though, to generate high quality text if you let it generate the words from left to right, just like with a language model, instead of having it generate the words in an arbitrary order? I can just imagine if I had to write all words in an arbitrary order whenever I wrote a sentence; it would be pretty difficult to know which word I should write at position seven, for example, if I hadn’t written any word before that.

this experiment by Stephen Mayhew suggests that BERT is lousy at sequential text generation:
http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/

although he had already eaten a large meal, he was still very hungry

As before, I masked “hungry” to see what BERT would predict. If it could predict it correctly without any right context, we might be in good shape for generation.
This failed. BERT predicted “much” as the last word. Maybe this is because BERT thinks the absence of a period means the sentence should continue. Maybe it’s just so used to complete sentences it gets confused. I’m not sure.
One might argue that we should continue predicting after “much”. Maybe it’s going to produce something meaningful. To that I would say: first, this was meant to be a dead giveaway, and any human would predict “hungry”. Second, I tried it, and it keeps predicting dumb stuff. After “much”, the next token is “,”.
So, at least using these trivial methods, BERT can’t generate text.

No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). Because of bi-directionality of BERT, BERT cannot be used as a language model. If it cannot be used as language model, I don't see how you can generate a sentence using BERT.

My answer is no longer correct. You may want to accept @soloice 's answer

Do alternate options exist for this?

Could one generate a sentence that doesn't make sense?  My understanding is the possible combinations of natural language words in a string is infinite (definitely countably infinite, but possibly uncountably infinite.)  Although most of the text generated would lack semantic significance, that random generation would also produce plenty of meaningful natural language sentences, given sufficient time and space?

@Astariul Your answer was never correct, as it was possible all along to generate text with BERT; we just hadn’t realized it yet ;)


text:
Can we teach machine to tie shoe lace?
Is it possible with any of machine learning methods to train machine to tie shoe lace? If possible how data should be interpreted for the training?
If we are using reinforcement learning, how will it learn to reach the best rewards?

Great question.  Very smart people are thinking about knots!
https://en.wikipedia.org/wiki/Conway_notation_(knot_theory)

@DuttaA it's a good point for certain questions, but what if this is just a high-level, theoretical question from a curious mind?

@DuttaA :)  One think I love about Conway's notation is that he uses it to teach children at math camps.  Apparently, there are a fair few kids who are interested in knots, and the field is surprisingly complex.  But as "theorics" are on topic here, I do quite like this question!

You would not believe how difficult this task is, assuming you want a humanoid robot to tie laces with humanoid hands. It’s possible of course but compared to what the current state of the art in machine learning is, this task is very very complex because it’s a physical system with unknown variables (Eg. coefficient of friction on laces), physical limitations (Eg. dexterity of robotic hands), and occluded vision (Eg. hand in front of laces) to name a couple of the issues one faces not to mention robots are expensive but I digress.
Robots can perform amazing preprogrammed sequences but if you wanted to have a robot to be able to tie any shoe in any situation, machine learning is the right approach.
Training data should include all of the things that humans use when they approach this problem:

proprioception (the location of joints in space aka where are your fingers)

a sense of touch, a robot would need a quite a few touch sensors
vision, although not actually necessary (because blind humans are capable of learning to do this task), it may speed up learning

Using reinforcement learning as a framework, one needs to be able to give the RL agent a signal that it is closer to tying the lace. This could be done in a few ways and it’s unclear what would be the best.
One method would be to train a separate model using supervised learning and pictures of shoes laced up to look at the current state of the bow (or lack thereof) and report if it looks like a tight bow. This method requires a whole other network but it may be the most versatile in the end although the RL agent may just learn how to place laces so it looks like a tight bow.
Although simple for humans, this is an interesting problem and thinking about how it could be implemented helps one understand how to apply ML in other areas.


text:
Concrete examples of unintentional adversarial AI behaviour
Are there any real-world examples of unintentional "bad" AI behaviour? I'm not looking for hypothetical arguments of malicious AI (AI in a box, paperclip maximizer), but for actual instances in history where some AI directly did something bad due to its direct action that was not intended behavior.
Interpretations of the meaning of "AI", "bad", and "unintentional" are left open due to obvious reasons. Be free with your interpretation, but use some common sense please.
Ex: Microsoft Tay became a racist not too long after being hooked up to the internet, thanks to internet trolls "teaching" her bad things.
I can't think of any other instances. So the following examples are just hypothetical scenarios to demonstrate what I mean.
Ex: A self-driving car drove off-track after being presented with an adversarial example, crashing into people.
Ex: Surgery bot goes haywire and accidentally kills someone.
Ex: Weaponized drone targets civilians against its design.

Define bad.

question edited to address your problem.

And that makes it even more broad. Does "bad" means unemployment or AI enslaving humans ? What exactly is it ?

I dunno. The examples you provide suggest you have an idea of what to expect, so do you have any examples of adversarial AI behaviour in the wild under those interpretations? "Be free with your interpretation, but use common sense please."

By "in the wild", you mean fully independent automata, not directed by humans?  (i.e. partisan, adversarial investment algorithms are not "in the wild") I think the use of the idiom is good, but just wanted to make sure I'm understanding your meaning.

question edited to address your problem. more specifically addressed to you: i'm just looking for instances that has "actually happened before", as opposed to being hypothetical (ai in box)

I believe any example of overturned AI sentence in court would count. I don't know of any - but I'm quite scared to think how many innocent people may be in prison, because the AI judged basing on ethnicity, social background, education level and other "risk assessment" factors, while downplaying facts of that specific case. Knowing such algorithms -, if 30 statistical factors say "he's the type of person that would do it" and 1 individual fact says "he didn't do it - he wasn't there", the AI will say "guilty."

the algorithm in question doesn't sentence or give verdict in court. it provides a metric for risk assessment, which judges then use when sentencing people. although strictly speaking this example fits the criteria above, its not really what i'm looking for. perhaps you also think this way even just a little, from the observation that you chose to comment here rather than post a full fledged answer.

There is the case of the tesla accident where the car was in autopilot and crashed into a truck because it appears the vehicle mistook a lightly coloured truck for the sky, killing the driver: https://www.newscientist.com/article/2095740-tesla-driver-dies-in-first-fatal-autonomous-car-crash-in-us/
Having said that, it appears the car had been trying to tell the driver to pay attention: http://uk.reuters.com/article/us-tesla-crash-idUKKBN19A2XC

The infamous Flash Crash of 2010 may qualify.
It didn't involve Artificial General Intelligence (which is still a hypothetical) or even "strong narrow AI" (such as AlphaGo) but does involve algorithmic decision-making, which is a form of basic Artificial Intelligence.
Algorithmic trading already represents a significant percentage of all market activity, and I suspect that percentage will only increase.
From Business Insider: Algos could trigger the next stock market crash


text:
Concrete examples of OpenCog's functionality
Does anyone know what specific tasks the OpenCog environment is capable of performing? I have glanced though their wiki and a few of the pages on Goertzel's site and the AI.SE. So far I could only find some technical documentation regarding theory and engineering, but nothing on concrete results.
From the technical description of AtomSpaces it seems that OpenCog is capable of some "representational inference", but I haven't come across any sources that concretely describes what it is capable of doing.
Apparently there is some collaboration between Sophia the Robot and OpenCog, but to what extent I am unclear. I am aware however that the dialogue functions is powered by ChatScript (though I also suspect that the high profile interviews Sophia gives are completely scripted...)
Can anyone provide concrete examples or evidence of OpenCogs' functional behavior. Like transcripts of chat, examples of reasoning, video or demonstrations of its emotion-emulating; and not just claims of functions.

It depends what you mean by "what OpenCog can do?".
OpenCog, at a high-level, is a loosely coupled collection of various theoretical and variants of conventional methods aimed at constructing the beginnings of an AGI.
With that said, it's purely applied uses are fairly limited. It can do some typical NLP and ML tasks if used correctly, albeit, it will almost always be ineffective when compared to a problem specific solution. However, this is understandable, as OpenCog is not geared at solving narrow problems.
OpenCog, as well as Goertzel's other projects(SingularityNet, AGI society), are broad, top-down attempts at formulating an AGI system.
Outside of AGI circles they are also not oft referenced or researched(although, I am utilizing some aspects of the system in work I am currently crafting).


text:
Examples of single player games that use modern ML techniques in the AI?
Are there any examples of single player games that use modern ML technique in its games? By this I mean AI that plays with or against the human player, and not just play the game by itself (like Atari).
"Modern ML techniques" is a vague term, but for example, Neural Networks, Reinforcement Learning, or probabilistic methods. Basically anything that goes above and beyond traditional search methods that most games use nowadays.
Ideally, the AI would be:

widely available (i.e. not like the OpenAI Five, which was only available for a limited amount of time and requires a high amount of computational power)
human level (not overpowered)

Ideally, the game would be:

symmetrical (the AI has the same agent capabilities as the player, though answers similar to The Director would be very interesting as well)
"complex environment" (more complex than, say, a board game, but a CIV5 game might work)

But any answer would be appreciated, as some of the criteria above are quite vauge.
Edit: the ideal cases listed above are not meant to discourage other answers, nor are they intended to be of strictly inclusionary (ie: any game would need to satisfy all of the above requirements)

There is Google Research Football, which is an open-source platform to develop reinforcement learning algorithms to play a game similar to FIFA or PES, although the football simulation is not as realistic as the current versions of FIFA or PES. You can play this game against different RL agents (e.g. DQN or IMPALA) and, of course, you can even develop your own RL agents and play against them. Here is a video that illustrates the environment. Here is the code and instructions to use it.
As far as I know, there isn't yet an AI that plays simulated football at a human-level (i.e. as good as the best human players). For example, I can regularly (although not always) beat the legendary level-AI at FIFA, but I also don't know the details about this AI (which could also be rule-based).

Of course, although you are looking for something more complex than a board game and that is just human-level (rather than superhuman), I think it is important to mention AlphaGo (and related algorithms/models/systems).

Beating the World’s Best at Super Smash Bros. Melee with Deep Reinforcement
Learning
Firoiu, Whitney, Tenenbaum created a RL agent that plays and defeats professional players in Super Smash Bros Melee. The RL agent first played against the built-in AI, and then via self-play.
Only one character playing on a single stage was trained. The character picked (Captain Falcon) has no "projectile attacks" to simplify training.

Not sure if this fits your requirements of the AI playing with the player, but I still wanted to mention it because to me it is the quintessential AI-based game:
AIDungeon, which is a text based story-telling game, where you can do literally anything. It's using GPT-2/GPT-3(paid) and has blown my mind several times. You've probably heard of it, but in case you haven't give it a try, only takes a couple of minutes to see what it can do.

i didn't find this to be nearly as playable as or interesting as the (very) handful amount of other text based rpgs have been

i have decided to award this answer with the bounty, since i have since found it to be quite expressive and interesting. however, it seems to lack coherency and breaks at times. its has given me very interesting dialogues however, but as i don't quite consider it to be a game, i will not mark this as the "correct" answer

I don't know about specific game titles but in terms of research University of Malta has a strong team working with application of machine learning to games. The key figure there used to be Georgios N. Yannakakis who published a lot of good papers and even wrote a book about content generation, smart game agents and, imho the most interesting, player modelling.

unfortunately this is paywalled, so i can't quite assess whatever is being discussed here, but thank you for the link. a paper mentioned in an article that introduces the book cites the paper "Predicting Player Behavior in Tomb Raider: Underworld" by Mahlmann et al. but i didn't find the object of prediction (when the player decides to stop) as very interesting.

There are many games where AI involved but less of them against the human player such as playing the video game. For example in this paper, they proposed a three-dimensional multiplayer first-person video game called Quake III Arena in Capture the Flag mode. Also, this paper and this paper show many games where AI involved and they show which kind of games that a human player can play against. I recommend also Awesome-Game-AI.


text:
Java - A good place to begin if over all goal is ML and Ai?
I am currently studying Java (Se && EE). I am wondering if it is a good platform for developing ML algorithms for AI.
Areas of interest: facial rec - Speech Rec - understanding conversation in group conversations.
Financial Institutions: Risk assessment ML, etc.

Hello james.professionally,I would suggest python and C/C++ for instance;Python has several powerful libraries or packages,that will help with machine learning such as NumPy.
look at C/C++; many machine learning algorithms are processor constrained so AI software engineers try to get as close to the metal as they can. C++ has a speed advantage over Java much (but not all) of the time. Java is often better for big collaborative projects.Also go through this post http://ai.stackexchange.com/questions/3019/can-lua-be-a-stable-portable-and-powerful-language-for-ai

Hello! Thank you for the information. I knew Python was held in high regard when it comes to Machine Learning. I have heard brief talks about Java in ML specifically Hadoop. I Had plans to start dabbling in Python at some point and with answers such as yours above, I think it would be a wise choice. Thank you!

We're just finishing up a development cycle for mobile applications utilizing Java and Objective C.  We're likely going to recode in C++, not only for the speed advantages, but to eliminate the need for distinct languages on iOS and Android. (i.e. if you're thinking about mobile applications, C++ has advantages.)

Sounds interesting, but why C++ in terms of advantages on mobile? apart from delivering better speed than java, Im not sure why.

You should see OpenCV. It has some very good libraries for Machine Learning and Artificial Intelligence related kinds of stuff. Also, you can easily import them in python, Java and C++.

If goal is to develop ML algorithms then focus on Maths concepts linear algebra, probability and statistics. Try out CS problem solving basic data structure and algorithms.
Python has good ML libraries but if you know java then you can pick python easily.

Thank you for your reply, @Vikram . I am currently studying Linear in my spare time and have a general interest in statistics. Will take a look at CS problem solving. Watching a lot of videos recent on Python. It does seem it may be quite an easy transition.

As others have pointed out, your level of maths/statistics knowledge is probably more important than your chosen programming language.  That is, particularly true w/r/t developing (presumably new) ML algorithms.  OTOH, from an "applied ML" perspective, where you just use pre-baked implementations of existing algorithms, one of the big questions is "do good libraries for these various operations exist in language $X"?
Where the value of $X is "Java" the answer is "yes".  There are tons of high-quality libraries of most popular and widely used ML algorithms.  There are also tons of libraries for nearly everything else, which helps when constructing wider systems which incorporate elements of ML/AI.
That said, w/r/t machine learning specifically, there are probably more extant libraries and what-not in Python or R, than in Java.  Also, from a "rapid development" point of view, you may find that Python has some advantages as as of a result of its dynamic typing and lack of "boiler-plate" as found in Java.
Net-net, I'd say Java is a fine choice, along with Python or R, with C++ also in the mix.  It may not be the perfect choice, but it's absolutely "good enough" and then some.
If you want a feel for some of the projects that exist now in various languages, to go mloss.org and use the filter by language feature.  Click around there and examine some of the options you find.

You can't "develop" ML algorithms without statistical knowledge, it simply doesn't work that way and it's impossible.
Programming is cheap for modelling, anyone who has done a computer science degree can do it. It's just like giving some inputs and giving something back. Lots of framework can do that for you, and it's easy.
In order for you to "develop" ML algorithms, you should pursue a mathematics degree. Generally, you're expected to have a PhD or something similar to "develop" ML algorithms.
If you're really interested in programming, you should do R and Python. Java is not a common data science programming language.


text:
Image comparison algorithm, trying to figure out how similar two "binary" forms are
I'm a student I'm completely new to this technology maybe my approach could be completely wrong, I want to create an algorithm that compares the similarity between two binarized images.
I'll explain:
I have 2 pictures as input. The RGB colors of these images can only be 0 or 255
(R = G = B = 255) or (R = G = B = 0). I take these two letters as an example.
1.  2.
I thought so: the 255 value is the background of the image which is white.
The 0 value is the shape (letter) formed in the image. So I thought of creating a matrix with 0 and 1 values where value 0 represents the background and value 1 represents the shape.
So now i would like to create an algorithm that understands the shape created in the two matrices and that returns a similarity percentage.
Update: I'm creating this app that tries to recognize the font of a text in a image (https://github.com/Sirvasile/Typefont), I want to create this algorithm to improve the comparison between the input letters and the alphabet of my fonts in the database.

I think you need to define what you mean by shape. One could compute the hamming distance between the images and get an answer but I don't think that is what you want. Try looking up convolutional NNs and MNIST. It should help.

I think you look for the meaning of the image (pic of 'a' = a) rather than the regular image similarity. For example, you would like the two images of the 'a' you posted to be classified to the same label instead of, for example, to be classified as similar to an 'o' image which would give a smaller Hamming distance or mean square error.
So as @Jaden Travnik suggested, MNIST digits classification is a good start. It classifies handwritten digits and has human-level solutions implemented by any existing CNN framework. (TensorFlow, Caffe, Theano, to name a few).


text:
Precise localization and characterization of rudimentary shapes with neural networks
I understand that there are flavors of (convolutional) neural networks that are useful for object localization and detection tasks of reasonable difficulty. In all of the examples I have seen so far, localization is formulated as finding the corners of a bounding box. Often, the fit is not expected to be very precise:

Conversely, I am interested in a task I want to achieve very precise localization and characterization of some simple shapes or objects. As an example of one of the simplest cases I can think of, my inputs will be images like the following:

Given this 60x60 image, I want my neural net(s), via regression, to tell me that the circle's diameter is 18px and its centre is located at (28, 21) from top left. (I will train it using similar 60x60 images with white circles of various sizes on black backgrounds.)
Later I am interested in dealing with similar tasks in the real world, e.g. spheres/cubes/cylinders with different viewing angles, light conditions, occlusions, etc. However, I am interested in solving this simplest case first. (One reason is that I can generate this data very easily.)
I have the following specific questions:

Has anyone used neural nets for this sort of tasks before? (e.g. precisely determining sizes and centroids of objects)
My understanding is that these things are at least theoretically possible using convolutional nets, or even sufficiently complicated vanilla fully connected nets. Is this correct?
What architecture(s) would be appropriate for these tasks?

Note: I am aware that fitting a bounding box to the circles and calculating its centre and size will solve this particular case, but it will not generalize to handle occlusions, changing lights, etc. I would like to move towards a method which can, for example, calculate the centroids and diameters of spheres in real-world B&W photos.

I think you're describing "object localization and detection" which combines object identification with discovery of its spatial placement in the field of view.  There's been a lot of work on this in the past 5 years using CNNs and variants.  It's very much in demand for pedestrian/obstruction detection, identification, and avoidance in autonomous car systems.
Here's a place to start:  https://github.com/kjw0612/awesome-deep-vision
Especially, take a look at "Faster R-CNN" by Ren et al, 2016.


text:
Why do action recognition algorithms perform better on ucf101dataset than HMDB51 dataset?
If we look at state of the art accuracy on the UCF101 data set, it is around 93% whereas for the HMDB51 data set it is around 66%. I looked at both the data sets and both contain videos of similar lengths. I was wondering if anyone could give an intuition as to why HMDB51 data set has been harder.

It is true that at first look, one could expect that classification between 101 categories would be harder than classification between 51 categories. However, many aspects play a role when it comes to action recognition applications.
For instance, the HMDB51 contains several categories about different facial movements like smiling, laughing, chewing,... and several other categories like eating, drinking. Such categories are not present in the list of categories of the data set UCF101 and are obviously among the most difficult categories to deal with. It also claims to have some bad quality challenging videos.
It is hard to predict in advance how difficult a data set will be to classify. We can imagine that when the state-of-the-art reach accuracy beyond 90%, it is time to build a data set that makes these methods fail to look for even more robust solutions. I don't know these data sets well, but the videos are present most probably more variability in terms of viewpoint, camera motions, illumination changes, image quality,... in the most difficult to classify data set.
Also, check on this page, the results announced for the UCF101 data set. I don't know where you found you accuracy value because the official website announces less than 43.9%. Some publications do not use the complete data set and use only part of it to show the performance of an approach they designed.
Finally, the official website of the HMDB51 data set reports the following:
"The UCF group has also been collecting action datasets, mostly from YouTube. There are UCF Sports featuring 9 types of sports and a total of 182 clips, UCF YouTube containing 11 action classes, and UCF50 contains 50 actions classes. We will show in the paper that videos from YouTube could be very biased by low-level features, meaning low-level features (i.e., color and gist) are more discriminative than mid-level fears (i.e., motion and shape)."
This could also explain why better results can be achieved...

The addition of clips from feature movies in the HMDB51, which do not appear in the UCF101 is one difference I noticed in the literature, but not enough to explain such a great disparity as the question author experienced. The idea about the facial movements seems the most likely. I have no better hypothesis, so I up-voted.

I found HMDB51 dataset to be so annoying , i personally looked at videos and found they dont even have a proper sequence of actions , the camera angle changes anytime.In short they are not realistic videos thats why DL or ML models dont perform well on those


text:
Change parameter in Karaboga's code of ABC algorithm
I'm working on a problem and need to use Karaboga's code of the ABC algorithm but I have some questions...
Does this formula for calculating a parameter have to be changed:
{/v_{ij}=x_{ij}+\phi_{ij}(x_{kj}-x_{ij}) */}

A standard one or this what Karaboga see is better for algorithm.
The second is the same question for this formula of calculating fitness:
{fFitness(ind)=1./(fObjV(ind)+1)}

Link to ABC algorithm coded using C programming language


text:
Can I do oversampling by copying the same image multiple times? Will it effect my neural network accuracy?
I am working on an image data-set. As you may have guessed it is imbalanced data. I have 'Class A, 19,000 images' and 'Class B, 2,876 images'.
So I did an undersampling by removing randomly from the majority class till it becomes equal to the minority class.
On doing this I am loosing lot of information from those 19000 images which I could get.
So I do an oversampling of minority class, by simply copying the 2,876 images again and again.
Is this undersampling method correct, will it effect my accuracy? I trained an Inceptionv4 model using this oversampled data and it is not at all stable and I am getting poor accuracy.
What should be my strategy ?

This would be a pretty fun opportunity to experiment a bit. I know there's been breakthroughs in OCR technology by training the network on images that have been altered.
Long story short, there was a pre-process that took the source images and did minor manipulations to them. This involved skewing, rotating, changing resolution & adding artifacts into the image to introduce "Noise". This allowed the network to learn to recognize characters more accurately. It may be worthwhile to create an image pre-processing pipeline that will do a variation of modifications to the source images. When training you may want to lower your learning rate & split up your data into 70% train & 30% test. Then you can loop through all the images repeatedly until you hit a maximum for the learning rate.
Some examples
https://matthewearl.github.io/2016/05/06/cnn-anpr/
https://medium.com/@shreyas.s/image-data-generation-for-optical-character-recognition-ocr-9b19300649c8

So if I do an augmentation of the minority class, of 2000 images to bring it to 19000 images. bringing the data from 10% to 90% using augmentation,don't  you think it will skew my data, create a bias issue. Basically I am flipping rotating , grey scaling,other than that I am not bringing in any new feature information necessary for the classifier to learn.As practically a flipped image or grey scale image would be a rare thing in my case since i will standardise my testing  data.What is your take on this ?

None of these algorithms mean anything without data. The model may not reach the full potential without more source images. It can become biased so that it over samples the class with 2000 images and it's over-represented in your results. The way you train it may also affect how much of an impact that will be.


text:
Studying the speech-generation model and have question about the confusing nature of model input and outputs
I am currently studying this model speech generation known as WaveNet model by Google using the linked original paper and this implementation.
I find the model very confusing in the input it takes and the output it generates, and some of the layer dimensions didn't seem to match based on what I understood from the WaveNet paper, or am I misunderstanding something?

What is the input to the WaveNet, isn't this a mel-spectrum input and not just 1  floating point value for raw audio? E.g. the input kernel layer shows as shaped 1x1x128. Isn't the input to the input_convolution layer the mel-spectrum frames, which are 80 float values * 10,000 max_decoder_steps, so the in_channels for this conv1d layer should be 80 instead of 1?

inference/input_convolution/kernel:0 (float32_ref 1x1x128) [128, bytes: 512]

Is there reason for upsampling stride values to be [11, 25], like are the specific numbers 11 and 25 special or relevant in affecting other shapes/dimensions?

inference/ConvTranspose1D_layer_0/kernel:0 (float32_ref 1x11x80x80) [70400, bytes: 281600]
inference/ConvTranspose1D_layer_1/kernel:0 (float32_ref 1x25x80x80) [160000, bytes: 640000]

Why is the input-channels in  residual_block_causal_conv 128 and residual_block_cin_conv 80? What exactly is their inputs? (e.g. is it mel-spectrum or just a raw floating point value?) Is the wavenet-vocoder generating just 1 float value per 1 input mel-spectrum frame of 80 floats?

inference/ResidualConv1DGLU_0/residual_block_causal_conv_ResidualConv1DGLU_0/kernel:0 (float32_ref 3x128x256) [98304, bytes: 393216]
inference/ResidualConv1DGLU_0/residual_block_cin_conv_ResidualConv1DGLU_0/kernel:0 (float32_ref 1x80x256) [20480, bytes: 81920]

I was able to print the whole WaveNet network using print(tf.trainable_variables()), but the model still seems very confusing.

There are two "inputs" into Wavenet:

the previously generated samples of the waveform, which are usually encoded into multiple channels, like into 256 channels using 8-bit mu-law encoding
local conditioning, which can be things like linguistic features such as phoneme classes (used in the original wavenet paper), or frequencies like mel spectrogram values (used in the Tacotron 2 paper)

The local conditioning signal usually has a much lower resolution than the waveform itself. For example, the Tacotron 2 paper mentioned using "50 ms frame size, 12.5 ms frame hop" to derive its mel spectogram values. At 24 kHz, each waveform sample has a duration of just 0.0416 ms. So in order to using the spectogram information to condition the waveform generation, it would have to be upsampled to spread out along the time dimension. (If you were locally conditioning using letters, you might turn "dog" into "dddddoooooooooggg" in order to use the letter "d" to generate 5 samples in the output waveform, etc.).

What is the input to the WaveNet, isn't this a mel-spectrum input and not just 1 floating point value for raw audio?

In the implementation you linked to, it looks like variables that start with "c" refer to "conditioning" signals. So "hparams.cin_channels" indicates how many channels the input conditioning signal has. If that signal is mel spectogram with 80 channels, then it would be set to 80.
It looks like the "inference/input_convolution" layer is processing the actual waveform, not the spectogram conditioning signal.

Is there reason for upsampling stride values to be [11, 25], like are the specific numbers 11 and 25 special or relevant in affecting other shapes/dimensions?

I suspect (though I'm not too familiar with tensorflow conventions) that the second dimension is time, which is why you're seeing the model upsample the conditioning signal in that direction. Those values may be required to match the resolution of the waveform being generated.

Why is the input-channels in residual_block_causal_conv 128 and residual_block_cin_conv 80? What exactly is their inputs?

The "c" in "cin_conv" appears to mean that it is for the conditioning signal, which is why is has a different dimension than the waveform's embedding size.
I hope this helps. You could also try opening up an issue on that github repo to try getting help directly from the author.


text:
Model unfit for some part of spiral data despite low error
I'm current testing a model for spiral data. After 500 epoches, loss is 0.04 but the result is still unmatch with some part of the training data. (bottom left)

(source: upsieutoc.com)
The model has 2 hidden tanh x 16 units running with ml5.js. I choose tanh because it seems to be smoother than ReLU. Apart from that they're the same.
Is this thing caused by the model or by ml5 itself?

It must be for ML5. The data work fine in TFJS.


text:
How can I train a neural network for another input set, without losing the learning of the previous input set?
I read this tutorial about backpropagation.
So using this backpropagation we are training the neural network repeatedly for one input set, say [2,4], until we reach 100% accuracy of getting 1 as output. And the neural network is adjusting its weight values accordingly. So once after the neural network is trained this way, suppose we are giving another input set, say [6,8], also then will the neural network update its weight values (overwriting previous values), right? This will result in losing the previous learning, right?

Yes, this is actually a limitation known as catastrophic forgetting.
A proposed way to deal with this is elastic weight consolidation that "remembers old tasks by selectively slowing down learning on the weights important for those tasks". See Overcoming catastrophic forgetting in neural networks for details. Another approach is Learning without forgetting.
If the tasks are different, the approach you are talking about is called transfer learning. You might want to have a look at multi-task learning as well
If the tasks are the same, you could try creating a join of both datasets and training on that.


text:
What is the difference between Kaldi and DeepSpeech speech recognition systems in their approach?
I would like to know how do Kaldi and DeepSpeech speech recognition systems differ algorithmically? Which one would be more accurate for continuous speech in time?

Both of them using the end-to-end approach for speech recognition. However, because of the code complexity in DeepSpeech, you can't tune the model for your work. Kaldi could be configured in a different manner and you have access to the details of the models and indeed it is a modular tool. I think Kaldi could be a better tool academically and also commercially. But, Deepspeech is a BlackBox and could be a proper tool if your work is near to the work of DeepSpeech.
Moreover, if you are working in a language that there is not enough data for learning, you have a dozen of tools such as grapheme-to-phoneme to establish your dataset to start the learning process. Also, you can start with other models of Kaldi to work on which do not depend on the big data to be learned. However, DeepSpeech requires many hours of samples that is not a common asset for everyone in every language.
Also, you can know more about the traditional technique of speech processing vs the contemporary technique (deep learning) in this medium post.

thanks a lot for your reply. As you touched the subject deeply, is it possible to say the difference between them algorithmically like GMM/HMM or LSTM/RNN models they use. how to do they differ here? Thanks once again for such depth.

@Ramki my pleasure. I think this link: https://medium.com/@jonathan_hui/speech-recognition-gmm-hmm-8bb5eff8b196 could be useful.


text:
What are the mathematical properties of natural exponential function that lead to its usefulness in artificial intelligence?
In mathematics, there is a proof that the following infinite series converges to a constant irrational number, denoted by $e$, called as Euler's number or Euler's constant or Napier's constant. The value of $e$ lies between 2 and 3.
$$1 + \dfrac{1}{1!} +  \dfrac{1}{2!} +  \dfrac{1}{3!} + \cdots$$
The natural exponential function, defined as follows, has some interesting properties
$$f: \mathbb{R} \rightarrow  \mathbb{R}$$
$$f(x) = e^x$$
It is used in several algorithms and in the definitions of functions like SoftMax.
I am interested in knowing the possible mathematical characteristics that lead this function useful in artificial intelligence.
The following are the properties I am aware of. But, I am not sure about how some of them will be useful

Non-linearlity: Activation functions are intended to provide non-linearity. So, it is a candidate for activation functions due to this property. You can check its graph here.

Differentiability:  Loss functions used for in back-propagation algorithm need to be differentiable. So, it can be a candidate for usage in loss functions too.

$$\dfrac{d}{dx} e^x = e^x \text{ for all } x \in \mathbb{R}$$

Continuity: I am not sure how this property is useful in algorithms. Intuitively, you can check from graph provided above that it is continuous.

Smoothness: I am not sure how this property is useful in algorithms. But seems useful. The natural exponential function has the smoothness property.

$$\dfrac{d^n}{d^nx} e^x = e^x \text{ for all } x \in \mathbb{R} \text{ and } n \in \mathbb{N}$$.
Are there any other properties like non-linearily, differentiability, smoothness etc., for the natural exponential function that make it superior to use in AI algorithms?

I think this question is a bit broad because a function may be used in different ways, in different contexts, and for different purposes. I think you're interested in ML (and maybe specifically in neural networks) and not the whole AI. If that's the case, I would start by replacing "AI" with "ML" or be more specific, then I would try to motivate a little more why you're asking this question. For example, in logic-based AI, maybe the characteristics that you mention are not even useful. Where would a function be used in those contexts? These are all issues that I see with this post.

@nbro But I am not sure there are many branches in AI that use $e^x$. And it is true that there are plenty of branches, but, if we view from the perspective of function only, there are finite properties for $e^x$. Am I wrong anywhere?

The question must be split in two:
Use in softmax: softmax is based in the concept of cross entropy, logistic regression, logistic function ... . All these concepts includes exponential or logarithmic in its formulation.
Use in activation functions:

monotonic: a non-monotonic function will give same output to different inputs, disturbing the concept of distance/error.
non-linear: a linear activation function collapses the layer with the next one, thus, it is useless.
differentiable: necessary to use back-propagation.

While all the properties you list are true, the essential parts are the Differentiability and the fact that you build a nice probability distribution from it.
The unnormalized scores the neural network puts out before we apply softmax to them may very well contain negative values. Applying the exponential function is a great way to build a probability distribution from a given vector, disregarding the sign of the values.


text:
Temporal Graph Neural Network for motion prediction
Temporal Graph Neural Networks have been used for motion prediction (or traffic forecasting) in the following recent papers:
Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction
Graph Neural Network for Traffic Forecasting: A Survey
I was wondering, how to use them or any of the available temporal graph neural networks for motion prediction when the graphs don't have the same number of nodes / edges?
Let's say I have recorded a motion of a skeletons that have 5 joints and 10 joints. Then, after the model is trained, I want to use that model for a motion prediction of skeleton with 7 joints. Each node (joint) has 3 attributes, coordinate x, y and z.
Is that even possible, or is there trivial way how to do that? Could something like graph padding (creating nodes without any edges) work in that case?
For instance, in the first paper when they initialize temporal attention or spatial attention layer, it depends on the number of nodes. It creates a learnable weight matrix with dimension N, therefore if we change the input function (i.e. inference) it fails due to impossible matrix multiplication.

Can you please put your specific question in the title? "Temporal Graph Neural Network for motion prediction" is not a question.


text:
How is the error calculated with multiple output neurons in the neural network?
Machine Learning books generally explains that the error calculated for a given sample $i$ is:
$e_i = y_i - \hat{y_i}$
Where $\hat{y}$ is the target output and $y$ is the actual output given by the network. So, a loss function $L$ is calculated:
$L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$
The above scenario is explained for a binary classification/regression problem. Now, let's assume a MLP network with $m$ neurons in the output layer for a multiclass classification problem (generally one neuron per class).
What changes in the equations above? Since we now have multiple outputs, both $e_i$ and $y_i$ should be a vector?

As you say, the outputs are modeled as a vector, each output in one vector component.
In regression problems:
The most common loss function, like in the scalar case, is the square error. Skipping constants, it is defined as:
$$E=\sum_i ||\mathbf{y_i}-\mathbf{\hat{y_i}}||^2 = \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})(\mathbf{y_i}-\mathbf{\hat{y_i}})$$
where:

$\mathbf{y_i}$ vector is expected value for sample $i$ (note I do not use the same naming convention than the question).
$\mathbf{\hat{y_i}}$ vector is network output for same sample
||.|| is vector norm
product of vectors is scalar/inner product.

The derivative respect some NN parameter $w$ is:
$$\frac{\partial}{\partial w}E=\frac{\partial}{\partial w} \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})(\mathbf{y_i}-\mathbf{\hat{y_i}}) = -2 \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})\frac{\partial \mathbf{\hat{y_i}}}{\partial w} $$
being $\frac{\partial \mathbf{\hat{y_i}}}{\partial w}$ the term that backtracking algorithm evaluates.
Multi-class classification problems:
Two options appears as most usual ones:

a) optimize the square error of probabilities. Target vector will be of the form (0,...0,1,0,...0) while network output will be something as (0.2,0.1,0.8,0.4,...). This case can be solved like regression ones.
b) optimize entropy. In this case, an usual loss function is cross-entropy:

$$ E = - \sum_c p(c) log(\hat p(c)) \text{  [definition]} $$
$$ E = - \sum_i \sum_c p_i(c) log(\hat p_i(c)) = - \sum_i log(\hat p_i(c_i)) \text{  [average]} $$
where:

$c$ is some class
$p(c)$ is probability of class $c$ in train dataset
$\hat p(c)$ is probability of class $c$ in network output
$i$ is sample number
$c_i$ is expected (correct) class of sample $i$
$p_i(c)$ is ground truth, usually taken as 1 if $c=c_i$, 0 otherwise as in latest expression.
$\hat p_i(c)$ is net output for sample $i$ and class $c$.

Assuming you're using softmax on the last layer for classification, it sounds like a simple application of cross entropy loss from here on out: https://datascience.stackexchange.com/questions/20296/cross-entropy-loss-explanation
Edit:

Rather than linking to another post (which can be lost, e.g. if the author of that post deletes it), please, explain (preferably, in your own words, i.e. without just copying and pasting) the concepts (in this case, the cross-entropy loss function).

authors: Виталий Князев


text:
Bias-variance tradeoff and learning curves for non-deep learning models
I am following a course on machine learning and am confused about the bias-variance trade-off relationship
to learning curves in classification.
I am seeing some conflicting information online on this.
The scikit-learn learning curve looks like the top 2 curves here:
(source: scikit-learn.org)
What I don't understand is: how do we read bias from this? If we look at this image
where each blue dot is a model. I think the bias would be the green curve being high. But high bias
indicates underfitting, right? So shouldn't the red curve be high then too?

High variance would be the gap between green and red, is this correct?
My question is how do the red and green curves relate to underfitting and overfitting,
and how do learning curves fit with the figure with the concentric circles? Is bias purely related to the red curve, or is a model with a low validation score and high train score also a high bias model?

Where did you take these pictures from? Can you also clarify what the score and fit_times (on the y-axis) are? I assume that the "cross-validation score" is the model evaluated on separate data. From the plots, it seems that you evaluate and train the data on the same number of samples. Overfitting typically happens when the performance of the model increases on the training data, but the performance decreases on the validation data. Under-fitting occurs when the model does not even perform well on the training data.

See https://en.wikipedia.org/wiki/Overfitting and https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff. If I interpret correctly your first two plots on the top, the right one should correspond to no apparent over-fitting and underfitting. The left one could indicate that there's some underfitting, because the performance decreases on the training data, even though it increases on the validation data, but you want your model also to perform well on the training data. I don't know how to interpret the other plots because I don't know what the x and y axes are supposed to represent.

Moreover, please, ask only one question per post. Here, you're asking many questions, and it's difficult to address all of them in detail. If you have many questions, ask one for each post. Given that I am not able to properly answer all your question, this really means that your post requires more details and that you should actually split it into multiple posts, one for each distinct question.


text:
Validation accuracy higher than training accurarcy
I implemented the unet in TensorFlow for the segmentation of MRI images of the thigh. I noticed I always get a higher validation accuracy  by a small gap, independently of the initial split. One example:

So I researched when this could be possible:

When we have an "easy" validation set. I trained it for different initial splitting, all of them showed a higher validation accuracy.
Regularization and augmentation may reduce the training accuracy. I removed the augmentation and  dropout regularization and still observed the same gap, the only difference was that it took more epochs to reach convergence.
The last thing I found was that in Keras the training accuracy and loss are averaged over each iteration of the corresponding epoch, while the validation accuracy and loss is calculated from the model at the end of the epoch, which might make the the training loss higher and accuracy lower.

So I thought that if I train and validate on the same set, then I should get the same curve but shifted by one epoch. So I trained only on 2 batches and validated on the same 2 batches (without dropout or augmentation). I still think there is something else happening because they don't look quite the same and at least at the end when the weights are not changing anymore, the training and validation accuracy should be the same (but still the validation accuracy is higher by a small gap). This is the plot:

Is there anything else that can be increasing the loss values, this is the model I am using:
def unet_no_dropout(pretrained_weights=None, input_size=(512, 512, 1)):
inputs = tf.keras.layers.Input(input_size)
conv1 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(inputs)
conv1 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv1)
pool1 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv1)
conv2 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool1)
conv2 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv2)
pool2 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv2)
conv3 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool2)
conv3 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv3)
pool3 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv3)
conv4 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool3)
conv4 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv4)
#drop4 = tf.keras.layers.Dropout(0.5)(conv4)
pool4 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv4)

conv5 = tf.keras.layers.Conv2D(1024, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool4)
conv5 = tf.keras.layers.Conv2D(1024, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv5)
#drop5 = tf.keras.layers.Dropout(0.5)(conv5)

up6 = tf.keras.layers.Conv2D(512, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
    tf.keras.layers.UpSampling2D(size=(2, 2))(conv5))
merge6 = tf.keras.layers.concatenate([conv4, up6], axis=3)
#merge6 = tf.keras.layers.concatenate([conv4, up6], axis=3)
conv6 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge6)
conv6 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv6)

up7 = tf.keras.layers.Conv2D(256, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
    tf.keras.layers.UpSampling2D(size=(2, 2))(conv6))
merge7 = tf.keras.layers.concatenate([conv3, up7], axis=3)
conv7 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge7)
conv7 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv7)

up8 = tf.keras.layers.Conv2D(128, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
    tf.keras.layers.UpSampling2D(size=(2, 2))(conv7))
merge8 = tf.keras.layers.concatenate([conv2, up8], axis=3)
conv8 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge8)
conv8 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv8)

up9 = tf.keras.layers.Conv2D(64, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
    tf.keras.layers.UpSampling2D(size=(2, 2))(conv8))
merge9 = tf.keras.layers.concatenate([conv1, up9], axis=3)
conv9 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge9)
conv9 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv9)
conv9 = tf.keras.layers.Conv2D(2, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv9)
conv10 = tf.keras.layers.Conv2D(1, 1, activation='sigmoid')(conv9)

model = tf.keras.Model(inputs=inputs, outputs=conv10)

model.compile(optimizer = Adam(lr = 2e-4), loss = 'binary_crossentropy', metrics = [tf.keras.metrics.Accuracy()])
#model.compile(optimizer=tf.keras.optimizers.Adam(2e-4), loss=combo_loss(alpha=0.2, beta=0.4), metrics=[dice_accuracy])
#model.compile(optimizer=RMSprop(lr=0.00001), loss=combo_loss, metrics=[dice_accuracy])

if (pretrained_weights):
    model.load_weights(pretrained_weights)

return model

and this is how I save the model:
model_checkpoint = tf.keras.callbacks.ModelCheckpoint('unet_ThighOuterSurfaceval.hdf5',monitor='val_loss', verbose=1, save_best_only=True)
model_checkpoint2 = tf.keras.callbacks.ModelCheckpoint('unet_ThighOuterSurface.hdf5', monitor='loss', verbose=1, save_best_only=True)

model = unet_no_dropout()
history = model.fit(genaug, validation_data=genval, validation_steps=len(genval), steps_per_epoch=len(genaug), epochs=80, callbacks=[model_checkpoint, model_checkpoint2])

I met the similar problem and I think the reason is the number of validation data is too small. The small number of data result in contingency——That means the data chosen as the validation set happens to perform well.

I am answering my own question here. The only additional thing that I found was that the average accuracy across a batch of data was slightly higher if the amount of samples in the batch was lower. So the validation set was only 15% of the data, therefore the average accuracy was slightly lower than for 70% of the data. I don't know why the more samples you take the lower the average accuracy, and whether this was a bug in the accuracy calculation or it is the expected behavior. Either way, if you have the same problem, one suggestion is plotting average accuracy vs number of samples and see if this is the reason why you get a lower training accuracy.

It's probably that smaller sample sizes are more prone to bias. It could just be the train/val split split the validation data into a bias set. Maybe changing the split could give more expected results?


text:
What is accelerated years in describing the amount of the training time?
As described in this article, it was written that GPT-3 took 405 V100 years to train in 2020.
I'm a bit confused about this definition, does that mean the process was accelerated like using a V100 GPU to train in 405 years?

The statement in which you mentioned that "GPT-3 took 405 V100 years to train" refers to the computational resources utilized in training the GPT-3 model. Specifically measured in terms of the equivalent time it would take if the GPT model is trained on a single Nvidia Tesla V100 GPU. To make it more understandable it does not mean that the training process took 405 years, but it indicates the computational intensity or amount of time it would have taken if it was trained on a single Nvidia Tesla V100 GPU.
Since GPT-3 is a very large model with 175 billion parameters, it requires very high resources in training. According to this article it takes around 1,024 Nvidia V100 GPUs to train the model, and it costs around $4.6M and 34 days to train the GPT-3 model.