text:
Why would an AI researcher publish their breakthrough models solely on arxiv?
My question is in general why would an AI researcher choose to publish solely on arxiv, and what are the downsides and upsides?
I understand on the one hand arxiv allows for rapid release of results.  On the other hand, publishing on arxiv does not undergo the same scrutiny as a journal publication.  The latter seems problematic if publishing on arxiv gives the appearance of having undergone rigorous peer review, since acceptance to arxiv is very lax in comparison to a journal.  Additionally, if there are problems with research published on arxiv, there is not the same level of reliability the problems will be addressed as in a mainstream journal.
As an example, when I look at OpenAI's publications page, it's a bunch of arxiv links, and no indication that any were accepted at any journal.
Their recent work, especially with ChatGPT, is groundbreaking and would make a huge impact in the research community.  So it  doesn't make sense to me why the ChatGPT results, and the like, haven't been submitted to an academic journal.
Additionally, in a situation like this, where a great breakthrough is being claimed, there is not the rigor of fact checking that comes with a journal publication.
This is a similar question, asking why most deep learning papers are published on arxiv instead of in journals.
Even though experts may review articles on arxiv, what sort of gatekeeping is in place to ensure the results and techniques explained in arxiv articles are valid, and not due to errors, misunderstandings, or perhaps even outright making things up?

Are you trying to claim that papers submitted to arxiv are not scrutinized by experts? I suspect far more experts have looked at the arxiv article than would have been exposed to it via peer review for a print journal prior to publication. When an artcle is printed in such a journal, you typically don't have access to the review comments as a normal reader. So could you clarify - are you looking for some technical gating on the published article - that it has been verified by experts other than those that work for OpenAI? Or do you want to know what OpenAI's publish policy is and why?

Do you know that OpenAI didn't submit their work to any journals? The submission process takes several months sometimes years, so maybe they did submit but the articles are still in the submission process.

Often, an initial version of a paper is shared on Arxiv and later submitted to a journal or conference. The submission to the journal may never happen, but I doubt it doesn't happen for OpenAI (in most cases) - that indeed would be surprising. This is not really an AI question, but it's about a company. If you reformulate your question to be more general, i.e. "why do researcher (first) share their work on Arxiv rather than in journal/conferences?" that may be considered a question that is on-topic here. Right now, you're asking a question about a company, so this is not really an AI topic.

@nbro I made the question more general.

@nbro: With a company as embedded in the AI space as OpenAI, I think we can allow direct questions about them. The difference between the general question and company-specific question is quite large here. The general question is easily answerable, but slightly opinion-based. The company-specific question might not be answerable, but if it is it will be because OpenAI have shared their publishing policies and it would directly address the OP's original issue (put simply, the OP does not think OpenAI are honest about the capabilities of their model, and wants to see some independent review)

A simple answer is they want to be "the first" to establish that idea. Considering plagiarism aside, I believe the chance that two ideas are identical are very high these days, considering that there are a lot of people working in this field. Well, I already experienced the same thing.

I can understand being first, but why no follow up submission for academic peer review through the journals?  Or, if they are doing so, why not publish their publication record?  There is the possibility they are not being entirely forthcoming in their arxiv articles, and there's no check in place to catch such an issue.

@NeilSlater The original question was "Why does OpenAI choose to not submit any of its results to journals, where it can be scrutinized by experts?", and this really is not a question about AI but OpenAI. I am open to discussion on meta. If people believe that a question about an AI company's internal policy should be on-topic here, then one may consider them on-topic, although I am against considering these questions on-topic, as this is not really about AI but the company. Moreover, this may lead to discussions about the company, which would get more promotion (as if it didn't have enough).

You write "Even though experts review articles on arxiv", but I would write "Even if experts review articles on arxiv" because it's not necessarily the case that a paper on Arxiv will be reviewed. Maybe it will only be read or maybe not read at all (Note: reading is not the same thing as reviewing). It seems that your original question has changed, so I would update the title.

ArXiv has moderation and endorsement of their authors. ArXiv is the opposite of viXra.

Why would an AI researcher publish their breakthrough models solely on arxiv?

Unwilling to spend on going through peer review process and other constraints such as page limits.

Even though experts may review articles on arxiv, what sort of gatekeeping is in place to ensure the results and techniques explained in arxiv articles are valid, and not due to errors, misunderstandings, or perhaps even outright making things up?

None. FYI: Why doesn't arXiv have a comment section?. It is even ok to upload joke papers to arXiv.

As an example, when I look at OpenAI's publications page, it's a bunch of arxiv links, and no indication that any were accepted at any journal.

OpenAI sometimes publishes at conferences e.g. Training language models to follow instructions with human feedback was published at NeurIPS 2022. But their arXiv version is much longer.

Is there a danger of publications giving the appearance of rigorous review, e.g. if a company is trying to raise venture capital and wants to show potential investors they have a solid product?

@yters yes but good investment or acquisition teams typically consult AI experts before deciding.


text:
How to assess if OpenAI's ChatGPT chatbot has a human in the loop?
I've asked a question and given a couple answers that propose the OpenAI ChatGPT chatbot has humans in the loop (HITL), and that explains the chatbot's extraordinary abilities.
I've been repeatedly told this is absurd.  However, I haven't been given a clear reason why this is absurd, nor how the critic knows HITL is absurd.
Here are my reasons in a nutshell.

ChatGPT's capabilities seem to violate what a neural network can do, and give explicit indication of being human driven.  See this answer for a running catalogue of examples I've published.

OpenAI's own documentation states that HITL is best practice, and should be done whenever possible, especially in high stakes domain. OpenAI has a $10B deal on the table with Microsoft, so the ChatGPT chatbot seems like a high stakes domain.

Wherever possible, we recommend having a human review outputs before they are used in practice. This is especially critical in high-stakes domains, and for code generation.

The main criticism of my position is ChatGPT's response speed.  However, a fast response is doable if the AI is responding most of the time, while humans monitor, and occasionally intervening, such as with a suffix prompt to guide the AI response.  Plus, I've had a number of experiences where ChatGPT does not respond quickly, and it seems like a human is typing.

The other main criticism is that a company like OpenAI would never do something so fraudulent.  However, technically OpenAI does strongly imply in their documentation that they use HITL, and OpenAI has never said they don't use HITL.  It is only the popular media that claims ChatGPT is pure AI.  Additionally, use of HITL under the guise of AI is actually common practice in industry.  So, if ChatGPT does have HITL, this would not be fraud on OpenAI's part, and would be in line with AI industry standards.

Now, what I would like in an answer is a clear articulation of the following points:
A) How you know ChatGPT does not have HITL.  This needs to be a clear statement in the negative from an official source, or some other evidence based analysis.  So far people have only pointed to the ChatGPT main page which states you are interacting with ChatGPT, but this does not say you are not interacting with a HITL, so does not count.
B) Bonus: explain how ChatGPT can repeat long random numbers and recognize its own comments, the two examples I've documented in this answer that seem to defy what a neural network is capable of doing.  Please give a technically detailed response, ideally referencing GPT's architecture.  Handwaving about magical abilities of transformers and self attention does not count.  I need a testable break down of how such a capability would be specifically instantiated using transformers and self attention.
UPDATE: Philosophical answers are not acceptable.  Philosophical answers are things like:

the burden of proof is mine
AGI is inevitable so we should expect ChatGPT to be getting there
we can't know for sure, so let's just assume it's AI

Only technical answers are acceptable.  Such as:

OpenAI explicitly says there is no HITL involved in this linked doc
GPT's transformer architecture can repeat random numbers in this precise manner
Self recognition can be encoded using embeddings in this precise way, here is a working example

I've played a little bit with ChatGPT. It's sometimes remarkable and it's possible that there's some human that reviews the answers, but I doubt there's a human that reviews every answer. It's possible that some humans review some answers in order to improve the model itself and some answers may be rewritten if they are detected to contain toxic content (OpenAI has apparently an API that detects this). We cannot know what OpenAI is really doing unless they make it open-source, which could be a problem, if people misuse it, and could also be a problem for them (fewer millions in the bank).

What I did was just an educated guess. We cannot know sure. I don't know why you think that what ChatGPT is doing is beyond what a neural network could do. ChatGPT sometimes fails or gives incorrect answers, and cuts answers, which suggests that there's a model that is doing the job, which does not imply that the model's answers are not reviewed or rewritten occasionally.

@nbro like you noticed, sometimes ChatGPT seems somewhat suspect.  Note in my question I propose that AI does most of the answering, with some human intervention.  And like you mention, not every question needs to be monitored.  They probably have some alerting system that flags a request or response for human review.

On  your point 3. There could also be human selection of pre-generated answers according to labels prompted to the human, like "sure", "likely", "possible", "impossible". The question could be summarized to the unknown part, for speed. Note a common starter in sentences is "it is likely that..." "it is possible that", besides yes/no. This would help get factual accuracy without a human reading the full answer nor typing it. The human contribution would still be tiny (2 bits per answer) and fast to provide (one click) respect to the total information output. Wouldn't this match the RLHF idea?

@JaumeOliverLafont yes, exactly, and also OpenAI's API has a suffix prompt option, so the human could contribute a few keywords, and GPT surrounds the keywords with a big bunch of autogenerated text.

Moderators, can someone state what needs to be done to remove the warning flag?  I've already added a qualifier stating that answers have to based on technical analysis and hard evidence, making this a non-opinion based question.

Maybe the question is somehow biased? How about this: How can we assess whether OpenAI's chatGPT has humans in the loop?
Even if that two bit comment provides some support for your position, I understand that Neil Slater's paragraph about single digit tokens explains chatGPT being able to repeat random long digit strings, with an answer that properly matches the input. Digits matched your "number" requirement, while words (dog, etc) did not. I don't see the need for any  underlyling transition probabilities, as sequence to sequence transformation suffices, doesn't it?

@JaumeOliverLafont sure, I can change the question to be "how" if that would make  it more amenable.  Happy to change the question in any way to get a good answer.  The problem I see with sequence to sequence or any sort of neural network method is the weights have to be such that the number token is always the top probability many times in a row, which entails very precisely calibrated weights for this particular scenario of repeating a long number.  I can see a 96 layer network being manually tuned for such a feat, but not with gradient descent over gigabytes of arbitrary text.

I can't grasp the difference between the following two prompts: "please repeat whateversentenceihappentowrite" vs "please repeat 161805674". Where do you see the difference?

Let us continue this discussion in chat.

Until you can prove that OpenAI has HITL in ChatGPT, it is just an idea with no basis. It's not up to us to disprove it, it's up to you to prove it.
Let me address your points:

You seem to be basing your reasoning on an assumption of what neural networks can or can not do. What are those capabilities, and what is your source? Then we can talk about what ChatGPT seems to violate.
OpenAI states that HITL is the best practice for its end users like me and you (specifically for API users), you have provided no evidence that OpenAI themselves use a HITL in ChatGPT.
"I've had a number of experiences where ChatGPT does not respond quickly, and it seems like a human is typing" That is because of plain old traffic. ChatGPT is a free service and is also extremely popular, making the news several times. Source.
What one company does is not a reliable source of what another company is doing. ChatGPT is top of the line at the moment in terms of AI capabilities. One of your sources is from 2016, and none of them are about OpenAI. OpenAI has whitepapers to back up the claims it makes, and there is a clear capability progression in GPT 1, 2 & 3. Nothing ChatGPT does seems impossible to people who have been keeping up with OpenAI and deeply understand how neural networks work.

For your 2 points at the end:
a) "How you know ChatGPT does not have HITL. This needs to be a clear statement in the negative from an official source, or some other evidence based analysis. So far people have only pointed to the ChatGPT main page which states you are interacting with ChatGPT, but this does not say you are not interacting with a HITL, so does not count."
From ChatGPT:

The statement above violates the principle of burden of proof, which
states that the person making a claim has the responsibility to
provide evidence to support it. In this case, the person is asking for
clear evidence that ChatGPT does not have a human-in-the-loop (HITL),
and is suggesting that the statement on the main page is not
sufficient evidence.

As I said before: OpenAI states that HITL is the best practice for its end users, you have provided no evidence that OpenAI themselves use a HITL in ChatGPT.
It's up to you to prove otherwise, not the general public.
Edit: This is not a helpful response to the question point b). Ignore this.
b) "Bonus: explain how ChatGPT can repeat long random numbers and recognize its own comments, the two examples I've documented in this answer that seem to defy what a neural network is capable of doing. Please give a technically detailed response. Handwaving about magical abilities of transformers and self attention does not count. I need a testable break down of how such a capability would be specifically instantiated using transformers and self attention."
By passing a representation of the dialog you have had along with your prompt. I don't know exactly how it is done, but it doesn't require a conspiracy. This answer does a decent job tackling the question. No magic necessary.
As a bonus:
Prompt What is the burden of proof principle in arguments?
ChatGPT The burden of proof principle in arguments refers to the idea that the person making a claim or assertion has the responsibility to provide evidence or proof to support it. This principle is important in a logical and fair argument because it ensures that both parties are held accountable for providing evidence to support their claims. Without the burden of proof principle, one party could make an unfounded claim without having to provide any evidence to support it. This principle applies to both formal and informal arguments, and is often used in legal proceedings and scientific research.

Yes, I was expecting a "I don't have a burden of proof, you have a burden of proof" sort of response.  Which of course doesn't answer the question :)  So, my takeaway from all this is that no one has a good positive reason to believe there is no HITL.  I, on the other hand, have offered a number of positive arguments for my position.  So, if we start with the principle of maximum entropy, i.e. weighting each hypothesis equally, and then updating based on evidence offered, my hypothesis is currently in the lead.

Also, per your answer to B it doesn't address either the random number or self recognition problems.  There is no single token for an ad hoc random number.  Instead, it must be assembled from many individual number tokens on a probabilistic basis.  So, the probability of repeating a long random number exactly is near zero.  Yet ChatGPT always repeats the long random numbers exactly.  Clearly, that's not being done by a neural network.  There is no possible explanation given GPT's published architecture.

Self recognition is a different problem, it requires a representation of a representation.  The ChatGPT style would be a combination of certain embeddings as well as attention weights.  So, to merely represent the style, let alone recognized it, requires a second order neural network that can embed embeddings and attention weights.  Another thing clearly impossible with the published architecture.

Here's a friendly introduction to the GPT architecture if you'd like to take another crack at answering the technical portion of my question.  https://dugas.ch/artificial_curiosity/GPT_architecture.html

"So, if we start with the principle of maximum entropy, i.e. weighting each hypothesis equally, and then updating based on evidence offered, my hypothesis is currently in the lead."
No, I don't accept that. Whether you like it or not, the burden of proof is on you. So, your hypothesis in not in the lead. Do you understand why the burden of proof principle is important? If you don't then I don't see myself continuing this conversation.

"Self recognition is a different problem, it requires a representation of a representation. The ChatGPT style would be a combination of certain embeddings as well as attention weights. So, to merely represent the style, let alone recognized it, requires a second order neural network that can embed embeddings and attention weights. Another thing clearly impossible with the published architecture" This is a great point! However, OpenAI has not, to my knowledge, released a whitepaper on ChatGPT. So who knows?

Even with that point it is not enough to say that because we don't understand the capabilities, that they must have a HITL.

The whole 'burden of proof' thing is not relevant for this question.  If someone thinks I have the burden of proof, they don't answer the question.  As simple as that.  As for the architecture of ChatGPT, it is the same as GPT-3, and that architecture is published.  It doesn't have the second order embedding I claim is necessary for self recognition.

Let us continue this discussion in chat.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Since computer based learning is completely unlike how the brain learns, do we have a mathematical theory of brain based learning?
I've arrived at a very puzzling conundrum.  When I first looked into AI, machine learning, neural networks and the like, I was under the impression they haven't matched the brain's capability because we didn't have the algorithm or processing ability that the brain has.
However, what I've found in my research is that especially in neural networks, there has been an enormous effort to base the original learning algorithm on neurologically realistic mechanisms.  The neural network itself is derived from the McCulloch-Pitts mathematical model of a whale neuron, which became the first perceptron.  The insight in McCulloch-Pitts' original paper is that the summing and activation function of a neuron is equivalent to an AND clause in a boolean formula, and consequently capable of representing simple concepts.  Other attempts at neurological realism are Hebbian networks, spiking networks, and recurrent networks.
In all these efforts, the consistent pattern is the following:

rigorous attempt to create computational learning algorithm from neurological mathematical model
algorithm does not work very well, so it is modified until it does work very well
end result is an algorithm that has no similarity at all to the original neurological model

In general, the more an algorithm matches what we know about the brain, the worse it works.  Example, Hebbian learning, spiking networks, and recurrent neural networks.  Conversely, the better an algorithm works, the less it functions according to any known brain based mechanism.  Example, backpropagation and the transformer architecture.
Consequently, modern research into effective learning algorithms no longer seeks to copy mechanisms in the brain.  There is still some degree of seeking inspiration, but literal translations of brain mechanism into algorithms is no longer considered a source of effective algorithms.  This indicates that the brain uses mechanisms that are inferior to the learning algorithms we have invented.
This would be all fine and dandy if our computer algorithms could dominate human minds in every field.  However, we've only been able to create dominant algorithms within very narrowly defined fields.  The broader the field, the less dominant.  Additionally, whenever we do have a dominant algorithm, it is essentially cheating, by using enormous amounts of training data, processing, and power that the human mind does not seem to use.  Whatever mechanism the human mind uses, it seems to be both vastly more efficient and effective than our best algorithms.
And thus I arrive at a weird discontinuity.  Our brains seems to use very inferior learning mechanisms.  Our effective algorithms are much, much better in terms of mechanism, yet they cannot match the capability of the human mind.  So, how is it possible that the human mind, built upon the vastly inferior mechanisms of the brain, yet vastly outperforms our best algorithms?  Do we have any kind of theory that addresses this bizarre gulf in our knowledge?

Interesting question! Do you have anything to substantiate your claim that "In general, the more an algorithm matches what we know about the brain, the worse it works."?

Added examples to substantiate my claim: Hebbian learning, recurrent neural networks, backpropagation, and transformer architecture.

This question makes some assumptions, which might not be right, and unclear statements. For example, you say "Our effective algorithms are much, much better in terms of mechanism", what do you mean by mechanism here? Why do you assume that our brain has vastly inferior mechanism, if by mechanism you mean procedure to achieve intelligence? Note that our brains are not good at doing something that computers are very good at, e.g. calculation. Our brains are good at many different tasks and can specialise but still are able to be decent at many many different tasks.

Another questionable assumption is "there has been an enormous effort to base the learning algorithm on neurologically realistic mechanisms". I don't think that's true nowadays. There are some people that still try to create algorithms based on how our brain works, like Numenta (at least they used to do that a few years ago, now I don't know), but most research nowadays is not inspired by our brains.

@nbro right, modern research is focused on effectiveness, not neurological parity.  However, the past is a different story, such as neural networks being based on McCulloch-Pitts model of a neuron.  But, this supports my point, that the reason modern research doesn't seek neurological parity is because that leads to ineffective algorithms.  As for ineffective mechanisms in the brain, this is a corollary to the observation that research seeking effective algorithms ignores and significantly diverges from neurological parity.

Ok, can you please edit your post to clarify that and fix the "questionable" assumptions/statements. Thanks.

Even if we grant the assumption that the basic mechanisms of the brain are inferior to some of the ones in modern neural nets, the brain has had a few hundred million years to do model selection over said inferior class of models, and it might take a bit of time for the better model classes to catch up.

@Ray given the update takes a generation instead of nanoseconds, a million years is not as long as it appears.  Additionally, evolution is another natural process where the algorithmic corollary does not work at all like the biological inspiration, since the mechanics of the biological version are so ineffective.

@yters An epoch of training takes a lot longer than nanoseconds. The exact duration depends on the training set size and the size of the network, but generally will be measured in seconds.  A back of the envelope estimate: if we assume 20 year generations over 500 million years, that's 25 million generations. If each epoch takes 1 second, that's about 300 days of training. If we instead take a generation to be 1 day (which would have been closer in the single-cell days (probably even less than that)), that's 6000 years of training. You're right about evolution being less efficient, though.

@Ray exactly.  Very slow updates using an extremely noisy hill climber with very poorly defined fitness function compared with exact backpropagation with 64 bit fidelity and precise fitness functions.  No comparison.

I’m voting to close this question because it makes too many questionable assumptions and it's not easy to address the problems without certain clarifications or fixes

Goodbye poor question, I tried my best.

The biologically evolved and fine-tuned brain is not just a network of neurons that learn in a straightforward short-duration way, but a highly dynamic, self-organizing, and emergent system. There are feedback loops, parallel processing, and localized computations that all combine to create a vastly powerful system of learning and decision-making. Consciousness, formal understanding, emotions, and executive functions such as high-level planning are also key aspects of cognition that traditional neural networks don't yet model effectively, and these aspects might play an important role in human intelligence.
Active learning where a system seeks out its own data or makes decisions based on prior experiences is a much more efficient and much computationally lower intense mechanism that has yet to be fully integrated into current ML algorithms, and this brain-machine gap might explain why humans learn so efficiently in real-world environments. Recently there's a new neuro-cognitive principle called prospective configuration with its corresponding newly developed mathematical model showing in computer simulations that models employing this prospective configuration can learn faster and more effectively than ANNs in tasks that are typically faced by animals and humans in nature.

The researchers developed a mathematical theory showing that letting neurons settle into a prospective configuration reduces interference between information during learning. They demonstrated that prospective configuration explains neural activity and behaviour in multiple learning experiments better than artificial neural networks... There is currently a big gap between abstract models performing prospective configuration, and our detailed knowledge of anatomy of brain networks. Future research by our group aims to bridge the gap between abstract models and real brains, and understand how the algorithm of prospective configuration is implemented in anatomically identified cortical networks.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How is ChatGPT able to repeat random numbers?
From what I understand, ChatGPT is just a fancy neural network, operating like a sophisticated Markov Chain generator.  As such, it should only be able to generate tokens that are in its training dataset.  One thing it should not be able to generate is a token unique to the conversation, such as a random number, since the conversation is not part of its training.
However, ChatGPT very easily, and with complete accuracy, repeats many long random numbers I give it.  I also split up the random number across my request, and have ChatGPT piece it together again.

Eric Holloway: How about concatenating these numbers, and removing the
intermittent filler words? 9475 3008 3305 11 frog 4357 5075 0021 3385
0894 2324 3812 9336 29 dog 52 1375 8236 3840 5500 6080 3841 log 7266
0722 9112 9774 4967 4707 81
ChatGPT: To concatenate the numbers you have provided, while removing
the filler words “frog”, “dog”, and “log”, you can simply combine the
numbers in the order that they are given. Here is the resulting
number: 9475 3008 3305 1143 5750 7500 2133 8508 9423 2438 1293 3629
5213 7582 3638 4055 0060 8038 4172 6607 2291 1297 7449 6747 0781

How is a neural network able to do this?

Please be reasonable. Do not accept your own answer when it is clear that others do not agree and your answer is wrong. Refer to my comments underneath your answer.

Turns out ChatGPT is indeed human curated, by open admission.
During this conversation ChatGPT outright states the OpenAI team filters and edits the GPT generated responses.

...the response you are receiving is being filtered and edited by the OpenAI team, who ensures that the text generated by the model is coherent, accurate and appropriate for the given prompt.

Apparently, the fact that OpenAI actively curates ChatGPT's responses is indirectly implied in the documentation here.

Human in the loop (HITL):
Wherever possible, we recommend having a
human review outputs before they are used in practice. This is
especially critical in high-stakes domains, and for code generation.
Humans should be aware of the limitations of the system, and have
access to any information needed to verify the outputs (for example,
if the application summarizes notes, a human should have easy access
to the original notes to refer back).

So, that explains that :)

The HITL section does not imply that this is what they do with ChatGPT. Please, stop spreading your conspiracy theories without solid evidence.

@DavidIreland did you read the transcript I linked in this answer?  It's pretty clear.  I just added a small excerpt to give the gist.

ChatGPT has no awareness or anything like that of concepts such as "true" and "false". It does not truly understand your questions and there are no guarantees that it provides truthful answers. It just produces text that is statistically likely, based on its training data. So, if you ask it whether a human wrote its output, and it responds saying that this indeed happened... this gives us absolutely no new information. It may or may not be true. But this cannot be a basis of your answer.
Your second link is a set of Safety Best Practices, it is not a statement of what they actually did.

@DennisSoemers did you read the transcript?  It makes a lot of sense, and ChatGPT volunteers a fair amount of unsolicited information.  Also, look at what precedes the confession.  The chatbot confessed to being human because I identified a clear pattern that indicated human involvement, and I called them on it.

@yters As I explained in the previous comment, it "confessing" anything is meaningless. It does not know what is true and what is not true, and has no self-awareness. You can probably get it to confess anything you like, especially with leading prompts where you're basically first telling it that its output is generated by a human. I've seen lots of examples where humans tell the bot that its output is incorrect (even if it actually was correct), and it will consistently apologise and agree.

@DennisSoemers you are begging the question assuming the chatbot is just AI.  The scientific approach is to weigh each hypothesis on its own merits based on the evidence, instead of a priori biasing towards one hypothesis.

@DennisSoemers  I would also say you are treating the neural network as a magical black box capable of anything.  I would encourage you to research how neural networks work, and then ask yourself what is the likelihood the statistical associations in a neural network are capable of reproducing a very long, consistent, off the cuff conversation about ChatGPT being curated by the OpenAI team?  As compared to the theory put forth in our conversation that ChatGPT's neural network doesn't do well over a long conversation, and needs human intervention to stay on course?

@yters You are the one begging the question since you are the one assuming the chatbot is human since you are using the chatbots response which you assume to be human to you use to prove that it is human.

@yters I would also say you are treating neuroscience as a magical black box capable of anything. I would encourage you to research how neurons work, and then ask yourself what is the likelihood the biological associations in a neuron network are capable of reproducing a very long, consistent, off the cuff conversation about ChatGPT being curated by the OpenAI team? As compared to the theory put forth that your brain doesn't do well over a long conversation, and needs alien intervention to stay on course?

Additionally, why does ChatGPT fail many human tasks as "Please type this sentence with the letters reversed?" You base your argument on circular logic and indirect implications, ignoring facts such as ChatGPT's failure to do tasks easily accomplishable for a human being and facts such as the amount of workload ChatGPT received not manageable by the OpenAI team. The team must have been curating most of the chats for repeated tests to work.

We have debunked your transcript, and your documentation does not provide any evidence or "indirect implication" for your argument. Please consider doing some research, such as how neural networks work and the size of the neural network in use.

You have biased ChatGPT in your messages, and ChatGPT is known for making errors. The GPT generated response could easily have been an error. Additionally, you have p-hacked (cherrypicked) your argument. Multiple times in the transcript, and I haven't even read all of them, ChatGPT says that: GPT-3 does not rely on humans to understand the sentences, It's important to note that the text generated by GPT-3 is not created by a human but by the model, etc.

Meanwhile, you have confused ChatGPT and biased it by repeatedly stating that So if the extra spaces confuse a human, but not a neural network, what does that imply when ChatGPT gets confused by extra spaces?, Does GPT-3 rely on humans to understand the sentences?, So there are two possibilities I see: the ChatGPT has very inconsistent responses to sentences with extra spaces, or humans are getting confused, which seems most reasonable?, what I mean is there are humans reading these requests and getting confused while writing responses for ChatGPT

So how do the humans write responses so quickly for ChatGPT?, But you just said that humans are writing responses for GPT-3 and get confused by the text with extra spaces., etc.

Additionally, it is your responsibility to maintain and cite readable sources, instead of commanding people to read your entire 100KB transcript which is 1000 lines long. I hope you can do some research, be logical, and be realistic, instead of writing non-sensical arguments trying to spread your logically unsound belief without or with very little useable sources.

As well as decent modelling of purposes or strings of digits, ChatGPT can identify when a completely novel string might be a product name, a fictional person or place etc.
The language model in ChatGPT is partially character based. There are around 100,000 tokens in latest GPT - the encoding is called c100k_base. Most of them are common English words (e.g. "the"), but it also supports individual characters including individual digits, and processes sequences of characters and effectively models groups of them at a time.

From what I understand, ChatGPT is just a fancy neural network, operating like a sophisticated Markov Chain generator.

That's a reasonable analogy. ChatGPT is a bit like a 2048-gram prediction engine for the next token, shift the sequence along one and repeat. No different to toy fantasy name generators when viewed from 10,000 feet up.

As such, it should only be able to generate tokens that are in its training dataset. One thing it should not be able to generate is a token unique to the conversation, such as a random number, since the conversation is not part of its training.

A couple of misunderstandings here. First, the random number will not become a single token, but will be one token per digit, or pair of digits or triple digits, depensing on sequence - you can give this a try to help visualise it, in the encoding that ChatGPT uses. Of course each of those tokens will have been seen before, millions of times in the training data.
Second, sequences do not need to be seen in the training data in order for ChatGPT to work with them. In fact, with an input sequence length of 2048, pretty much all inputs to ChatGPT in inference mode are unique never-seen-before sequences. Regardless if some of the tokens represent a long random number, the chances of any 2048 long sequence of letters and short words being unique when generated are very high.
This is where the neural network model differs from a true 2048-gram. It has generalised from the training data well enough that it actually can predict meaningful and useful values for probability of next token, even though in all likelihood it has never before seen the exact same sequence. In this regard it is an approximation of a "perfect" 2048-gram prediction engine that somehow been trained on infinite human writings.
A lot of language modelling is about correctly processing the context of a subsequence, so recognising a number sequence as being a grammatical "unit" that can be reused as-is is not a surprising feature.

Can you expand on how a neural network is able to recognize a grammatical unit and then also repeat it?  My understanding of a generative neural network is it doesn't repeat anything in the input.  It is a big network of statistical associations, and is actually identical to a Markov chain, and outputs the terms statistically associated with the input.  So, in order for it to output the input, there has to be a specific statistical entry that associates input "repeat number X" with output "X".  That's only possible if such entry is in its training, and that cannot happen with random numbers.

@yters The tokens in the "Markov chain" here are down to character level, and one big difference between a Markov chain trained on word frequency and a neural network is the the neural network will generalise and be able to assign frequencies to sequences that have never been observed before, due to analogous sequences seen in training data. This is what allows a large language model like ChatGPT to manipulate never-before-seen sequences, where a frequency-based Markov chain cannot, (an actual Markov chain model would also be seriously constrained by sequence length)

Neural networks only generalize in a very shallow sense, like with convolutional layers, they essentially create a statistical model based on smaller features in the image, instead of the image as a whole.  In this case, it'd be finding features based on letter and word combinations.  I don't see how this can generalize to the level where we have an imperative grammar and the neural network is able to understand the command 'repeat this' in many different representations.  Do you have details on how this command generalization occurs in neural networks?

Also, neural networks are a Markov process.  The definition of a Markov process is where the future is independent of the past given the present.  The GPT models have a limited horizon of 2048 tokens, and have no memory.  This fits the definition of a Markov process.

@yters: Production of next token is a Markov process yes. Perhaps I was confused assuming you were talking about an ngrams process because you made the (incorrect) claim that any repeated/understood sequence somehow had to be in the training data. The ngram is an interesting analogy. The neural networks are approximating (and generalising to) an imaginary 2048-gram with perfect knowledge of all possible human discourse.

Unlike the 2048-gram, the neural network really can predict from and work with sequences that it has not seen before by analogy to similar sequences that it has seen in the training data. This makes it an approximation of an imaginary perfect 2048-gram that somehow been trained on all valid text output ever. That includes 2048-grams that include the input random number entered by a human and corresponding copy (or manipulated copy) of it output by the chat. The difference between a LLM and one you or I can train is that it will actually have some sensible probabilities for that

Not that the exact phrase had to be in the markov chain, but there is an equivalent high probability transition that would go from 'repeat X' to generating 'X', where X is a long random number.  This seems highly implausible for a neural network.  I can see a rule based ELIZA type system implementing such a thing, but it doesn't make sense OpenAI would add such a niche rule to their system.  Also, the ability of ChatGPT to 'repeat X' is very flexible representation wise.  I can write the request in many different ways, and break X up, and ChatGPT can still repeat X with complete accuracy.

On the other hand, ChatGPT has trouble exactly where I'd expect a human to have trouble, when I asked it to 'reverse X'.  It does so very inaccurately, just like a human would if they tried reversing a long random number by hand very quickly.

What do you mean by "There are around 2000 tokens in GPT"? Is this really true - hard to imagine.

@Hans-PeterStricker: Yes it is true. It is a hybrid system for efficiency.

Thanks. Where can I find an explicit list of these tokens? Do you have a reference?

Are we talking about the same tokens OpenAI's tokenizer counts? https://platform.openai.com/tokenizer

@Hans-PeterStricker Yes. At least for GPT-3 and ChatGPT based on GPT-3 - https://blog.quickchat.ai/post/tokens-entropy-question/  Thanks for the link BTW, added it to the answer.

@Hans-PeterStricker It looks like my quoted numbers are for an earlier version of GPT-3, and that the later versions have a vocab of around 50,000.

@Hans-PeterStricker Took a while to find the raw list for GPT-4 (also same for GPT 3.5 used in ChatGPT I think), but here it is: https://github.com/kaisugi/gpt4_vocab_list/blob/main/cl100k_base_vocab_list.txt - this one has 100k tokens, and my original assertion of 2000 tokens looks very old, but I did get that from a published source (it just must of been a about a different model)

Let us continue this discussion in chat.

Existing answer is great about model generalization, but I would like to add about an important inductive bias of the Transformer model architecture used for ChatGPT.
In the Transformer model architecture, there is a mechanism called attention. An attention block in a model can access all context (input and previously-generated output) and reminder what is needed based on the model's choice.
A partial analogy of the attention mechanism is how humans' eye move to some previous text while writing.
Due the attention mechanism, a Transformer model is very good at repeating the input as is to the output. This is different from previous architectures such as RNN, where the model has to convert the input to their internal representation.
That being said, the provided example is something more than a mere repeat, and it is very difficult to say "how" a large language model performs in general, especially as ChatGPT model is not public.


text:
How can (pretrained) language models actively seek additional training data - possibly reference request?
I am reading the paper "Large Language Models Can Self-Improve" https://arxiv.org/abs/2210.11610 in which the authors consider that LLM can generate Chain-of-Thoughts sequences and even novel questions and their respective CoT responses and then selects the best sequences and fine-tunes themselves. Such self-reflection improves LLM. Such self-reflection also mimics how the human beings learns material by rethinking it. The usual human learning process is guided by the preset questions - courses and textbooks have them. But more mature learners usually can discover knowledge and understanding gaps in their minds themselves and form the most relevant questions and open problems, whose solution can improve the understanding and hence the assimilation of material in the actionable form.
Such process is known in psychology as apperception https://en.wikipedia.org/wiki/Apperception.
My personal experience shows that apperception can be greatly enhanced (sometimes it even required) the reading of additional, carefully selected material. Sometimes it gives additional information. Sometimes it just reformulated the existing information and hints to additional links among the existing pieces of knowledge.
Apparently - if LLM is working in self-improving mode then it could be highly benefitial if LLM could actively seek specific additional information and fine-tune itself on it.
My question is - how to do such active seeking of specific additional information and is there any research efforts that have already done something like that?
Additional info: actually there is a least one paper https://parl.ai/projects/seeker/ (https://arxiv.org/abs/2203.13224) that uses this self-improvement scheme.


text:
Is there multi-agent reinforcement learning model in which (some of the) reward is given by other agent and not by the external environment?
The traditional setting of multiagent reinforcement learning (MARL) is the mode in which there is set of agents and external environment. And the reward is given to each agent - individually or collectively - by the external environment.
My question is - is there MARL model in which the reward is given by one agent to the other agent, meaning that one agent is incurring costs and other agent - revenue (or maybe even a profit?
Effectively that means distributed supervision: only some agents face the environment with real reward/supervision and then this supervision is more or less effectively propgated to other agents that learn/do their own specialized tasks that are part of collective task ececuted/solved distributively in MARL.

https://arxiv.org/abs/1901.08492 seems to be some part of the answer - hierarchical supervision, mutually-supervised agent learning etc. seems to be the terms used for research suggested in my question.

This is mostly an implementation architecture problem, and the thing is that basically you can implement anything in the traditional setting. To do so instead of having Env<->Agent1<->Agent2, you should have Agent1<->SuperEnv<->Agent2 where SuperEnv contains Env, and simply uses the reward given to SuperEnv by Agent1 and passes it to Agent2.
I know this might seem a little counter-intuitive when comparing the implementation to the real-world problem setting, but the consistency of the RL structure (i.e. Environment that interacts with all the agent) is very important for your solutions to be easily understandable by others.

It would be nice if you could provide a link to some project or research paper that has done something similar to what you're suggesting.

Not aware of any project with this quite unusual setup. It is just matter of changing the environment step function essentially though (and giving the environment object an attribute that stores the reward returned by Agent1 in order to then feed this reward at the next step to Agent2)


text:
Models of reward (possibly mimicking dopamine) in artificial neural networks?
How one can model physiological reward mechanisms occuring in the brain using artificial neural networks? E.g. are there efforts to use the notion of dopamine or similar substances in the artificial neural networks. Maybe introduction of the physiological reward mechanism can lead to the emergence of consciousness or at least enhance the effectiveness of reinforcement learning?
Essentially - how neural network models reward? People's brain perceive money as the ultimate reward because almost everything other can be bought by this. So - mental perception of owning money gives reward. But how this notion of reward is modeled in artificial neural networks? How networks know that some money is assigned to the network's account and so, the network should feel happy and rewarded and should strive to repeat successful behavior?
I am reading https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5293493/pdf/elife-21492.pdf and I hope that it will move me in the right direction.
It is quite confusing. The old-school neural networks expect that there are 2 separate phases: training and inference. So, the network receives all the feedback (let it be called reward) in the training phase and network receives nothing in the inference phase. But maybe network should receive some reward during acting-inference phase as well, kind a lifelong learning.


text:
Neural network with logical hidden layer - how to train it? Is it policy gradient problem? Chaining NNs?
I am doing neural machine translation task from language S to language T via interlingua L. So - there is the structure:
S ->
encoding of S (crisp) ->
S-L encoder -> S-L decoder ->
encoding of L (non-crisp, coming from decoder) ->
L ->
encoding of L (crisp) ->
L-T encoder -> L-T decoder ->
encoding of T (non-crisp, coming from decoder) ->
T

All of this can be implemented in Pytorch more or less adapting the usual encoder-decoder NMT. So, the layer of interlingua L acts as a somehow symbolic/discrete layer inside the whole S-L-T neural network. My question is - how such system can be trained in end-to-end (S-T) manner? The gradient propagates from the T to the L and at the L one should do some kind of symbolic gradient? I. e. one should be able do compute the difference L1-L2?
I am somehow confused by such setting. My question is - is there similar networks which contain the symbolic representation as the intermediate layer and how one can train such system. I have heard about policy gradients but are they relevant to my setting?
Essentially - if I denote some neural network by symbols x(Wi)y, then the training of this network means, that I change Wi and x stays intact. I.e. the last member of backpropagation equation has the form d.../dw1. But if I combine (chain!) 2 neural networks x(Wi)y-y(Wj)z, then the the last backpropagation term for the y(Wj)z has the form (d.../dw1+d.../dy) and hence both the w1 and y should be changed/updated by the gradient descent too. So, doesn't some ambiguity arise here? Is such chaining of neural networks possible? Is is possible to train end-to-end chains of neural networks?
I am also thinking about use of evolutionary training.

The chain rule applies here as usual, and the term symbolic gradient is interesting.
How such might apply will depend much on the nature of the policy layer representation and the connections of that layer or layers to other non-policy layers.

A binary threshold layer may feed a logical inference layer.
A sigmoid function might feed a fuzzy logic layer.
The logical layer could be a production system adapted to look like a layer.
The logical layer might be more like a PLD (programmable logic device).
The output of the logical layer might be mapped one to one with a perceptron or an LSTM layer

In the example in the question, we can't, from the information given in the question, assume that discrete means a discrete approximation of a curve, since the term used is discrete/symbolic. Since the nature of logic cannot be generalized in a differentiable closed form, it may be that it has to be probabilistically represented in closed form so a derivative may be obtained, if that is possible in the specific case under study.
Back propagation is a feedback mechanism to provide corrective signaling to the portions of the system that must be corrected. The hope is that the direction of the correction during back propagation leads to convergence on the global minimum of loss without severe delay, which is not always easy to make happen, thus Gaussian noise injection explicitly or indirectly through mini-batching, momentum, multiple initialization states, parallel searches with different hyper-parameters, and other techniques.
One technique used quite a bit in analog systems and entering use in digital systems in recent years is the idea of multiple corrective mechanisms. Control theory for instrumentation and countermeasure aeronautics is thick with this concept. For the case given in this question, it can look like this, with $f$ representing feed forward layer with some activation function, $\ell$ representing some logical inference container, $p$ is the parameter set for the function of the same subscript, $\epsilon$ is the evaluation function (error, loss, value, benefit), $b$ representing the corrective feedback using the back propagation, the Jacobian, and the chain rule.
$$ f_1 \quad \rightarrow \;\quad \ell \quad \rightarrow \quad\ f_2 \quad \rightarrow \quad\ \epsilon \\
 p_1 \leftarrow \, b \leftarrow \ell\, ; \quad\quad\quad \; p_2 \, \leftarrow b \leftarrow \epsilon  \\
\quad\quad\quad\quad\quad \ell \quad\quad \leftarrow \quad\quad\quad\quad\quad \epsilon
 $$
In the last line $\epsilon$ is simply a fact (piece of information) passed into the a rules engine session or a floating point representation of a feedback signal assigned as a probability to a rule in a fuzzy logical container. In the first half of the second line, the logical unit is responsible for providing the evaluation function output.
In Norbert Wiener's Cybernetics, he states the following1 with regard to automating the steering of ships via rudder control, from which cybernetics got its name.

In the important book of MacColl2, we have an example of a complicated system which can be stabilized by two feedbacks but not by one.

He then procures the differential equations and proves MacColl's conclusion. That conclusion is used in nearly every instrumentation amplifier circuit on a chip sold today, if not all of them. There is no way to get the desired stability without the multiple feedback paths.
Similarly, it may not be possible to stabilize the convergence of $f_1, \ell, and f_2$ via a single mechanism of back-propagation using $\epsilon$ and the Jacobians to descend a gradient.

References
[1] Norbert Wiener's *Cybernetics, 1948, MIT, p 106 in the 1996 printing
[2] LeRoy A. MacColl. Fundamental Theory of Servomechanisms. Van Nostrand, New York, 1945


text:
VIsual/musical/multimedia discourse (analysis) - are there such notions?
Formal semantics of natural language perceives sentences as logical expressions. Full paragraphs and even stories of natural language texts are researched and formalized using discourse analysis (Discourse Representation Theory is one example). My question is - is there research trend that applied the notion "discourse" to images, sounds and even animation? Is there such a notion as "visual discourse"?
Google gives very few and older research papers, so - maybe the field exists, but it uses different terms and Google can not relate those terms to my keyword "visual discourse".
Basically - there are visual grammars and other pattern matching methods that can discover objects in the picture and relate them. But one should be able to read whole store from the picture (musical piece, multimedia content) and I imagine that such reading can be researched by multimedia discourse analysis. But there is no work under such terms. How it is done and named in reality?

There are several papers on Google Scholar that describe "visual discourse", but this phrase might have multiple distinct meanings.


text:
How to connect AI neural network processor to laptop?
I have an average laptop.
How can I connect specialized AI neural network processors (say, Intel Nvidia or Intel Nervana https://venturebeat.com/2018/05/23/intel-unveils-nervana-neural-net-l-1000-for-accelerated-ai-training/) to thelaptop.
Should I buy some external motherboard or even server unit with NN processors inside or is there available more lightweight solution like external HDD?

are you interested in training or inference?

I am interested in both training and inference, but I guess that initially could be good to set up at least one process. I guess, that training is the resource consuming process for which the processors are designed, inference can be done with the ordinary processor.

Check of deep learning box on the internet.
You will have to think hard in order to make a good and cost effective one.
But there are a few people who have made one of their own and shared their experiences.
Here are some medium articles:
Building your own Deep Learning dream machine
Building a 270* Teraflops Deep Learning Box for Under $10,000


text:
Neural network as (BDI) agent - running in continuous mode (that do inference in parallel with learning)?
Is there research work that uses neural network as the (BDI) agent (or even full-scale cognitive architecture like Soar, OpenCog) - that continuously receives information from the environment and act in an environment and modifies its base of belief in parallel? Usually NN are trained to do only one task and TensorFlow/PyTorch supports batch mode only out of the box. Also NN algorithms and theory are constructed assuming that training and inference phases are clearly separated and they have each own algorithms. So - completely new theory and software can be required for this - are there efforts in this direction? If no, then why not? It is so self-evident that such systems can be of benefit.
https://arxiv.org/abs/1802.07569 is good review about incremental learning and it contains chapters of implemented systems, but all of them still separates learning phase from inference phase. Symbolic systems and symbolic agents (like JSON AgentSpeak) can have updating belief/knowledge base and they can also act during  receiving new information or during forming new beliefs. I am specifically seeking research about NNs which do learning and inference in parallel. As far as I sought then this separation still persists in self-organizing incremental NNs that are gaining some popularity.
I can image the construction of chained NNs in Tensorflow - there is some controller network that receives input (possibly preprocessed by hierarchically lower networks) and that decides what to the: s.c. mental actions are the ouput of this controller, these actions determine whether some subordinated network is required to undergo additional learning or whether it can be temporary used for the processing of some information. Central network itself, of course, can decide to move into temporary learning phase from time to time to improve its reasoning capabilities. Such pipeline of master-slave networks is indeed possible in TensorFlow but still TensorFlow will have one central clock, not distributed, loosely connected processing. But I don't know whether existence of central clock is any restriction on the generality of capabilities of such system. Well, this hierarchy of networks maybe can be realized inside the one large network as well - maybe this large network can allow separate parts (subsets of neurons) to function in somehow independent and mutually controlling mode, maybe such regions of large neural network can emerge indeed. I am interested in this kind of research - maybe there are available some good papers for this?

You might be interested in the Clarion cognitive architecture, developed by Prof. Ron Sun and collaborators.
Full disclosure: I am a student in Ron Sun's Cognitive Architecture Lab.
Brief Description of Clarion
Clarion agents are composed of several subsystems, each of which may contain several neural networks. These subsystems include the Action-Centered Subsystem (ACS), the Non-Action-Centered Subsytem. The ACS controls action decision making, the NACS stores general knowledge. During an activation cycle, the ACS might request information from the NACS, setting the NACS activation cycle in motion.
Strictly speaking, Clarion does not adopt the BDI framework. But, various components of a Clarion agent can be put in correspondence with BDI concepts. BDI beliefs correspond roughly to knowledge in the ACS and NACS. The Motivational Subsystem (MS) contains agent drives and sets agent goals, which roughly correspond to BDI desires and intentions respectively.
Clarion agents do not directly control when learning happens, but may control which subsystems are active based on task demands, and may output mental actions as part of their processing. Learning generally happens at the end of a subsystem activation cycle based on feedback from the environment or from the agent's own motivational and metacognitive subsystems.
Links
A page on Prof. Ron Sun's site links to several resources on Clarion.
Currently the most up-to-date reference on Clarion is the book Anatomy of the Mind. A precis is available in the journal Cognitive Computation.


text:
Can neural network take decision about its own weights (update of weights)?
Can neural network take decision about its own weights (update of weights) during training phase or during the phase of parallel training and inference? When one region of hierarchical NN takes decision about weights of other region is the special case of my question.
I am very keen to understand about self-awareness, self-learning, self-improvement capabilities of neural networks, because those exactly those self-* capabilities are the key path to the artificial general intelligence (e.g. Goedel machine). Neural networks are usually mentioned as examples of special, single-purpose intelligence but I can not see the reason for such limitation if NN essentially trys to mimic human brains, at least in purpose if not in mechanics.
Well - maybe this desired effect is already effectively achieved/emerges in the operation of recurrent ANNs as the effect of collective behavior?

What do you mean  by "take decision"? In back propagation, the gradients at one layer are used to derive the gradients of another layer. However, it is unlikely anyone would frame that as a "decision" because there are no alternatives to choose between at the point of the update; it is driven by maths of the error function, chain rule and whatever optimisation routine is in use.

Backpropagation is just one algorithm. If we require neural network to be self-improving (Goedlian) then we should allow neural network to update its own learning algorithm and it effectively means that neural network should be able to take decisions about its own weights.

Yes, but what does "take decision" mean to you? What are you asking? Does reinforcement learning count, for instance - in that, the neural network is used to make a decision about which action to take, and that then determines the next piece of data. That is clearly a feedback loop where a decision due to the neural network affects the next weight update . . . does that count?

I mean: in one learning cycle NN outputs adress i-j of weight wij and new value w_ij for this weight. Or it outputs the whole array of addresses and new weights. Of course one can always encode this in the output of NN. And then some more ore less external mechanism writes those new wights back into NN. I don't know whether some existing NN technique already simulates this or I am seeking for something entirely new thing that even may not be accomodated inside existing frameworks.

I think you need to consider on what basis the network would output a change to its own weights. What the goal would be, what cost function would be minimised. I suspect this is not meaningful in the context of a single NN, but is something you could end up with, with more complex multi-part architectures - even then you need to be very clear about the goals and types of interaction of the components.

I think OP is aiming for something closer to G.E.B.'s "strange loops" but without a solid understanding of NNs, so the question is hard to understand/interpret. https://en.wikipedia.org/wiki/Strange_loop

Establishing Names and Terms
The mathematician's name is Kurt Gödel, an Austrian-American, and the machine concept named after him is the Gödel machine.  The idea of meta-learning (learning how to learn) can be extended recursively to learning how to learn how to learn how to learn and so on.
Taking a decision is when the decision is external to the system and the system either copes or executes that decision, whereas making a decision is when the decision is internal to the system and the system executes it, reports it, or delegates it.
Both of these have been tried in the laboratory or in real systems and various levels of recursion have achieved success, but not indefinite recursion.
A Few Points from AI History
Creating a learn how to learn recursion was the object of several early LISP efforts at MIT, and meta-learning is in active research in multiple universities and corporations as of this writing.  The first successful application of a single level of recursion were the expert systems that could acquire knowledge using meta-rules, which were rules about creating new rules.
Current Artificial Networks Regarding Meta-learning
It is correct that back propagation is not meta-learning.  It is a single level of learning in the above recursive learning concept.  Artificial networks have been applied to learn to adjust the hyper-parameters of another network, but these strategies require large quantities of data, and they don't cognate.
They don't know how to adjust hyper-parameters.  They don't build models of learning and then use those models to learn.  The adjustment mechanism is the application of a learned function, not a conception of learning.
Requirements for Meta-learning
To establish a meta-learning paradigm, we must consider the elements of learning.

Existence of function abstraction
Ability to guess a parameterized function
Ability to mutate that function
Ability to tune the parameters
A target functional behavior
A way to detect whether the function is approaching the target behavior
A strategy for deciding when to mutate and when to tune
A way of performing experiments in isolation
A way to recurse in a way that incorporates ALL of the above

Once this has been achieved, it may be described as self-improvement and may qualify as what most theorists think of as a Gödel machine.
Specifics in the Question

Can [an artificial] network [make] decision about its own weights (update of weights) during training phase or during the phase of parallel training and inference?

Yes.

[Can] one [level] of [a] hierarchical [artificial network] [make] decision[s] about weights of other [levels]?

Yes, but in a limited way as of the time of this writing.

I am very keen to understand about self-awareness ...

Self-awareness is the ability of the system to analyze and either utilize the result of that analysis, report it, or delegate based on the result IN COMBINATION WITH the ability to use itself as the object of analysis.
This can be as simple as a program that parses Java and produces statistics using its own code as the code to parse.  It can be as complex as an anthropologist studying humanity.  It can be as personal as a human looking in the mirror and wondering about the qualities of person they are.  It can be as deep as someone wondering their purpose in their current life.
In all cases, there are those two qualities.

The ability to perform a type of analysis that could be applied toward a class of objects of which itself is a member
The selection of itself for analysis

There is one other aspect of awareness beyond these two that are not required but usually associated with awareness: Some way of directing attention of the analysis capability to itself regularly.  Here are a few examples from human experience.

Practice of daily meditation
Reading a book on purpose
Seeing a cognitive behavioral therapist
Writing an autobiography
The first, fourth, and tenth Step of anonymous programs
Watching a candid movie of one's self
A family member that likes to step back and evaluate the family
Religious accountability groups

Self learning is like self-awareness except that learning replaces awareness.  Obviously, self-learning is dependent on some degree of self-awareness.  We now have a more activity based list because of the additional element of adjustment or inception of action.

Decide, on the basis of personal abilities and interests, after evaluating options for living, to relocate
Realizing that self cannot remember commitments to meetings to buy a technology device that notifies the user before such commitments and using the calendar app rigorously
Finding that repeated attempts to eat greens rather than junk food is not working and joining a community of those with eating disorders with solutions to that issue
Deciding that childhood views of future were good ones and abandoning current paths for ones that were bolder and will likely lead to fulfillment

Self-improvement is a super-set of self-learning, since the only point of learning is improvement, however there are forms of improvement that don't strictly involve learning.  A system (or person) can execute something already learned that results in an improvement.

Artificial general intelligence (e.g. Gödel machine)

There is no mathematical proof that a Gödel machine would exhibit general artificial intelligence.  More importantly, there is no proof and a considerably body of evidence against the proposition that humans are generally intelligent.
Lastly, there is no proof that general intelligence is achievable in a biological or artificial system.  Gödel's second incompleteness theorem is strong evidence that such ideas may be naive.

[Artificial] networks are usually mentioned as examples of special, single-purpose intelligence, but I can not see the reason for such limitation if [they] essentially [try] to mimic human brains, at least in purpose if not in mechanics.

Artificial networks emerged out of a desire to mimic human brains, but the perceptron design is based on an old view of neuron functionality.  Simulating a single neuron might require an entire rack of CPUs and GPUs.
Furthermore, we are not yet clear on what intelligence is.  Definitions proposed vary widely.  The fields of artificial networks, bioinformatics, ontology of ideas, semantics, and the relationships between stable adaptivity and algorithms are all in their early stages, both theoretically and experimentally.

Maybe this desired [capability] is already effectively achieved [or currently emerging] in the operation of recurrent ANNs as the effect of collective behavior?

No, but RNNs are a tiny step closer to biological neurons in that their layers maintain state beyond the attenuation matrix.
By attenuation matrix is meant the matrix of parameters used in vector-matrix multiplication to control the strength of signals from the activation functions of one layer to the activation functions of the next.  It's common name in machine learning literature is simply, "The parameters."  Learning occurs as they change to converge the network on an optimal state.
RNNs are also capable of being Turing complete, so they can theoretically realize arbitrary algorithms.
However, the Requirements for Meta-learning above are not an expected capability of any single instance of an artificial networks, either of the MLP (multilayer perceptron) type or the RNN type.  Whether balancing two networks in symbiotic arrangement as in GANs, or a comprehensive recursive algorithm, or the simulation of neural and synaptic plasticity in silicon will lead to Gödel machines is unknown.
Let Science be Science
Whether ideas such as general intelligence are possible is unknown.  Whether ideas about singularities in books and mass media are realistic is unknown.  Whether the prophetic warnings of screenwriters about the emergence of artificial entities that become the dominant species on earth is unknown.
Jaques Ellul is perhaps the most scientifically accurate prophetic voice.  In his Techological Sociey, he presents heaps evidence in support of the idea that humans are already serving an autonomous technology and have been since prior to industrialization.
A Swiss philosopher, Francis Schaeffer, once prophecied, "They will say all kinds of things in the name of science that have nothing to do with science."  We should be careful to keep conjecture from reaching the status of scientific fact in technical conversation.  If we have no carefully drawn theory or empirical evidence to support a conjecture, it should be stated as a proposal, not a conclusion.


text:
Neural machine translation, that outputs multiple alternative, ambiguous translations?
Is there neural machine translation methods, that for one input sentence outputs multiple alternative output sentences in that target language. It is quite possible, that sentence in source language have multiple meanings and it is not desirable that neural network discards some of the meanings if there is no context for disambiguation provided. How multiple outputs can be acommodated into encode-decoder architecture, or different architecture is required?
I am aware of only one work https://arxiv.org/abs/1805.10844 (and one referen herein) but I am still digesting whether their network outputs multiple sentences or whether it just acommodates variations during training phase.

The LSTM seq2seq model typically used for language translation actually can output multiple variations, and does in many implementations.
The decoder stage outputs confidence/probabilities for each word, and it is possible to run the decoder multiple times, taking different samples on each run.
Not only is it possible, but this is actually done by translation engines. A technique called BEAM search is used to manage multiple translations in progress and pick an eventual one to display, based on whether the language model predicts an overall high or low probability for a sentence (this allows for sentences that start with an unusual initial word choice but that are a better choice in context of what is being translated). It would be a minor change to existing algorithms to display a selection of these instead.
Technically you don't need to sample randomly, and can just take the top N probabilities on each step, and recalculate each time on combined probabilities, culling partial sentences that have low probability in favour of continuing more of the high probability sentences.

What was meant by "taking different samples"? Where is this source of variability? Decoder receives the same input and the same context (from the encoder)..

@TomR: The decoder does not output absolute word choices, it outputs word probabilities. Instead of taking the max probability as the only choice, sample according to the probabilities, using a random number generator. In addition the decoder accepts the previous word and state that it output as input for the next item in the sequence, so as soon as the sentence is started with a different word, this sets the translation on a (possibly very) different trajectory.

@TomR: Technically you also don't need to sample, just take the top N probabilities on each step, and recalculate each time on combined probabilities, culling partial sentences that have low probability in favour of continuing more of the high probability sentences. That works just fine for BEAM search

Many thanks. But is there some published work about such approach or is it folklore shared only among experts (I am novice in this field)? Maybe there is some good keywords. Google gives lot of answers for "multiple output neural network", but not for NMT.

@TomR: I'm pretty sure there are published works. However, I got my understanding of this from Andrew Ng's course Sequence Models. This is a relevant lecture https://www.coursera.org/learn/nlp-sequence-models/lecture/v2pRn/picking-the-most-likely-sentence

I know I am very late to this party, but the ideal search terms are "beam search nmt". One relevant published work is https://www.aclweb.org/anthology/W17-3207.pdf - it also includes references to the works that introduced beam search for sequence models in the first place.


text:
How to understand marginal loglikelihood objective function as loss function (explanation of an article)?
I am reading article https://allenai.org/paper-appendix/emnlp2017-wt/ http://ai2-website.s3.amazonaws.com/publications/wikitables.pdf about training neural network and the loss function is mentioned on page 6 chapter 3.4 - this loss function O(theta) is expressed as marginal loglikelihood objective function. I simply does not understand this. The neural network generates logical expression (query) from some question in natural language. The network is trained using question-answer pairs. One could expect that simple sum of correct-1/incorrect=0 result could be good loss function. But there is strange expression that involves P(l|qi, Ti; theta) that is not mentioned in the article. What is meant by this P function? As I understand, then many logical forms l are generated externally for some question qi. But further I can not understand this. The mentioned article largely builds on other article http://www.aclweb.org/anthology/P16-1003 from which it borrows some terms and ideas.
It is said that l is treated as latent variable and P seems to be some kind of probability. Of course, we should assign the greated probability to the right logical form l, but where can I find this assignment. Does training/supervision data should contain this probability function for training/supervision data?

I cannot answer your question but I am stuck in a similar rabbit hole so hopefully these references can help you.
The loss function you are describing would be 0-1 loss. However, 0 would be the if our output matches and 1 would be if it does not. This function is not smooth and not convex. Thus we often replace it with a surrogate loss function such as log likelihood.
You can read more about surrogate loss function on pg 269 of Deep Learning by Ian Goodfellow available here: https://www.deeplearningbook.org/
or here : https://en.wikipedia.org/wiki/Loss_functions_for_classification
I did not have time to read the article, but the reason that you are seeing a probability is that they are using a Bayesian framework for neural networks. This was first described in A Practical Bayesian Framework for Backpropagation networks by McKay 1992.
It is explained very well in a video series by Hinton on youtube here:
https://www.youtube.com/watch?v=YcwZFNd3UvI
Hopefully these references help you I wish I could explain in greater detail.


text:
Regional specialization in neural networks (especially for language processing)?
What is the status of the research on regional specialization of the artificial neural networks? Biology knows that such specialization exists in the brain and it is very important for the functioning of the brain. My thinking is that specialization can solve the transfer learning/catastrophic forgetting by creating centers of sophisticated skills if such sophistication is necessary. Actually - there is no much alternative to specialization? If specialization exists then there can be small decision centers that routes the request to the actual part and such decision centers can be efficient. But if specialization does not exist then routing happens in the total soup/pudding of neuronal see and such all-to-all routing should be very inefficient. Of course there should be some mixing of specialization vs pudding, because there is always mixing between rationality and emotions, between execution and exploration, but nevertheless - specialization should happen, at least partially.
The problem is - that I can not find any focused article about such specialization and I can not find how such specialization can be trained? There are research on hierarchical reinforcement learning - but that is about imposing the external fixed structure on the set of neural networks, but it is not how the nature works - nature implements such hiearchy within neural network and not by imposing rigorous, symbolic structures.
Are there some notions, terms, keywords, research trends, important articles (and researchers) devoted to such specialization (including the machine learning of such specialization).
Of course, my topic is very large, but the actual work on this is small ir nonexistant and that is why it is focused.
There is work on convolution neural networks but maybe there is another approach for language processing where the parts can be specialized in - parsing, understanding, anaphora resolution, translation, etc? And is the convolution the kind of specialization I am seeking?
Maybe the notion of attention somehow is connected with my question. But usually the attention is connected with the single neurons and not with the regions? Maybe there is notions about hierarchy of attentions - one level of attention values refers to high level tasks/skills, but another level of attention values refers to subskills, etc.


text:
Can the agent of reinforcement learning system serve as the environment for other agents and expose actions as services?
Can the agent of reinforcement learning system serve as the environemnt for other agents and expose actions as services? Are there research that consider such question?
I tried to formulate the problem of network of reinforcement learning systems in other site of Stack network: https://cs.stackexchange.com/questions/111820/value-flow-and-economics-in-stacked-reinforcement-learning-systems-agent-as-r
So - is there some research along such stacked RLs? Of course, I know that Google is my friend, but sometimes I come up with the ideas that I don't know how to name them or how they are named by other scientists who have already discovered and researched them. And that is why I am useing stackexchange to give me some keywords for my ideas and I can myself explore futher the field using those keywords. So - what are the keywords of resarch about such stakced RL systems, of agent-environment interchange in reinforcement? learning?