text:
What would the commercial application of a conscious AI look like/be?
Sometimes, but not always in the commercialization of technology, there are some low hanging fruits or early applications, I am having trouble coming up with examples of such applications as they would apply to a conscious AI.
As per conscious I would propose an expanded strict definition: the state of being awake and aware of one's surroundings along with the capability of being self aware.
Thanks.

Why do you need consciousness? Strong AI would (by definition) give you a replicable commodity that's at least as smart as a human. It may be that consciousness isn't necessary for that.

Answering a question with a question ?, Strong AI ? What is that, whose definition ? How do you know that consciousness is not required to be as smart as a human ?

I don't see how there can be any real answers to this.  Given that we barely (if at all) understand what consciousness is, and can't even be sure that other humans are conscious, I think we can only speculate in SWAG (Scientific Wild Assed Guess) fashion about potential applications for a conscious AI.   At any rate, it's not definitively established that a "conscious" AI could do anything that a non-conscious one can't.

@kenorb - the generally accepted definition of Strong AI is 'as smart as a human, in a non-task-specific fashion'. No-one knows whether consciousness is required for this, so Occam's Razor says there's little point in assuming it's necessity in advance of supporting evidence.

@NietzscheanAI: 'non-task-specific' and 'as smart as a human' are contradictory in my view /research.

@Keno - See the definition of 'Strong AI'.

@NietzscheanAI : Where ? ( Are you talking about that single speculative article): https://www.ocf.berkeley.edu/~arihuang/academic/research/about.html

Or Wikipedia, for example. Do you have a proposed alternative definition of Strong AI?

They may be just for fun. If you had a robot that understood you, could hold a conversation with you about your interests, and even had goals of its own (good or bad), it wouldn't really need to do anything special. People would buy it like it was a toy or game.
Also, they might be usable as programmers, artists, designers, anything creative that a computer can't successfully do on its own.
It really just depends on what you define as 'consciousness'. Does it just understand what it's supposed to do, decide if it wants to, and if so, complete the task? Or does it wonder about religion, politics, moral situations, etc. that even regular humans don't fully understand? If it was pretty much just a human, it wouldn't be any more useful than one. Of course unless it can solve problems super quickly and effectively, then it would just be a really good worker.

Thanks, let's just start with the strict definition of consciousness: "the state of being awake and aware of one's surroundings", i would also like to add the capability of being self aware, I've edited my question.

With that definition, people and companies could buy and use the AI to do things like making self-driving cars (already done), computers that run businesses, computers that ask and answer their own questions about physics, quantum mechanics, and other paradoxes. A self-aware computer would be able to analyze a situation in much more detail than a human, and apply new information to other things. For example, it took us humans 57 years after microwaves were proven to exist for us to realize they could cook food. One of these machines would figure it out within minutes or seconds.

I have implemented a system which does exactly this, i say that it would pass the turning test. though I don't feel the world is ready for my system yet. they still have far to go in other sections..of tech...i need more processing speed, so i have being working on a 256 bit CPU which will also be quad call ^.^

256bit cpu roughly supports  just over 11579209000000000000000000000000000000000000000000000000000000000000000000000000 GBs of Memory

The answer can be simplified, if consciousness means human consciousness then.
What would the commercial application of a Human look like/be ?
So now every one know the commercial applications of Humans.

Consciousness is not a scientific concept. Fringe scientists who theorize about consciousness are generally shunned as psudo-scientific heretics by the hard science community. Conciousness is a meta-physical or philosophical concept.
"I think, therefore I am." is the only proof that consciousness exists that I am aware of. Therefore, you cannot even prove that a person other than yourself is conscious. So how could anyone even prove that a computer program is conscious? What would be the observable difference between a program that IS conscious, and a program that simulates the results of consciousness?
I don't believe that you can program conscious AI, nor could you prove that you have done so. Consciousness isn't something that can ever be marketed. You can only market the AI on the basis of it's problem solving capabilities.


text:
Semantic Segmentation how to upsampling
Many of the architectures that do semantic segmentation like SegNet, DilatedNet (Yu and Koltun), DeepLab, etc. do not work on high resolution images. For such benchmarks like Cityscapes, what is a standard/practical approach for such methods to perform on the benchmark?
I've tried to look into the paper, but I couldn't find such details. There's an article mentioning that they output at 1/8 of input images than do interpolation (usually 2, 4 or 8 times) from their results, but the article does not specify which upsampling techniques are the most reasonable one.

You can always train your network with higher resolution images. There is nothing preventing you to do that if you don't have any restriction about inference time.
Also, the paper is actually mentioning how to upsample. Check for the phrase "Upsample using deconvolutional layers".
The most common ways to upsample are either using the deconvolutional layer or just by simple resize methods (like image resize).

It kinda fascinates me to know that people just employ simple bilinear-interpolation image resize to scale up and down. Do you have experiment how much artifact that would introduce, say image and annotation might be mis-align due to the scaling process?

Nope. As soon as you scale as floating point, it should be fine.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
Do AI-based code-generators guarantee correct output?
I haven't spent much time looking at AI-based code-generators. What mechanism is used to generate code and how is it different than standard NLP?

Human coders can't guarantee correct code, and we don't have any way to check whether code is correct (except in some trivial cases). How could we possibly expect more of AI coders?

They have no such guarantee. This is true for many reasons, but the simplest of which is:

The definition of correct output is very underspecified.

We do not have a rigorous way of defining exactly what sort of program is being defined by the natural language, so it would make no sense to think an AI could rigorously be proven to fulfill that requirement.
While that is itself sufficient, there is one more fundamental reason why there is no such guarantee:

While it depends on the code generation solution, many are based on GPTs or similar types of architectures that are probabilistic and prone to hallucination. In fact, any sort of Machine Learning based solution will defy such proofs. At best we can make a type of PAC argument

You don't need to make a PAC argument, since we actually have test sets available. PAC guarantees tell you that if your function class is simple enough, your test error will be within some interval of your train error. In practice this doesn't matter at all since we can't calculate the bounds, and our function classes aren't simple.
Fortunately it doesn't matter: If we have some representative set of tasks, and the model is successful 80% of the time, we can just use statistics 101 stuff to get a bound on its error rate.

Even when you think it is correctly specified, when you have a multi-step process of additions and modifications to the original code it's understandably hard for AI to 'remember' or even understand what context is still relevant and what is not. Hence the output being unexpectedly wrong at times.

Do AI based code generators guarantee correct output?

No. There is no written guarantee by any provider.
Researchers have measured "accuracy" (there's more than one way to think about this of course, since code may solve a problem, but be poorly written), and they do not find it is 100% accurate. A value of 80% accuracy was suggested in a survey by Akevlon.

What mechanism is used to generate code and how is it different than standard NLP?

It is the same models as NLP, except for data it has been trained on (although LLMs like ChatGPT are general enough that the exact same model - trained on code, poetry, newspaper articles, novels etc - performs coding alongside natural language tasks). This may be augmented at generation phase, because it is possible to use compilers to filter out things that don't work at all. But compilers cannot determine that a task is performed correctly or that suitable names are used for variables etc.

Is it guaranteed to even be syntactically correct? As in generate a compilable program? A while back I saw a research presentation where basically they tried to mix RL, ML and classic NLP(tokens and shit) to guarantee a compileable program.

@FourierFlux - in general no, but as I point out in the answer, the makers can in theory use a compiler to filter against multiple generations and pick one with least errors. I don't know if anyone does that, but I suspect some code gen services will because it is a lot easier than deploying the AI side of the model

Current implementations do not.
However, if you had an exact and executable definition of "correct output", such as a comprehensive test suite, it could be combined with the code generator. Most language models are quite adept at improving the code based on feedback, so for most assignments this would lead to code that fulfills all test cases.
There is still a risk of over-specialization, leading to code that works correctly for the test cases, but not for inputs outside that.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How does PCA work when we reduce the original space to 2 or higher-dimensional space?
How does PCA work when we reduce the original space to a 2 or higher-dimensional space? I understand the case when we reduce the dimensionality to $1$, but not this case.
$$\begin{array}{ll} \text{maximize} & \mathrm{Tr}\left( \mathbf{w}^T\mathbf{X}\mathbf{X}^T\mathbf{w} \right)\\ \text{subject to} & \mathbf{w}^T\mathbf{w} = 1\end{array}$$

Related: https://math.stackexchange.com/q/3736092

You might want to have a look at the wikipedia article of PCA, where it says:
"The $k$th component can be found by subtracting the first $k − 1$ principal components from $\mathbf{X}$:"
$$\hat{\mathbf{X}}_k = \mathbf{X} - \sum_{s=1}^{k-1}\mathbf{X}\mathbf{w}_s\mathbf{w}_s^T$$
Then you repeat the process to find the next component:
$$\mathbf{w}_k = \arg\max \mathbf{w}^T\mathbf{\hat{X}}^T_k\mathbf{\hat{X}}_k\mathbf{w}$$
$$\text{s.t. } \mathbf{w}_k^T\mathbf{w}_k = 1$$

You can also understand the logic from the view of constrained optimisation.
Introduce a Lagrange function:
$$
\mathcal{L} = \text{Tr} (w^{T} X X^{T} w) - \lambda w^{T} w
$$
And take the derivative with respect to $w$:
$$
\frac{\partial \mathcal{L}}{\partial w} = 2 (X X^{T} - \lambda) w
$$
For the general case of dimension $\geqslant 1$ $w$ is a set of vectors $w = (w_1 w_2 \ldots w_n)$.
This expression vanishes, if for some index $i$ $w_i$ is an of eigenvector of $XX^{T}$ with the eigenvalue $\lambda_i$, and all other components are set to zero. In other words, stationary points are the eigenvectors of $X X^{T}$.
The condititon $w^T w = 1$ imposes the orthogonality condition on the eigenvectors. In fact, going back to the initial functional, one sees, that $w_i X X^{T} w_j = \lambda_j w_i^{T} w_j = 0$ for $i \neq j$. Therefore, we have finally:
$$
\mathcal{L} =\sum \lambda_i - \lambda
$$
Which is maximized for any $k \geq 1$, by taking $k$ largest eigenvalues.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How do neural networks play chess?
I have been spending a few days trying to wrap my head around how and why neural networks are used to play chess.
Although I know very little about how the game of chess works, I can understand the following idea. Theoretically, we could make a "tree" that included every possible outcome of a chess game. Through knowledge provided by chess experts, we could identify how "favorable" certain parts of this tree are compared to other parts of the tree. We could also use this tree to "rank" optimal chess moves based on how the chess board appears in the current turn (e.g. which pieces you and your opponent have left and where these pieces are situated).
The problem is, this tree would be so enormous that it would be impossible to create, store and "search" (e.g. with the MinMax algorithm):

I understand that perhaps this tree can be created using data to limit the size of the tree based on scenarios that are more likely to appear compared to all possible scenarios. For example, if a player wanted they could spend the whole game aimlessly shifting their "Rook" back and forth - theoretically, this outcome could occur but no player (in their sane mind) would ever do this. Thus, the tree could be constructed using actual data from millions of chess games. This for example could tell us : Based on historical data and given the current setup of the chess board, 21% of games were won when the immediate next move involved moving the Queen to "F5" vs only 3% of games were won when the immediate next move involved moving the Knight to "F5". I suppose at each move, the data based tree could be queried to rank the optimality of each next move by checking the proportion of "terminal nodes" that resulted in wins for each possible move given the current chess board.
However, I still see 2 problems with this approach:

It is possible that we might run into a scenario(s) that never occurred within the historical data, rendering the tree useless in this scenario

This tree still might be too large to efficiently store and query.

This is probably why neural networks are being used to play chess - I tried to do some readings about this topic, but I can't seem to fully understand it. In this case, what exactly would the neural network use as a loss function? I don't see how the loss function in this case is continuous, and thus how could gradient descent be used on such a loss function?
Could someone please recommend some sources (e.g. YouTube Videos, Blogs, etc.) that show how a neural network can be used to play chess.

You're raising multiple issues here. First, you're asking what the loss function would be. Another issue that you raise is: what to do if we have no data for a particular situation. I understand that the overall issue is how can neural networks be used in the context of chess, but you should narrow the scope of your posts to just one specific question. Is your main question "What the loss function would be in this case?" or is the question in the title (which is broad)? You don't describe what the input and output of the neural network would be, so it's not clear if you know that or not.

I think its also worth pointing to the chess programming wiki which covers the basics (and not so basics) of chess engines. Very useful resource if you're planning on developing your own engine be it using a hand-crafted or NN-based evaluation function. Specifically NNUE may be what you're looking for.

For the large game tree analysis, you could analyze it with combinatorial game theory—no need to get into statistics and historical data. As you say though, it's a moot point since the game tree would be unfathomably big.

"It is possible that we might run into a scenario(s) that never occurred within the historical data, rendering the tree useless in this scenario" - the ability to generalise to unseen data is one of the main reasons AI is used in the first place. One could easily generate the tree dynamically (because it's trivial to generate all the moves you can make from a specific board state), with each board state / node in the tree (at some depth) being evaluated by the neural network to generate a score used by minimax. So unseen data shouldn't be a problem at all (if you've seen similar enough data)

Just a comment on your tree. It actually exists when there are 6 or maybe by now even 7 pieces left. And due to this tree they found out that some chess rules are actually wrong. There is a 50 move rule to prevent games from going for ever. If no piece has been taken and no pawn advanced in 50 moves, the game is considered to be a draw because no progress can be made to win the game. But due to the tree they found out, that in certain situations there can be a combination of 200 moves without taking a piece and advancing a pawn that leads to a win. So the rule was based on a bad guess.

Minimax and related algorithms are used to play chess. That is how chess programs have worked for many years (with some additions such as standard opening playbooks). They do not need to process the whole game tree. There are a few different techniques used to reduce the effective search space. One of the most the most impactful is to truncate search after a certain depth if it has not resolved the game, and to score the position reached using some form of heuristic function that takes the board state as input and outputs a number. The impact from this has two parts:

The search space is reduced by a large factor.

The strength of the resulting automated player is limited by how effective the heuristic is at reflecting the strength of the game position. A perfect heuristic could allow a very limited search (just one ply look ahead, or up to some point within the game) to play perfectly. However, it is a practical impossibility for many games - if a game is "unsolved" that means by extension that there is no known perfect heuristic.

There are other ways to reduce search space or search it more efficiently, but this search depth plus heuristic pattern is very common in turn-based games. This is how IBM's Deep Blue worked for example, and it did not use neural networks. The heuristic function in Deep Blue was constructed by experts extending already well-studied basics of chess playing, such as assigning simple points value to each side's remaining pieces (e.g. a rook is worth more than a bishop).

What exactly would the Neural Network use as a Loss Function?

There are two main ways that a neural network can be applied in a game like chess:
1. To predict the chances of winning from a given position
(or equivalently score a heuristic for the position)
In reinforcement learning terms, this would be the value function for that state (game position).
This can be used alongside tree searches, the main difference being that instead of an expert designing the heuristic, the neural network will have learned one from observing games. Those games could be a database of games played by humans, or generated by a learning agent playing against itself.
The loss function that drives learning here would typically be a cross-entropy loss comparing the eventual win/loss with the predicted probability. This is noisy of course, because the same position may lead to win, draw or loss depending on relative strengths of the players. However, as the agent player becomes stronger, the prediction will start to match it, and more closely approximate the probabilities from optimal play.
2. To predict the best move from a given position
This is also called a policy function.
This is a multi-class prediction problem, that would use the normal multiclass log loss function. Similar to learning the value function, it can be learned from observation assuming all action choices that lead to a win are the best choices. Again this is noisy, but should converge on an approximation of optimal play provided the input is from master-level play database, or is from a working self-play system.

I don't see how the Loss Function in this case is continuous, and thus how could Gradient Descent be used on such a Loss Function?

Once you move away from trying to have the neural networks process a game tree directly, the problem becomes clearer. You need to run the game tree processing using suitable tree search algorithm (minimax, negamax, MCTS etc), and have neural networks for the value and/or policy function estimations at each node, which they can learn from any relevant data.
If you want to create self-learning agents, as opposed to training them from a pre-existing database of games, then you will need to study systems that can generate suitable data from experiencing game play directly. This adds another outer layer of logic alongside the tree search algorithm that helps decide what data to collect and how to score it so it can be used as training data for the neural networks.
There is quite a lot you could study about these kinds of learning systems. I recommend for the basics that you look into reinforcement learning (RL), and the go-to book for many people there is Sutton & Barto's Reinforcement Learning: An Introduction, which lays out the theory and terminology used in RL and a variety of basic learning algorithms. It would be a diversion from building a game-playing program to study it all, and it does not delve into all elements of game-playing agents in great detail, but it is IMO a very good grounding into the learning parts of any such agent.
In addition, one state-of-the-art self-play learning algorithm is Deep Mind's Alpha Zero. It might be a bit of a steep learning curve to dive straight into how it works, but the algorithm is at its core surprisingly simple and elegant. It may be possible, given some basic knoweldge of terms from RL (so you know what a policy is), to get straight into it based on some tutorials like this one that learns Othello.

Thank you for this answer! I will take some time and try to digest it!

This is a good question. your understanding in general is correct. Indeed, data can be used to construct a proper evaluation of a move/board position and recommended moves based on its history (at least alphago does).
Regarding your first point, it is possible that scenarios that never occurred in historical data could occur, but that's not a problem if your valuation procedure is strong. Typically this is modeled by some continuous function that takes your input board and gives some scalar value. Doing so implicitly assumes you learn the patterns among data pretty well so that ideally you can extrapolate sufficiently in new scenarios. If you had used a lookup table for every board state, you would be screwed in both of the respects you mentioned above.
Regarding your second point, problems related to storing the tree itself and efficiently querying has other solutions that essentially involve a clever pruning method (minimax, and its variants) and/or an efficient cache, like zobrist hashing (see chessprogramming wiki for more techniques).
Now to the core of the question, how do you evaluate and setup a loss function?
There are several ways to do this though: roughly you can think about this as falling into the following buckets

(i) a reinforcement learning approach, which   Alphago uses (https://www.chess.com/article/view/whats-inside-alphazeros-brain) . This is interesting because RL is quite a different paradigm of learning than supervised learning. I'd recommend reading through some of sutton/barto (http://www.incompleteideas.net/book/the-book-2nd.html) to see how questions of policy learning and move recommendations can be derived in general; this is quite a readable book. Then read about alphazero and alphago https://www.science.org/doi/10.1126/science.aar6404.

(ii) supervised learning.
When I initially tried this out, I trained a neural net to output a value at least as good as the moves outputted by a grandmaster (see Should I use neural networks or genetic algorithms to solve Gomoku? for more details). Another way can be based on whether the player won or lost at the end, assuming that in general their moves were better when they won than when they lost (a little strong of an assumption to me personally). See https://www.cs.tau.ac.il/~wolf/papers/deepchess.pdf - They randomly sample moves from a game white won and lost, and learns to choose between the moves with a neural network.

To be explicit, once you setup the appropriate loss or policy optimization problem, you can perform gradient descent as usual. what you are optimizing would of course differ based on if you choose the RL route or the supervised learning route.
Since this is in the first part of your question, I will additionally say that NNs help in chess because of this complex problem of how to valuate a position based on some unique progression and state of the game, and the combinatorial explosion (which is somewhat rectified by a move recommender in alphago, and/or intuitively to me, better evaluating positions may allow for more robust move selection given less plies). Without a NN, some more static evaluator or less complex function approximator may be too simplistic for robustly learning `good moves' in chess.
It is interesting to see that some methods do not require you to actually encode which moves are legal and which are not.


text:
Huge dimensionality of input and output — any recommendations?
At work there is an idea of solving a problem with machine learning. I was assigned the task to have a look at this, since I'm quite good at both mathematics and programming. But I'm new to machine learning.
In the problem a box would be discretized into smaller boxes (e.g. $100 \times 100 \times 100$ or even more), which I will call 'cells'. Input data would then be a boolean for each cell, and output data would be a float for each cell. Thus both input and output have dimensions of order $10^6$ to $10^9$.
Do you have any recommendations about how to do this? I guess that it should be done with a ConvNet since the output depends on relations between close cells.
I have concerns about the huge dimensions, especially as our training data is not at all that large, but at most contains a few thousands of samples.

Motivation
It can be a bit sensitive to reveal information from a company, but since this is a common problem in computational fluid dynamics (CFD) and we already have a good solution, it might not be that sensitive.
The big boxes are virtual wind tunnels, the small boxes ('cells' or voxels) are a discretization of the tunnel. The input tells where a model is located and the output would give information about where the cells of a volume mesh need to be smaller.

Hello. Welcome to Artificial Intelligence Stack Exchange. It might be a good idea to describe your problem at a higher level. What are these boxes? What do the cells represent? What do the floating-point values associated with each cell (I suppose these are your "labels") represent?

Thanks, @nbro. It can be a bit sensitive to reveal information from a company, but since this is a common problem in computational fluid dynamics and we already have a good solution, it might not that sensitive. The big boxes are virtual wind tunnels, the small boxes ('cells' or voxels) are a discretization of the tunnel. The input tells where a model is located and the output would give information about where the cells of a volume mesh need to be smaller.

@md2perpe if the model exists, is it confined exclusively to one cell? If not maybe you could do some generalisation of the inputs/outputs based on whether the model exists within an area of space. Its approximately what the CNN attempts to do (smoothing + sub-sampling) but by doing it manually instead imposing your external knowledge of the system it could help reduce dimensionality exponentially whilst also speeding up learning

@questions. The model will exist; without it the task is meaningless. The model will span a lot of cells, but we're looking for numbers in the cells outside of the model. The model will probably be placed in the middle of the space, with some margin around (like this but in 3 dimensions and with higher resolution).

@md2perpe by "numbers in the cells outside of the model" i assume you mean outputs where the input is 0. If the number of cells the model occupies is deterministic (w/ the margin) maybe there exists some parameterisation that can approximately describe the number of cells which are 1. A motivating idea is how one describes 1D gaussians by their mean and standard deviation instead of trying to find how each individual input in $\mathbb{R}$ is mapped.

@questions. Correct; interesting output is where input equals 0. The number of cells the model occupies is quite deterministic for a given model. It should be approximately proportional to the volume of the model. How do you mean that this could be used? The output is known to depend on the form of the model, but just scale with the size of the model.

@md2perpe How you use that information to define input features is the crux of the problem. For your case one could first try implementing a 3D coordinate system associating each cell with a coordinate and then, for instance, passing the coordinates of the cells that are on the perimeter of the model as the input (this wont work if the perimeter is constantly changing as that causes the size of input changes).  Without much further context it's hard to suggest a truly viable input but this hopefully provides you with an idea of how one could attempt this.

The purpose of these features is to, in a succinct way, be able to uniquely define the "true inputs". There may be various ways to pick feature sets and the results during learning can vary a lot based on this. So you should try come up with different input features and compare the performances

@questions. So your idea is to reduce the number of dimensions of the input by using some representation of the boundary of the model?

yes that was my suggestion, although it may not work in practice because NNs require a fixed size input. But hopefully that provides you with a new perspective on how to reduce dimensionality of your problem - If you feel satisfied with my response I can write this up as an answer

@questions. The number of dimensions of the output is still the same, though. And because of that, I'm not sure if reducing the dimensions of input helps much. But perhaps there is some way to also reduce the number of dimensions of the output.


text:
Are there heuristics that play Klondike Solitaire well?
Are there heuristics that play Klondike Solitaire well?
I know there are some good exhaustive search solvers for Klondike Solitaire. The best one that I know of is Solvitaire (2019) which uses DFS, (see paper, code).
I searched the web for heuristics that plays as a human would play, with no backwards moves, however, I found only one.  In that paper, they report on a win-rate of 13.05%.  In comparison, human experts reach 36.6% win-rate in thoughtful solitaire which is Klondike Solitaire where the location of all the cards is known. Source: Solitaire: Man Versus Machine (2005).
Are there any other published heuristics for Klondike Solitaire?
When determining if a heuristic is interesting, I would consider its win-rate and the similarity to how humans are playing.

Does The complexity of Solitaire interest you?

While this paper is interesting, my interest in Solitaire is from a practical point of view.  I wish to estimate the difficulty of different setups.  One way is by getting the output of a DFS solver and the solution of an ID-DFS optimal-solution solver.  Second way is to examinee the win-rate of heuristics.  The third option is to examinee the win-rate of human.


text:
What is the purpose of hidden nodes in neural network?
If I have a set of sensory nodes taking in information and a set of "action nodes" which determine the behavior of my robot, why do I need hidden nodes between them when I can let all sensory nodes affect all action nodes?
(This is in the context of evolving neural network)

Look for the idea called "Hierarchical representation"

A feed forward neural network without hidden nodes can only find linear decision boundaries. However, most of the time you need non-linear decision boundaries. Hence you need hidden nodes with a non-linear activation function. The more hidden nodes you have, the more data you need to find good parameters, but the more complex decision boundaries you can find.

It appears that the topic is much more complex than I initially thought. What field of study is this in, and is there a name for this specific aspect of neural networking?

@user289661 This is one of the most basic aspects of artificial neural networks. It's in machine learning.

And you generally want more than one hidden layer, because while a sufficiently large network can capture any function to an arbitrary precision, for certain computations a shallow network would need exponentially more nodes than a one layer deeper network that does the same.

@Peteris While I agree that you generally want more than one hidden layer, I wonder if you have a source for the claim that a shallow network needs exponentially more nodes than a one layer deeper network.

@MartinThoma https://arxiv.org/pdf/1603.00988.pdf or https://arxiv.org/pdf/1509.08101.pdf are some sources; the particular proofs of course are different for different types of functions; It generally starts with Hastad 1987 "Computational limitations of small-depth circuits" and in general this concept is the counterpoint to the universal approximator theorem - while yes, even the simplest architectures can implement anything, for certain functions they require an unreasonable amount of resources.

@MartinThoma http://deeplearning.cs.cmu.edu/pdfs/Delalleau_Bengio_2011.pdf can also be relevant.

Normally one node/layer applies linear fitting of the the input to the hypothesis, in other words uses linear function ($y = ax + b$). Adding layers chains liner functions, potentially allowing fitting higher order functions. A great explanation can be found here.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Is known math really enough for AI
As an Electronics & Communication Engineering student I've heard some stories and theories about "The math we have is not enough to complete a thinker-learner AI."
What is the truth? Is humankind waiting for another Newton to make new calculus or another Einstein-Hawking to complete the quantum mechanics?
If so, what exactly do we need? What will we call it?

Maybe.
AI has a long history of encountering mathematical impossibilities and then working around them already. While the individuals who solved these problems don't get as much press as Newton, Einstein, or Hawking, a case could be made that their contributions to human knowledge are on a similar scale. Unfortunately, their results don't relate to physical systems, so they can be harder to explain to the layperson.
Something to keep in mind is that your question assumes the correctness of the "Great Man Theory" of science history, which holds that science advances by the efforts of exceptional people, and that we need to wait around for more such people to appear for science to advance. This view of scientific history is overly simplistic, and probably wrong. For example, most (all?) of Newton's discoveries were likely to have been made by someone else, around the same time, if he didn't make them (see, e.g. Leibnitz, who actually did discover calculus at the same time), so a better view might be of a large community of researchers who gradually develop more advanced models based on each others' work.
To answer your question, I've listed out some past examples of problems that were overcome, and some outstanding problems that might require the development of new mathematical tools to solve properly. Keep in mind though, that we can't know for sure whether new tools are needed: maybe existing tools are sufficient, but no one has applied them in the right way yet!
Some past problems that plagued AI and required the development of better mathematical approaches:

The combinatorial explosion problem appeared to preclude having an AI system reason about probabilities and causation in a logically correct way. This problem was solved by the development of Bayesian networks and causal networks, with the work being led by Judea Pearl and his students. Pearl won a Turing award for this, but has received very little coverage in the popular press.
Many optimization problems that AI systems need to solve are NP-hard. This means that no general purpose algorithms exist that can give exact solutions to these problems in a polynomial number of steps. Because of this, the problems were initially viewed as intractable. The development of PTAS algorithms, and a deep understanding of phase-changes in NP-Hard problems (pioneered by Cheeseman et al.) led to an ability to identify exactly what makes these problems hard, to identify which subparts cause the hardness, and to practical algorithms that solve all sorts of problems in these domains.
Much effort in AI was spent designing new classification methods, and arguing about why one might or might not be better than another. The No Free Lunch Theorems, along with other work in computational learning theory provided a clear mathematical framework for understanding how systems can learn, and where their limits are.
As a more recent example, games involving chance, like Poker, have state spaces so vast that they cannot be searched through effectively, even with heuristics. Phrasing these games in the language of Game Theory, and proving the convergence of the counter-factual minimax regret algorithm provided solutions. Bowling et al. were at the center of this work. It got some press coverage, but most of the coverage focused on "computers can play poker", and not the exiting, more generalizable, lessons AI researchers had learned in the process.

Some problems that are still outstanding, and might require new mathematical techniques:

Are there efficient algorithms for solving problems in the complexity classes PPAD, NP, or #P? Many AI problems fall into these categories. We do not know the answer definitively, and although most researchers suspect that no such algorithms exist, understanding why not seems like it could also provide major research advantages. Existing proof techniques do not seem likely to crack this problem.
We have no good mathematical models of subjective experience. While some researchers (like Churchland) think this is not likely to play a major role in the development of intelligent systems, a concrete model could either support or refute that view, and might provide solid frameworks for current problems in AI like the study of motivation.
The original project of the social sciences was to provide mathematical models of individual humans, and then of human societies. This has mostly been abandoned (Psych is still at it, but Economics prefers to study rational agents instead of humans, and most of the others gave up mathematical modeling entirely to pursue the methodologies of the humanities instead of the sciences). Nonetheless, the lack of sound mathematical descriptions of human behaviours is becoming a major topic within AI, with work on norms, trust, emotion, and other topics. If a mathematical framework were developed to describe the actions of human societies or of individual humans, then AI would be advanced significantly (along with many other fields!).


text:
Concatenating the positional encoding
As per "Attention is all you need" etc., positional encoding is added to the embedded word vector input. My knee-jerk reaction is that this would muddle the "signal" of the word vector. Since word order is not preserved, this additive noise, for instance, could make a word look like a different word (in a different position): $w_a + p_a = x = w_b + p_b$.
Is performance (from shorter input dimension) the main reason to add the positional information versus concatenating, or is adding theoretically sound?
Edit: I think a source of confusion in all of this is the assumption of whether the model adopts a pre-trained word embedder, like Word2Vec, or trains their embedder from scratch. Adding positional encoding to a pre-trained embedder might throw off the attention dot product?

You may want to do some calculations to understand just how little the positional embeddings move the word embeddings to see if that helps you intuitively.

Range from PE would be [-1,1] which I imagine is also the word embed range, no?

I'd say elegance is the main reason in conjunction with the evidence that the addition works well: If you'd concatenate you'd need another dimensionality reducing transformation to make subsequent residual connections work. Also check this related question, I like the orthogonality argument in the first answer, second answer is my own attempt of explanation ;)

How is this question different from Why are embeddings added, not concatenated?" that @Chillston linked above?


text:
Could a large number of interconnected tiny turing-complete computer chips be patterned across a wafer to simulate a programmable neural network?
The Intel 8080 had 4500 transistors and ran at 2-3.125 MHz. By comparison, the 18-core Xeon Haswell-E5 han 5,560,000,000 transistors and can run at 2 GHz. Would it be possible or prudent to simulate a neural network by backing a chip chock-full of a million interconnected, slightly modified intel 8080s (sped up to run at 2 GHz)? If each one modeled 100 neurons you could simulate a neural network with 100 million neurons on a single chip.
Edit: I'm not proposing that you actually use a million intel 8080s; rather I'm proposing that you take a highly minimal programmable chip design like the intel 8080's design and pattern it across a wafer as densely as possible with interconnects so that each instance can function as one or a few dozen fully programmable neurons each with a small amount of memory. I'm not proposing that someone take a million intel 8080s and hook them together.

The building unit of a neural network is called perceptron. It cannot be represented by single transistor because it should hold arbitrary (float) value, over multiple computational iterations. (While the transistor is only binary, and does not work as memory on its own.)
Furthermore, the strengths of the NN is in it's flexibility, which you would lose if you were to bake it on silicon. In a NN you can vary the:

number of layers
connections between units
activation functions
and many, many more meta parameters

The NNs, once trained on a particular problem, are really fast to make a prediction for a new sample. The slow and computationally heavy task is the training - and it's during the training that you need flexibility to mess with the model and the parameters.
You could bake a trained NN model on a chip, if you need the computation time of a prediction to be really fast i.e. in order of nanoseconds (instead of a millisecond or a second on a modern CPU). That will have a significant downside - you won't be able to ever update it with newer NN model.

I'm not proposing that a neural network be represented as a network of individual transistors. I'm proposing that it's represented with a massively parallel architecture based on linking together hundreds of thousands of ulta-light-weight chips capable of universal computation, so that they can be repurposed to simulate any type of neuron or any connection architecture that one might desire.

@JorgePerez it's pretty much what you get with CUDA cores in modern GPU - highly parallel individual computational units. It differs from a full CPU core, as a full core has overhead features. A processor would have individual registers and a way to address the whole memory, while with CUDA you it's assumed you execute the same commutation on all cores, thus the memory is shared.

Theoretically it might be possible but practically it is not.
You can argue by using the analogy of a Turing machine. You can say that the Intel 8080 is a turing machine hence it can run any program including a neural network given infinite time and memory.
Inspite of the above you will face insurmountable challenges in implementing your system.
CPU's are designed to handle calculations in a sequential manner, most AI algorithms are distributed. You need a GPU (or an AI ASIC) to process the algorithms  in a massively parallel manner for a significant speedup.
Additionally GPU's are excellent at floating point math, floating point arithmetic involves numbers with a variable number of decimal places which are key in running neural networks. For example an Intel core i7 6700k is capable of 200 Giga-FLOPs (floating point operations per seconds) while on the other hand an NVidia GTX 1080 GPU is capable of about 8900 Giga-FLOPs which is a significant difference. (Tyler J 2017)
If you decide to use the intel 8080 (0.290 MIPS at 2.000 MHz), you will require millions of processors and billions of dollars just to compute at one gigaflop. You can follow this link to see the cost of computing over the years https://en.wikipedia.org/wiki/FLOPS
Another problem concerns RAM. To efficiently run a neural network you need to fully load it in RAM. It will be a huge challenge to squeeze a neural network in the 64 Kb of RAM that an Intel 8080 processor offers.
The network bandwidth problem will also be a huge bottleneck. Modern GPU's support high speed technology to communicate between the GPU's. For example NVidia's NVLink has a peak speed of around 80 GBps. While  PCI-E 3.0 runs at around 30 GBps. Without a high speed interconnection bandwidth you will not achieve any speedup inspite of using a distributed system with many processors.
Additionally you will face significant challenges in programing neural network algorithms for your 8080 processor based system. Most programmers today follow the standards of object oriented programming which enables code reuse, simplified design and maintanance. Besides, OOP languages such as Java, C++ and Python have libraries that significantly simplify the process of programming a neural network.
When the 8080 processor was designed back in 1974 OOP had not yet been concieved, they were also using programming tools i.e. compilers that would be considered archaic with todays standards. I mean good luck debugging that system.
Last but not least, you need big data (or atleast a substantial dataset) to train your neural network on. Without training on a big data set your model will be ineffective. The 8080 supported around 200 Kb of storage. For comparison the MNIST dataset is around 14 GB in size. This means that your processor cannot support the neccessary storage of any ML dataset.
For the above reasons my conclusion is that the 8080 processor provides insufficient resources necessary to implement any effective DL algorithm. Networking millions of them together will not provide any substantial speedup for a DL algorithm.

It's been done (essentially). This guy at the following link has used a series of FPGAs to emulate hundreds of 8080s, using them to train a neural network to play Gameboy games.
https://towardsdatascience.com/a-gameboy-supercomputer-33a6955a79a4
IBM's True North being used in Darpa's SyNAPSE program is also very close to what you suggest. https://en.wikipedia.org/wiki/TrueNorth
Also of interest may be SpiNNaker and Intel Loihi.

Interesting experiment. Although it demonstrates the architecture is practical and useful for some purposes, it isn't solving the problem that the OP has asked about - making a large NN out of lots of 8080s. A lot of the answers here touch on the problem with that (namely the requirement for shared data, ruling out a set of full processors that communicate), perhaps you could extend this one to capture the difference between OP's proposal and practical deep learning chips.


text:
What are recent AI software systems and research papers close to J. Pitrat's ideas?
J. Pitrat (born in 1934) was a French leading artificial intelligence scientist (the first to get a Ph.D. in France mentioning "artificial intelligence"). His blog is still online and of course refer to most of his papers (e.g. A Step toward an Artificial Artificial Intelligence Scientist, etc.) and books, notably Artificial Beings: the conscience of a conscious machine (his last book). He passed away in October 2019. I attended (and presented a talk) at a seminar in his memory.
What are recent AI systems or research papers related to the idea of symbolic AI, introspection, declarative metaknowledge, meta-learning, meta-rules, etc.?
Most of those I know are more than 20 years old (e.g. Lenat Eurisko; I am aware of OpenCyC). I am interested in papers or systems published after 2010 (perhaps AGI papers with actual complex open source software prototypes).
-see also the RefPerSys system-

Today one of the challenges is learning representations/concepts that are causally invariant. Once we have good representations then we can work on the reasoning aspect. There are 2 camps of people today. One believes that symbolic manipulation cannot be achieved properly by deep networks. Hence, they separate the task of extracting a lower-dimensional representation of objects from the visual scenes from the task of reasoning with knowledge-graphs. The other camp feels that we can do end-to-end training of a neural network and it can learn how to jointly learn a good lower-dimensional representation for each symbol along with learning how to reason with them. I am no expert at this but here are a few papers that I find are worthy for you to read -

Neuro Symbolic Concent Learner Paper, Code
Learning Reasoning Strategies in End-to-End Differentiable Proving Paper, Code
Neuro-Symbolic Visual Reasoning: Disentangling “Visual” from “Reasoning” Paper
Knowledge Infused Learning (K-IL):
Towards Deep Incorporation of Knowledge in Deep Learning Paper
Visual Concept-Metaconcept Learning Paper Project Page
CVPR 2020 workshop on Neuro-Symbolic Visual Reasoning and Program Synthesis youtube videos

If you are looking for community maintained lists, here is one  list of papers


text:
What AI conferences in Europe should I consider submitting papers to explaining the ongoing work on RefPerSys?
https://afia.asso.fr/journee-hommage-j-pitrat/ is a seminar on March 6th, 2020, in Paris (France, European Union), in honor of the late Jacques Pitrat, who advocated during all his professional life a meta-knowledge and reflective approach. (You need to register to attend that seminar).
Pitrat's blog is available (in spring 2020) on http://bootstrappingartificialintelligence.fr/WordPress3/ (and some snapshot of his CAIA system is downloadable here - but no documentation; however you might try to type L EDITE on stdin to caia). He wrote the Artificial Beings : the conscience of a conscious machine book describing the software architecture of, and the motivations for (some previous version of) CAIA. See also this A Step toward an Artificial Artificial Intelligence Scientist paper by J.Pitrat.
What AI conferences (or AGI workshops) in Europe should I consider submitting papers to explaining the ongoing work on RefPerSys?
That RefPerSys project (open source, open science, work-in-progress, with contact information) is explicitly following J.Pitrat's meta-knowledge approach. Feel free to follow or join that ambitious open-source (actually free software) project.

Probably to as many as possible. Average accept rate of papers is around 20%. You can find the best conferences on AI & ML Event.


text:
block of worlds with position aware goal stack planning
Can someone suggest an AI approach to moving blocks, one at a time, assuming control of an robotic arm, to get from the initial state on the left to the final state on right, preferably using goal stack planning.
actions

Pickup() — to pick up a block from table only
Putdown — to putdown a block on table only
Unstack — unstack a block from another block
Stack — stack a block on another clear block only

property functions

On(x,y)
Above(x,y)
Table(x)
Clear(x)

Your best bet would be to formulate the problem in PDDL, which should be fairly easy, and then use a standard planner to generate a plan from that description.
In PDDL you describe the properties and the possible actions, the start state and the goal state, and the planner will then take this to produce a sequence of actions that leads from the start state to the goal state. There is a planner available on-line that you can use.


text:
What AI designs are suited for producing title replacements?
Problem: "For a given news article, generate another title for the article if the article is to be published under a different Publication."
Which algorithm will be well suited for this? Should I use naive Bayesian or any NLP algorithm?


text:
What is a Dynamic Computational Graph?
Frameworks like PyTorch and TensorFlow through TensorFlow Fold support Dynamic Computational Graphs and are receiving attention from data scientists.
However, there seems to be a lack of resource to aid in understanding Dynamic Computational Graphs.
The advantage of Dynamic Computational Graphs appears to include the ability to adapt to a varying quantities in input data.  It seems like there may be automatic selection of the number of layers, the number of neurons in each layer, the activation function, and other NN parameters, depending on each input set instance during the training.  Is this an accurate characterization?
What are the advantages of dynamic models over static models?  Is that why DCGs are receiving much attention?  In summary, what are DCGs and what are the pros and cons their use?

Two Short Answers
The short answer from a theoretical perspective is that ...

A Dynamic Computational Graph is a mutable system represented as a directed graph of data flow between operations.  It can be visualized as shapes containing text connected by arrows, whereby the vertices (shapes) represent operations on the data flowing along the edges (arrows).

Note that such a graph defines dependencies in the data flow but not necessarily the temporal order of the application of operations, which can become ambiguous in the retention of state in vertices or cycles in the graph without an additional mechanism to specify temporal precedence.
The short answer from an applications development perspective is that ...

A Dynamic Computational Graph framework is a system of libraries, interfaces, and components that provide a flexible, programmatic, run time interface that facilitates the construction and modification of systems by connecting a finite but perhaps extensible set of operations.

The PyTorch Framework
PyTorch is the integration of the Torch framework with the Python language and data structuring. Torch competes with Theano, TensorFlow, and other dynamic computational system construction frameworks.

———   Additional Approaches to Understanding   ———

Arbitrary Computational Structures of Arbitrary Discrete Tensors
One of the components that can be used to construct a computational system is an element designed to be interconnected to create neural networks.  The availability of these supports the construction of deep learning and back-propagating neural networks.  A wide variety of other systems involving the assembly of components that work with potentially multidimensional data in arbitrarily defined computational structures can also be constructed.
The data can be scalar values, such as floating-point numbers, integers, or strings, or orthogonal aggregations of these, such as vectors, matrices, cubes, or hyper-cubes.  The operations on the generalization of these data forms are discrete tensors and the structures created from the assembly of tensor operations into working systems are data flows.
Points of Reference for Understanding the Dynamic Computation Concept
Dynamic Computational Graphs are not a particularly new concept, even though the term is relatively new.  The interest in DCGs among computer scientists is not as new as the term Data Scientist.  Nonetheless, the question correctly states that there are few well-written resources available (other than code examples) from which one can learn the overall concept surrounding their emergence and use.
One possible point of reference for beginning to understand DCGs is the Command design pattern which is one of the many design patterns popularized by the proponents of object-oriented design.  The Command design pattern considers operations as computation units the details of which are hidden from the command objects that trigger them.  The Command design pattern is often used in conjunction with the Interpreter design pattern.
In the case of DCGs, the Composite and Facade design patterns are also involved to facilitate the definition of plug-and-play discrete tensor operations that can be assembled together in patterns to form systems.
This particular combination of design patterns to form systems is actually a software abstraction that largely resembles the radical idea that led to the emergence of the Von Neumann architecture, central to most computers today.  Von Neumann's contribution to the emergence of the computer is the idea of permitting arbitrary algorithms containing Boolean logic, arithmetic, and branching to be represented and stored as data -- a program.
Another forerunner of DCGs is expression engines.  Expression engines can be as simple as arithmetic engines and as complex as applications such as Mathematica.  A rules engine is a little like DCGs except that rules engines are declarative and meta-rules for rules engines operate on those declarations.
Programs Manipulating Programs
What these have in common with DCGs is that the flow of data and operations to be applied can be defined at run time.  As with DCGs, some of these software libraries and applications have APIs or other mechanisms to permit operations to be applied to functional details.  It is essentially the idea of a program permitting the manipulation of another program.
Another reference point for understanding this principle at a primitive level is the switch-case statement available in some computer languages.  It is a source code structure whereby the programmer essentially expresses, "We're not sure what must be done, but the value of this variable will tell the real-time execution model what to do from a set of possibilities."
The switch-case statement is an abstraction that extends the idea of deferring the decision as to the direction of computation until run time.  It is the software version of what is done inside the control unit of a contemporary CPU and an extension of the concept of deferring some algorithm details.  A table of functors (function pointers) in C or polymorphism in C++, Java, or Python are other primitive examples.
Dynamic Computation takes the abstraction further.  They defer most if not all of the specification of computations and the relationships between them to run time.   This comprehensive generalization broadens the possibilities of functional modification at run time.
Directed Graph Representation of Computation
That's what the Dynamic Computational model is.  Now for the Graph part.
Once one decides to defer the choice of operations to be performed until run time, a structure is required to hold the operations, their dependency relationships, and perhaps mapping parameters.  Such a representation is more than a syntactic tree (such as a tree representing the hierarchy of source code).  Unlike an assembly language program or machine code, it must be easily and arbitrarily mutable.  It must contain more information than a data flow graph and much more than a memory map.  What must that data structure that specifies the computational structure look like?
Fortunately, any arbitrary, finite, bounded algorithm can be represented as a directed graph of dependencies between specified operations.  In such a graph, the vertices (often represented as nodes of various shapes when displayed) represent operations performed on the data and the edges (often represented as arrows when displayed) are digital representations of information originating resulting from some operation (or system input) and upon which other operations (or system output) depend.
Keep in mind that the directed graph is neither an algorithm (in that a precise sequence of operations is specified) nor a declaration (in that data can be explicitly stored and loops, branches, functions, and modules may be definable and nested).
Most of these Dynamic Computational Graph frameworks and libraries permit the components to do computations on the component input that support machine learning.  Vertices in the directed graph can be simulations of neurons for the construction of a neural net or components that support differential calculus.  These frameworks present possibilities of constructs that can be used for deep learning in a more generalized sense.
In the Context of Computer History
Again, nothing mentioned thus far is new to computer science.  LISP permits computational schematics to be modified by other algorithms.  And generalized input dimensionality and numerocity is built into a number of longstanding plug-and-play interfaces and protocols.  The idea of a framework for learning dates back to the same mid-Twentieth Century period too.
What is new and gaining in popularity is a particular combination of integrated features and the associated set of terminology, an aggregation of existing terminology for each of the features, leading to a wider base for comprehension by those already studying for and working in the software industry.

Contemporary (trendy) flavor of API interfaces
Object orientation
Discrete tensor support
The directed graph abstraction
Interoperability with popular languages and packages that support big data, data mining, machine learning, and statistical analysis
Support for arbitrary and systematic neural network construction
The possibility of dynamic neural network structural adaptation (which facilitates experimentation on neural plasticity)

Many of these frameworks support adaptability to changing input dimensionality (number of dimensions and the range of each).
Similarity to Abstract Symbol Trees in Compilers
A dependency graph of inputs and outputs of operations also appears within abstract symbol trees (AST), which some of the more progressive compilers construct during the interpretation of the source code structure.  The AST is then used to generate assembler instructions or machine instructions in the process of linking with libraries and forming an executable.  The AST is a directed graph that represents the structure of data, operations performed, and the control flow specified by the source code.
The data flow is simply the set of dependencies between operations, which must be inherent in the AST for the AST to be used to create execution instructions in assembler or machine code that precisely follows the algorithm specified in the source code.
Dynamic Computational Graph frameworks, unlike switch-case statements or AST models in compilers, can be manipulated in real-time, optimized, tuned (as in the case of plastic artificial nets), inverted, transformed by tensors, decimated, modified to add or remove entropy, mutated according to a set of rules, or otherwise translated into derivative forms.  They can be stored as files or streams and then retrieved from them.
This is a trivial concept for LISP programmers or those that understand the nature of John von Neumann's recommendation to store operational specifications as data.  In this later sense, a program is a data stream to instruct, through a compiler and operating system, a dynamic computational system implemented in VLSI digital circuitry.
Achieving Adaptable Dimensionality and Numerocity
In the question is the comment that one doesn't, "Need to have data set -- that all the instances within it have the same, a fixed number of inputs." That statement does not promote accurate comprehension.  There are clearer ways to say what is true about input adaptability.
The interface between a DCG and other components of an overall system must be defined, but these interfaces may have dynamic dimensionality or numerocity built into them.  It is a matter of abstraction.
For instance, a discrete tensor object type presents a specific software interface, yet a tensor is a dynamic mathematical concept around which a common interface can be used.  A discrete tensor may be a scalar, a vector, a matrix, a cube, or a hypercube, and the range of dependent variables for each dimension may be variable.
It can be the case that the number of nodes in a layer of the system defined in a Dynamic Computational Graph can be a function of the number of inputs of a particular type, and that too can be a computation deferred to run time.
The framework may be programmed to select layer structure (an extension of the switch-case paradigm again) or calculate parameters defining the structure sizes and depth or activation.  However, these sophisticated features are not what qualifies the framework as a Dynamic Computational Graph framework.
What Qualifies a Framework to Support Dynamic Computational Graphs?
To qualify as a Dynamic Computational Graph framework, the framework must merely support the deferring of the determination of algorithm to run time, therefore opening the door to a plethora of operations on the computational dependencies and data flow at run time.  The basics of the operations deferred must include the specification, manipulation, execution, and storage of the directed graphs that represent systems of operations.
If the specification of the algorithm is NOT deferred until run time but is compiled into the executable designed for a specific operating system with only the traditional flexibility provided by low-level languages such as if-then-else, switch-case, polymorphism, arrays of functors, and variable-length strings, it is considered a static algorithm.
If the operations, the dependencies between them, the data flow, the dimensionality of the data within the flow, and the adaptability of the system to the input numerocity and dimensionality are all variable at run time in a way to create a highly adaptive system, then the algorithm is dynamic in these ways.
Again, LISP programs that operate on LISP programs, rules engines with meta-rule capabilities, expression engines, discrete tensor object libraries, and even relatively simple Command design patterns are all dynamic in some sense, deferring some characteristics to run time.  DCGs are flexible and comprehensive in their capabilities to support arbitrary computational constructs in such a way to create a rich environment for deep learning experimentation and systems implementation.
When to Use Dynamic Computational Graphs
The pros and cons of DCGs are entirely problem-specific.  If you investigate the various dynamic programming concepts above and others that may be closely tied to them in the associated literature, it will become obvious whether you need a Dynamic Computational Graph or not.
In general, if you need to represent an arbitrary and changing model of computation to facilitate the implementation of the deep learning system, mathematical manipulation system, adaptive system, or another flexible and complex software construct that maps to the DCG paradigm well, then a proof of concept using a Dynamic Computational Graph framework is a good first step in defining your software architecture for the problem's solution.
Not all learning software uses DCG's, but they are often a good choice when the systematic and possibly continuous manipulation of an arbitrary computational structure is a run time requirement.

"PyTorch is the integration of the Torch framework for the Python language" – I think that this statement may be misinterpreted as "PyTorch is a wrapper library for the Torch framework for the Python language", which would be a false statement. You should probably rephrase it.

"With Dynamic Computational Graph frameworks, unlike with switch-case statements or intermediate AST models in compilers, one can manipulate the operations" ... "translate them" – What do you mean by "translating operations"?

In short, dynamic computation graphs can solve some problems that static ones cannot, or are inefficient due to not allowing training in batches.
To be more specific, modern neural network training is usually done in batches, i.e. processing more than one data instance at a time. Some researchers choose batch size like 32, 128 while others use batch size larger than 10,000. Single-instance training is usually very slow because it cannot benefit from hardware parallelism.
For example, in Natural Language Processing, researchers want to train neural networks with sentences of different lengths. Using static computation graphs, they would usually have to first do padding, i.e. adding meaningless symbols to the beginning or end of shorter sentences to make all sentences of the same length. This operation complicates the training a lot (e.g. need masking, re-define evaluation metrics, waste a significant amount of computation time on those padded symbols).  With a dynamic computation graph, padding is no longer needed (or only needed within each batch).
A more complicated example would be to (use neural network to) process the sentences based on its parsing trees. Since each sentence has its own parsing tree, they each requires a different computation graph, which means training with a static computation graph can only allow single-instance training. An example similar to this is the Recursive Neural Networks.

Dynamic Computational Graphs are simply modified CGs with a higher level of abstraction. The word 'Dynamic' explains it all: how data flows through the graph depends on the input structure,i.e the DCG structure is mutable and not static. One of its important applications is in NLP neural networks.

Many deep neural networks have a static data-flow graph, which roughly means that the structure of its computation (its computation graph) remains stable over different inputs. This is good since we can leverage this feature for performance, such as by mini-batching (processing a bunch of inputs at once).
But some neural networks could have a different computation graph for each input. This causes some problems (batching difficulties, graph construction is computationally expensive), and hence these networks are a bit hard to use.
The paper you link overcomes this problem by proposing a method that can batch several computation graphs into one. Then, we can do our usual NN techniques.
Benefits are speedups, which incentivizes researchers to explore different structures and be more creative I guess.

The advantage of Dynamic Computational Graphs appears to include the ability to adapt to a varying quantities in input data. It seems like there may be automatic selection of the number of layers, the number of neurons in each layer, the activation function, and other NN parameters, depending on each input set instance during the training. Is this an accurate characterization?

This is incorrect.


text:
Is the neuron adequately comprehended?
It is possible that the signal handling of a neuron is outside the engineering comprehension of the most astute of human brains, even after the relationships of inputs to outputs are statistically characterized and the mapping of genetic information to neuron structure and composition is complete.  Deep comprehension of the detailed function of neurons within their role in adaptive system behavior seems to be lacking, and no rational proof of the feasibility of comprehending neurons has yet been published.
A visual representation of neuron functionality might look like the visual representation of a genome, a blur of information, when viewed in its entirety.  The economy of the world is much more complex system, astronomically more daunting.  Even the map of the metabolic processes of earthen life (shown below) defies comprehensive display in a single view.
Comprehending these complex biological systems can only be done piecemeal.  Studying these systems in their entirety may take a single mind years (or even millennia), and the distortions of memory over time may prohibit placing the whole system within the finite capacity of the cerebral cortex.
Yet computer science has no doubt advanced.  Many features of the mammalian brain have already been simulated, and those simulations exceed the capacities and speeds of mammals.  Examples include mail sorting, flight planning, and game playing.  With such progress, the simulation of the less obviously mechanical functions of the human mind appeared to many to be within the grasp of scientific advancement.  Other AI theorists have expressed their doubts.
The question of whether a system of a few billion elements can be simulated without first simulating a generalization of its elements arose in the twentieth century.  The spiking neuron model is an example of research inspired by that question.
Is the most elemental unit of biological computation, the neuron1, comprehensible by a brain of comprised of neurons?
If the answer is no, which is quite possible, then new approaches to AI may be required to simulate the capabilities of the mind that are less obviously computational in nature – capabilities like intuition, inspiration, compassion, jazz composition, or revolutionary thought.

[1] https://www.ncbi.nlm.nih.gov/pubmedhealth/PMHT0024269/

"Is the most elemental unit of biological computation, the neuron..." citation needed

i was just trying to point out that it's a bit too speculative and broad to make a claim as fundamental as  that. does ionic exchange inside the cell count as computation? also i think Hameroff has some research that counters that claim

No, here is why. No approach can simulate the mind with 100% accuracy. a major notion that AI theorist refuse to note is that you cant take an orange and by virtue of technology turn it into an apple lets apply the same logic here.
neurons are temporary things in our brains, daily we are trimming our brains and growing our brains, in order to "Engineer" a human Brain a probability whatever or not that neuron would die or not would have to be present. But that probability is flawed as well because, Nature is not math. In this case we are trying to apply something to Math that is unpredictable. Math is something we have used to try to rationalize the way nature works but it out right fails. If nature is completely math then predicting a hurricane would be easy.
So to answer your question, no because a Neuron is more then a light switch, it is not mathematical in nature. their is no universality in everyone. plus their is more variables that you would have to account for that are way beyond a simple decision and a simple mathematical formula.


text:
How can I improve this word-prediction AI?
I'm relatively new to AI, and I've tried to create one that "speaks". Here's how it works:
1. Get training data e.g 'Jim ran to the shop to buy candy'
2. The data gets split into overlapping 'chains' of three e.g ['Jim ran to', 'ran to the', 'to the shop', 'the shop to'...]
3. User enters two words
4. Looks through the chains to find if the two words have been seen before.
5. If they have, finds out which word followed it and how many times.
6. Work out the probability e.g: if 'or' followed the two words 3 times, 'because' followed the two words 1 time and 'but' followed it 1 time it would be 0.6, 0.2 and 0.2
7. Generate a random decimal
8. If the random decimal is in the range of the first word (0 - 0.6) pick that one or if it's in the range of the second word (0.6 - 0.8) pick that word or if it's in the range of the third (0.8 - 1) pick that word
9. Output the word picked
10. Repeat from 4 but with the new last two words e.g if the last words had been 'to be' and it picked 'or' the new last two words would be 'be or'.
It does work, but it doesn't stick to a particular topic. For example, after training with 800 random Wikipedia articles:

In the early 1990s the frequency had a plastic pickguard and separate hardtail bridge with the council hoped that the bullet one replaced with the goal of educating the next orders could revert to the north island or string of islands in a new urban zone close to the west.

As you can see the topic changes many times mid-sentence. I thought of increasing the number of words it considered from two to three or four, but I thought it might start simply quoting the articles. If I'm wrong please tell me.
Any help is greatly appreciated. If I haven't explained clearly enough or you have any questions please ask.

Welcome to AI!  (That is very elegant output, imo, nonsensicality aside;)

How come you pick one at "random"? It seems like that be less accurate than just picking the one with the highest decimal

Seems like recurrent neural networks (RNN) should work for your use case. An excellent introduction is available at: The Unreasonable Effectiveness of Recurrent Neural Networks


text:
Existing programs that find out words with same meanings
I'm wondering if these 2 specific programs already exist and if not how hard would it be to write them:

A program that would figure out (by only "reading" large amounts of texts in human language 1 and 2) which words in second language have the same meaning as a word in first language.  You would give for input texts in both languages and for output you would get for every word in first language a list of words in second language that are most similar to it with a probability that they mean the same thing.
A program that would figure out which words have the most similar meaning by analyzing large amounts of texts in one human language.

I'm planning on writing these two programs and it would be nice if I could get existing programs that do this so that I could compare results of my program to those of existing programs.

A point of consideration: words are only one of many forms of structure in a language. That is to say, different languages will construct the context around synonymous words in slightly different ways. Any good model outputting a translation of sorts will almost certainly have to account for this heterogenous word context to some degree. For a great book that covers some interesting ideas around differences in mechanics between language, check out Through The Language Glass by Guy Deutscher.

Can you give an example of what you mean? Yes for example order of words can be different in different languages. But the way one word is related in meaning to other words is same in all languages, because languages live in the same worlds where logic is the same.

That’s a peerless sentence right there! @Bobs Yes, the short of it is that the variation in grammatical constructs and the structure of relationships between words (among other things) need to be understood by your program (this is not covered in the book I mentioned). Given the complexity of what your asking, this program almost certainly has at least one statistical learning model at its core. Also, thinking of synonyms we can see how context is critical: a given synonym in language A undoubtedly does not share the same set of synonyms for its synonym in language B.

For the first answer, the general way is using the seq2seq model:
https://www.tensorflow.org/tutorials/seq2seq
(a more specific example of the above category is this paper:
http://emnlp2014.org/papers/pdf/EMNLP2014179.pdf)
For the 2nd answer, one way is to use word2vec and apply supervised learning on the groups of similar words:
https://towardsdatascience.com/word-to-vectors-natural-language-processing-b253dd0b0817
https://spacy.io/usage/vectors-similarity

Let us simplify case 1 of the question: assume two files, first one with numbers written in indo-arabic numeral system (i.e. 123, 9, 186754, ...), second one in roman numeral system (XX, LCVI, IX, ...). How do you match pairs of symbols with same meaning ?
Without external information or assumptions, you can not. You could made the hypothesis that probability of one specific number is the same in both examples, and base your pairing on it. But then you need to find two input files that fulfills this condition.

It is possible to get this information. Because same words in both languages are used in sentences that have same structure. For example lion and Africa are often used in same sentence, or stars and universe and telescope. But on the other hand stars and walking are less related, you fly to stars not walk, and aeroplane also flies not walks, and people fly on aeroplanes. So there you get these specific relationship between these words that are true in both languages and specific structure of relationships is true only for these specific words. Thats how you can know they must mean the same.

How to compare these structures is not yet entirely clear to me. It is complicated. But you have to use analogies, statistics and logic. Neural networks only use analogies or simple similarity but ignore logic, that is their problem I think.

About the examples you provide: in the pure case, you can know from text1 that there are a statistic relation between "lion" and "Africa", from text2 a relation between "estrellas" and "universo". Is "estrellas" synonym of "lyon" ? You can improve the algorithm starting by some well known fixed matches (the external information I refer in m answer),  i.e 5=V, and, from this one and using statistics, infers new matches like 6=VI.  However, in natural language, all these approach will have a lot of troubles. Nice as theoretical exercise, but easier to use an existing dictionary.

Well the ultimate goals is to write a program that will be able to learn itself new languages just from reading large amounts of texts. That this is possible we know from human babies, who are capable of this.

Remember the case of Egyptian scripts: no translation until rosetta stone (external information) was found.

So you are saying this is very hard program to write? Good to know so I wont feel bad when I fail. I will read about Egyptian scripts, sounds interesting. Maybe my program will be able to decrypt them, it should

You have my best wishes in your target. Yes, when you say "I'm planning on writing these two programs", be prepared to say "I'm planning to research on this issue and implement some test programs". You will be famous if you decrypt Voynich manuscript.


text:
Recommended Modelling Technique for Influencer Marketing Scenario
I have an approximately 90,000 row dataset that has information of social media profiles which has columns for biography, follower count, language spoken, name, username and the label (to identify whether the profile is that of an influencer, brand or news and media).
Task: I have to train a model that predicts the label. I then need to produce a confidence interval for each prediction.
As I have never come across a problem like this, I am just after some suggestions of what models I should be using for a situation like this? I am thinking Natural Language Processing (NLP), but not sure.
Also, for NLP (if a suitable method), any codes or advice to help me implement for the first time on Python would be greatly appreciated! Thanks in advanced

It depends very much on the structure of the data.
I would think about feature extraction first, which could be certain words occurring in the bio, and a class of user name ('real' name, numerical id, etc). Once you have a set of features for each data item, turn them into a list of feature vectors.
Then run them through a number of machine learning algorithms. This is where the shape of the data matters, as some algorithms will work better than others. I would try eg decision trees (ID3), which are very efficient once trained (but they don't give you a confidence interval). But any other ML algorithm might work. They will all have trade-offs with speed of training, memory requirements, and speed of classification; some will give you a class-label probability, others will just give you one label.
The best way would be to use a sample, and identify which algorithm works well and fits your specific requirements. Then use that for the full data set.
Alternatively you could just use, for example, the Stanford ML classifier. That will give you a confidence interval, and will probably work reasonably well.

Thank you so much Oliver - this is really helpful. I will try and implement as you have said

Note that selecting the right features might have a bigger impact on the outcome than the actual ML algorithm, so it's worth trying out several alternatives there as well.


text:
Approaches to an algorithm for crossing a road
I want to write an algorithm which indicates to a robot the first point in time when it is reasonably safe to cross a road.  Assume that the robot's goal is to travel to a location that requires a road crossing and that the robot is ready to cross.
Simple algorithms and decision making will probably not suffice.  What features and capabilities must the algorithm have to provide crossing safety?  What existing AI methods might be useful to consider for this endeavor?
For the first iteration, we can assume the traffic pattern is normal in that no vehicles are driving over the curb and there are no high speed chases or other safety related abnormalities.

Any AI algorithm depends on the environment, and available actuators and sensors. In our case, the environment is a road, street, etc. The primary actuator includes wheels (or legs) of the robot. Sensors include a camera, sonar system, etc.
A simple Model-based reflex algorithm can work in your case:
function MODEL-BASED-REFLEX-AGENT(percept) returns an action
    persistent: state, agents' current conception of the world state
                model, description of how hte next state depends on current state and action
                rules, a set of condition-action rules
                action, most recent action, initially none
    state <-- UPDATE-STATE(state, action, percept, model)
    rule <-- RULE-MATCH(state, rules)
    action <-- rule.ACTION
    return action

Most of the terms and functions are self-explanatory and I will try to explain the important points. This implementation helps the robot to keep track of the external environment by maintaining internal state that depends on the percept history. Updating the internal state requires how the environment works without our agent in it. For e.g., Cars stop at red light and start moving on the green signal. Another thing that is required is how the actions of our agent will affect the world. Since your robot is only trying to cross the road there are not many cases here. Simple things include, stopping motors will stop the bot and starting them will move it forward.
The algorithm above shows how the current percept combined with the old internal state result in generating the updated description of the world, based on agent's model of how the world works. Thus, agent's model of how the world works is the most important part. UPDATE-STATE is responsible for creating new internal state description. The actual implementation of this will depend on the environment and technology that is being used.
The above algorithm has been taken from the book, Artificial Intelligence: A Modern Approach, for agents working in the partially observable environment. Implementation of many functions also depends on how complex you want your robot to be. For e.g., we haven't considered the case of looking up while crossing the road. This is possible that something might be falling from the sky, highly improbable, but possible. The list like this never stops in real life environments.

You can use Imagination-augmented to predict wrong and fatal actions. Deep mind is working on it and I believe it is a great solution to many problems. You need to predict whether the best solution is to take a few steps forward so you do not get hit or go back a few steps.


text:
Are artificial networks based on the perceptron design inherently limiting?
At the time when the basic building blocks of machine learning (the perceptron layer and the convolution kernel) were invented, the model of the neuron in the brain taught at the university level was simplistic.

Back when neurons were still just simple computers that electrically beeped untold bits to each other over cold axon wires, spikes were not seen as the hierarchical synthesis of every activity in the cell down to the molecular scale that we might say they are today. In other words, spikes were just a summary report of inputs to be integrated with the current state, and passed on. In comprehending the intimate relationships of mitochondria to spikes (and other molecular dignitaries like calcium) we might now more broadly interpret them as synced messages that a neuron sends to itself, and by implication its spatially extended inhabitants. Synapses weigh this information heavily but ultimately, but like the electoral college, fold in a heavy dose of local administration to their output. The sizes and positions within the cell to which mitochondria are deployed can not be idealized or anthropomorphized to be those metrics that the neuron decides are best for itself, but rather what is thermodynamically demanded.1

Notice the reference to summing in the first bolded phrase above.  This is the astronomically oversimplified model of biology upon which contemporary machine learning was built.  Of course ML has made progress and produced results.  This question does not dismiss or criticize that but rather widen the ideology of what ML can become via a wider field of thought.
Notice the second two bolded phrases, both of which denote statefulness in the neurons.  We see this in ML first as the parameters that attenuate the signals between arrays of artificial neurons in perceptrons and then, with back-propagation into deeper networks.  We see this again as the trend in ML pushes toward embedded statefulness by integrating with object oriented models, the success of LSTM designs, the interrelationships of GAN designs, and the newer experimental attention based network strategies.
But does the achievement of higher level thought in machines, such as is needed to ...

Fly a passenger jet safely under varying conditions,
Drive a car in the city,
Understand complex verbal instructions,
Study and learn a topic,
Provide thoughtful (not mechanical) responses, or
Write a program to a given specification

... requiring from us a much more radical is the transition in thinking about what an artificial neuron should do?
Scientific research into brain structure, its complex chemistry, and the organelles inside brain neurons have revealed significant complexity.  Performing a vector-matrix multiplication to apply learning parameters to the attenuation of signals between layers of activations is not nearly a simulation of a neuron.  Artificial neurons are not very neuron-like, and the distinction is extreme.
A little study on the current state of the science of brain neuron structure and function reveals the likelihood that it would require a massive cluster of GPUs training for a month just to learn what a single neuron does.

Are artificial networks based on the perceptron design inherently limiting?

References
[1] Fast spiking axons take mitochondria for a ride,
by John Hewitt, Medical Xpress, January 13, 2014,
https://medicalxpress.com/news/2014-01-fast-spiking-axons-mitochondria.html

In my opinion, there are many functions in our brain. Surely much more than  the artificial neural network nowadays. I guess this is the field of brain science or cognitive psychology.
Some brain structures may help for certain applications, but not all. Neural network though is a simplest form of our brain, but has the most general usages. On the other words, if you want to improve the neural networks, different fields or different functions may needs totally different structures. You can refer this as so many types of neural networks nowadays for different applications.

In the perceptron design generally used in Artificial Neural Networks, we know precisely what a single neuron is capable of computing. It can compute a function
$$f(x) = g(w^{\top} x),$$
where $x$ is a vector of inputs (may also be vector of activation levels in previous layer), $w$ is a vector of learned parameters, and $g$ is an activation function. We know that a single node in such an ANN can compute precisely that, and nothing else. This observation could be interpreted as "of course it is limited; it can do precisely his and nothing else".
The universal function approximation theory tells us (very informally here) that if a Neural Network is "big enough", has at least 1 hidden layer, and has non-linear activation functions, it may in theory learn to approximate any function reasonably well. If we add recurrence (i.e. an RNN), we also get, in theory, Turing completeness. Based on this, we could say that they are not particularly limited in theory... but of course there are many complications in practice:

How big is "big enough"?
How do we effectively learn our parameters? (SGD is the most common approach, but can get stuck in local minima; global optimization methods like evolutionary algorithms wouldn't get stuck... but I don't believe that they're famous for being fast either).
etc.

Just the observation that they may not be highly limited in theory of course doesn't mean that there wouldn't be anything else that works better in practice either. I can very well imagine that a more complex model (trying to simulate additional functionality that we also observe in the brain) may be more capable of learning more complex functions more easily.
An important caveat there is that it tends to be the case that more complex function approximators tend to be more difficult to train in practice. We understand very well how to effectively train a linear function approximator. They also typically aren't very data-hungry. The downside is, they can only approximate linear functions.
We also understand quite well how to train, for example, Decision Trees. They're still quite easy models to understand intuitively, they can learn more complicated functions than just linear functions. I'd say we have a worse understanding of how to train them well than linear functions, but still a good understanding.
ANNs as they are used now... it looks like they are more powerful in practice than the two mentioned above, but there's also still more "mystery" surrounding them (in particular the deep variants). We can train them quite well, but we don't understand everything about them as well as we'd like.
Intuitively, I'd expect that trend to continue if we try to imitate the brain more faithfully. I wouldn't be surprised if there exist more powerful things out there, but they'll be more complex to understand, more difficult to train, maybe also more data-hungry (current ANNs already tend to be very data-hungry).

Great point about difficulty of training more complex function approximators!  No free lunches, here in the sense of greater complexity comes with costs.