text:
Comments on my proposed "Jitter" neuron
I have an application of neural networks (standard MLP architecture) where I want to forecast a tanh output (ranging from -1 to +1) with about 1500 input features in ~700 samples.
Each sample represents a snapshot of database tables at a given time of day - f.ex. at 11:50.
Since I have so few samples, the network is VERY sensitive to overfitting. Although it seems overkill to use neural networks, I find them to perform better than my initial experiments with other approaches, because I can very precisely tune their objective function and regularization, along with a nice possibility for multinomial regression.
A severe problem in the task at hand is that the data in the tables may be adjusted over time. So the database might say that a given feature was last changed at 2016-01-01 11:45, but in reality, a slight change was made a 2016-01-15 07:38. In practice, this means that the training data is unrealistically precise, and that I may instead want to consider it a ballpark estimate.
For example, a value of 153.18 may instead be considered a value of ~140-165.
To alleviate the problem, I have theorized a "jitter" neuron that - during training time, at each iteration - "jitters" the input by subtracting or adding a random value from the feature. The random value to add or subtract is a ratio of the standard deviation of the feature, and the ratio is itself random.
The layer is placed right after the input neurons and has as many neurons as there are input features. In my example, the layer is therefore 1500 neurons wide.
For example, consider an input feature that - across all training samples - has a standard deviation of +- 100.
At training time, a data sample is loaded with the feature value of 1122 (for the sake of example, lets ignore input standardization), and the jitter neuron will incur a randomized change in its value.
We define a "jitter" ratio of 0.1, meaning that the ratio to "jitter" the feature with is drawn from a standard distribution with a standard deviation of 0.1 and a mean of 0. In the example, a random roll of the dice lands us in +1.5 standard deviations, outputting a jitter ratio of 0.15.
Given that the training set has a standard deviation of 100, we add to the input feature (that has a value of 1122) a final jitter of +15, resulting in an input feature value of 1137.
The same operation goes for all input features. Pragmatically, I do this in keras by generating a jitter-ratio-matrix with mean=0 and std=0.1. The std-value can be an arbitrary amount, but we must not jitter the input data too much.
The intuitive justification is that it is representative of my real world scenario. Without going into too many details, the input features are typically things such as weather, whose forecasts are naturally unstable, and any changes to the database values backwards in time is likely to be a "typical", small adjustment.
On a more theoretical level, the justification is that it prevents overfitting to specific input features in some samples, as the next iteration across the same data sample will output significantly different output values after the forward pass through the network, as a slight change in the input feature may have a profound impact on the output after the feature has undergone compounding, non-linear activations.
Additionally, I perceive the jitter-neuron to be a form of data-augmentation, like it is done for especialle image classification; instead of the architect generating n augmented samples according to some heuristic (for example, in images, it is normal to crop and rotate the same image in several different ways to enlarge the training set), the jitter neuron generates a theorhetically infinite amount of augmentations at runtime.
I keep imagining the parameter hyperspace for the neuron weights in the first layer; instead of having many potential pits (i.e. areas of overfitting), the jitter smooths out these pits, as the jittered input data now generates a different loss value.
Early experiments do not provide very good results, but not bad either. They just, sort of... Stay the same... However, my dataset is rather small, and it is notoriously difficult to fit with any model.
Therefore, I ask of you what you think of the concept of jitter neurons:

Does it seem sensible?
Any theoretical reason that it should/should not work?
Is it, for some statistical reason or other, inherently inferior to dropout? (notice that I also add dropout, albeit smaller rates)
Any proposals for making it better?
Comments, etc.

Well, adding gaussian noise is a very common regularisation method.
Maybe this paper is interesting to you. They also have very small datasets.
In the end there is only so much you can get out of a given dataset.


text:
Has Tic-Tac-Toe been solved with a single heuristic?
By this I mean a single axiom for placement.  (I'm working on a solution that involves positional valuation and vectors, and appears to be solid, but my assumption would be this has been done previously, like most mathematical techniques one "discovers";)

Why would you assume?

@quintumnia because there are a lot of smart people, and, in general, whenever I'm thinking through a problem from first principles, my conclusion almost always reflect what has already been proven or defined.  While it's true that per M games I might be thinking about matricies in a new way (it appears that no one has thought about Sudoku the way I am, which has led to some pretty exciting results) but Tic-tac-toe has been very thoroughly analyzed.

What do you mean by question? I'm sure you already know of minimax, does that count? Tic-Tac-Toe is also simple enough to store the entire possible game tree in memory

@k.c.'aktual'ly I guess it depends on how many rules are being applied in conjunction with the minimax algorithm.  (For context, I'm currently working on axiomatic intelligence, and the solution I'm envisioning involves no lookahead/gametrees.)

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
Markov Model for a Traffic Intersection
I need some help in developing a Markov Model for a crossroads there is no one way road and i am assuming at this time that traffic is only allowed to go straight no turns are allowed. There are 4 roads and 4 signals(agents).

What are you trying to observe? Markov models are powerful for making observation from a hidden variable. What are you starting states and what is the injected information?

I am trying to find a optimal policy for reducing the waiting time of traffic on signal. and about the sates I am confused i do not know how to start and proceed with multiple agents.

If you draw your intersection maybe we can get a better idea of what you mean.
Is it a requirement to have multiple agents? If you can restrict that the all of the traffic lights are never set in such a way as to cause accidents (all green), it should reduce to a smaller state space and be easier to consider.

As far i am able to think, you need traffic data for all roads, Based on that you can identify congestion time/traffic using MM/HMM/SSM. Based on traffic at each road at same time, you can optimise the waiting time.

it is ready now you can try it @ https://github.com/aawadall/SmartTrafficIntersection it is text based though, you can extend it to have a graphical component to it, or if you want you can port it to whatever language you prefer

Why not use a single agent to control the intersection with the following rules:

define 8 traffic lights
Each light has two possible values, 0 or 1, with 0 = Red and 1 = Green
Lights are as follows:

0 = North to South
1 = North to East (Left Turn)
2 = East to West
3 = East to South (Left Turn)
4 = South to North
5 = South to West (Left Turn)
6 = West to East
7 = West to North (Left Turn)

map lights into a bitmap of 8 bits, with the least significant bit (left bit) is the North to south

Given the above, we have the following valid moves, which are both State Space  and Action Space :

State/Move 0 - Vertical Moves only   = 10001000
State/Move 1 - Horizontal Moves only = 00100010
State/Move 2 - Originated from North only = 11000000
State/Move 3 - Originated from East only = 00110000
State/Move 4 - Originated from South only = 00001100
State/Move 5 - Originated from West only = 00000011
State/Move 6 - Vertical Diagonal only = 01000100
State/Move 7 - Horizontal Diagonal only = 00010001

i.e. S = {10001000, 10001000, 00100010, 11000000, 00110000, 00001100, 00000011, 01000100, 00010001}
and  A = {10001000, 10001000, 00100010, 11000000, 00110000, 00001100, 00000011, 01000100, 00010001}

Now we have a ∈ A and s ∈ S and at any given s, a will cause the next state s' to have the same value of a
Final component to complete the moel, is the reward signal, which can be simply the negative sum of time spent per car in the intersection
i.e.     rt+1 = -∑ci ∈ Cars time(ci)
a single agent, will be able to optimize the solution to avoid punishment (negative value caused by wait time)
Update
You can increase the state space to double or triple its size to accommodate for time, by appending a time counter component to the bitmap.
in addition to an extra (9th) action of do nothing,
This should give a more subtle traffic controller. e.g. Vertical Traffic Only, at timer = 0 can be represented by 100010000000 or in hex 880, then at time 1 = 881, and so on until time 15 to be 88F.
Update 2
You can find the source code here and here

By the way, I started implementing this solution in C#, if you are interested, I can send you an invite to its git repository

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Can reinforcement learning algorithms be applied to computer vision problems?
Can reinforcement learning algorithms be applied to computer vision problems? If yes, what are some examples of these applications?

Yes, there are a couple of ways to apply reinforcement learning in computer vision problems. This mainly employs the principle of "applying the algorithm -> evaluating the outcome -> adopting the best outcome".
The following are a couple of examples that use reinforcement learning in computer vision.

CAD2RL: Real Single-Image Flight Without a Single
Real Image.
End-to-End Training of Deep Visuomotor
Policies

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Build Android News Recommendation app
Please I am interested in building a news recommendation app that will be powered by artificial intelligence. I have being making a whole lot of research on this and will be glad if somebody can tell what is needed to build such app in Android.
It is something similar to this app

Welcome to AI!  If I understand your question, you are looking for advice on what type of method would be suitable for this problem?  The question might benefit from a title such as "AI methods for News Recommendation App? (Android OS)"  Also, you should probably indicate whether all functions will be local to the mobile device, or if it will be able to take advantage of networked processing and storage (databases).

Actually I am looking for advice and will be glad to get one.  Meanwhile, I am open to whichever methods that work either locally on the phone or through network processes.  Definitely, I would want to host my database in a server somewhere else, not on the phone

Let's get serious here.and those upvotes should tell us why?


text:
Machine learning algorithm for xml manipulation
Given a virtual game map (picture) and a racing car at the map's starting point, I'm trying to build an algorithm that would help me generate a route that would get the car from the beginning to the endpoint.
route definition: a very complex .xml file that includes all the data the car needs in order to navigate the road successfully.
What I have:

I have thousands of different maps and I could get any number of maps I'd like.
For every map, I have a complex external algorithm using picture analysis that builds a route for me - although this route is not so accurate.
For some percentage of the maps, I also have a specific "good working car solution" - which is basically a manually built route file.

What I'm trying to achieve:
Basically, I'm trying to improve the complex route generator algorithm which I don't have access to.
I want to use the percentage of manual routes I have and compare them to the routes that are generated for the same maps. From the comparison and the differences between the manual and the automatically created route files, I want to build a second-step smart algorithm that "learns" what should be changed in the automatically generated routes and have that algorithm run as a second step and make it as close as possible to the handmade route.
My final goal is to be able to build an accurate route for maps I don't have the manual route for. I want to be able to produce an accurate route for every map - using both the external algorithm and the one I will build on top of it.
This entire subject is very new to me and I'd like to get your advice as to how should I approach this complex problem, and what could make this task easier for me.

Terminology
There are two uses of the word map in this discussion.

Road maps are construed below as images of road maps.
Mapping input to desired output is the skill the system must learn.

The set of examples used to teach the system from an existing mapping of input to output is called a labeled data set and the associated type of learning is called supervised.
Training Resources

Unreliable labeled road map examples and access to more also unreliable labeled road map examples
A large number of unlabeled road map examples and access to more road maps
A smaller number of labeled road map examples

Inputs

Virtual game map as an image
Starting position
Target ending position

Output

The fastest route in sufficient detail to fully direct car movement — Assuming that the route should be optimized for soonest arrival time, not for fuel conservation, minimal tire wear, safety, or minimal distance, since the car was identified as a race car.

Accurate Valuation of the Resource Inventory
The low quality routes from the black box service provide neither examples of desired system behavior nor examples undesirable system behavior.  The former case would be good training data.  The later would be good to use in an adversarial architecture based on the design of GANs and their variants.  The uncertain quality of the labeled data from the black box service makes it ambiguous and, from an information science point of view, of zero value.
Comparing the low quality routes and the high quality routes may be interesting, but not particularly useful given the current state of machine learning.  The objective of your initial project phase is to teach the machine to generate a high quality route from the low quality route, not produce a comparison report.  To do that without processing the images a substantially large overlap between these two sets would be required.

The smaller number of labeled road map examples for which the labels were created manually
Corresponding labeled road map examples for which the labels were created by the black box service

That approach would require you to correct bad routes to create a sufficient training set.  Without overlap, you have no training data for artificial network training.  It may be possible to create an algorithm of GAN style that learns how to correct the black box service output using a concept called cross-entropy, but it would require processing the images and would likely be more difficult to do so than to replace the black box service altogether.
If your ultimate goal is to create a working system that generates routes from images and start and and positions, I suggest discounting the external service altogether, and discard the idea of creating a route improver sub-system.  That you wish to improve upon the black box route generator's algorithm is noble but ultimately a time-consuming distraction to reaching the ultimate goal.
Replace the unreliable thing with the reliable thing you design and build and to which you will have full and open access to improve.  Your system will ultimately have to deal with the images in your code either way, and that's the most challenging aspect of the overall task.  Just learn about CNNs and RNNs and get right to it.  That's my advise anyway.
A Note on Feedback
For any non-trivial route, the car will not likely stay on the road if given only turn, acceleration, and deceleration instructions.  Detection of the edges of the road would normally be needed as a feedback mechanism to accompany the route instructions.  The only way around this is to make the visualization sufficiently accurate that cumulative errors never exceed the distance that would put the car off pavement.
A Note on Data Representation
JSON has a more flexible and terse structure when it comes to homogeneous arrays than XML, and it is easy to convert XML to JSON.  Furthermore, a transportation route is often represented as a directed graph, and there are many algorithms already conceived in graph theory that have been implemented in graph libraries in most languages.  For instance, shortest path, path equivalence, path concatenation, and detection of rings (driving in circles) are one line calls to these libraries.  Because JSON, for the reason given, has overtaken XML in many domains, the number of graph libraries that can read and write JSON directly has overtaken the number that can read or write XML directly.  The tooling for JSON analysis and visualization has surpassed that of XML at this point too.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is the difference between a normal processor and a processor designed for AI?
What is the difference between a normal processor and a processor designed for AI?

There have indeed been very recent reports of AI-oriented processors being integrated into smartphones, such as this Forbes article from 9/28/17.

Parallel processing is very important when computing ANNs. Though they are not designed for AIs, GPUs are widely used in machine learning because of their higher capability of paralel processing and it is due to their higher number of cores comparing CPUs.
All in all, the dedicated hardware should have GPU-like architecture.

Specialized AI hardware takes advantage of highly parallelizable nature of many neural network designs. GPUs are designed for pixel crunching - coincidentally very paralelisable too. This is why they often offer orders of magnitude better performance (in NN tasks) than CPUs. That being said, GPUs have shortcomings to. First, many of GPU's built in features are not utilized in neural network applications- in many cases accurate floating point operations are not necessary, data for simple calculations involved in NNs. If you remove all the unused features from GPUs and use that extra space for more compute units, you get something like Googles' https://cloud.google.com/blog/big-data/2017/05/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu  TPU.

Some of the newer Nvidia GPUs have half precision operations(fp16) supported also, which allows code to use less memory, and less compute time, but as you mentioned, googles TPU still does have more compute cores because it does not need to support full precision or double precision.


text:
Perceptron learning algorithm: different accuracies for different training methods
So, my question is a bit theoretical. I have been trying to implement a perceptron based classifier with outputs 1 and 0 depending on  the category. I have used 2 methods: The example by Example learning method and Batch learning method. I also have defined another method which will measure accuracy according to the formulae number_of_samples_classified_correctly/total_number_of_samples(I'm not sure this should be the correct definition for accuracy and you are welcome to suggest a better measure). Now there are a few confusions i'm facing. Firstly, the accuracy of example by example learning is different from batch learning by 2%. Also the best accuracy achieved in both cases is depending on the slopes. So where exactly is the mistake?(Batch learning algorithm=error*input_vector( where error can be 1,-1 or 0 ) summed over all input vectors and then added to weights).

For initial slope[1,-1] giving an accuracy of 88% example by example learning
For initial slope[1,-1] giving an accuracy of 88% batch learning
For initial slope[1,1] giving an accuracy of 84% example by example learning
For initial slope[1,1] giving an accuracy of 86% batch learning

In Brief:
re-train your dataset. I believe where you get lower accuracy scores, your model has not converged to the final state. duplicate your dataset multiple times and create a bigger one, then train your model with it.
In Detail:

number_of_samples_classified_correctly/total_number_of_samples(I'm not sure this should be the correct definition for accuracy and you are welcome to suggest a better measure)

This is a valid accuracy metric. In fact if the value is acc, then 1-acc is called misclassification error. So your metric is good unless you have some class imbalance, where you need to use other metrics such as Cohen's Kappa score.

the accuracy of example by example learning is different from batch learning by 2%. Also the best accuracy achieved in both cases is depending on the slopes.

I strongly believe both strange results happen because the number of your instances (i.e. examples) is low, or let's say your learning rate is small. As you know Perceptron algorithm corrects the weights of the decision hyper-plane by delta learning rule: it reads each instance, calculates the error (in case of binary classification {-1,0,1}) and updates the weights by c.x.E where:

c is learning constant/rate/step
x is the data instance
E is the error

Thus the weights (or slopes) change a little bit every time. There is no guarantee that your model reaches its final states after all the instances are given to it. Since when you start training with different initial weights, or change the order of instances given to your model, the distance or from or moving speed toward the final state changes. So as I mentioned above, re-train your model and let it converge to the final state OR increase your learning step (learning rate) . I believe all the 4 mentioned-above-accuracies will be the same then; I mean not only you should have the same model for different initial weights, but also for batch and online training.
Please update your post with new findings and question my claims if those were false.

Actually i compensated the low learning rate by a large number of epochs..with 100000 epochs for learning rate=0.001...also i took the best accuracy among the weights and plotted it..also somehow the accuracy measure is not working because for batch learning the final classifying line is going above both the data sets i.e for both the data sets value is negative  for all training vectors yet the accuracy is still 88%....also i am not strictly using delta rule for batch learning, i am taking the error to be either [1 ,0, -1]..maybe now you understand where the problem lies?

Also as for higher learning rates it is causing overflow above a learning rate of 0.3 with very high weights...basically it is having an amplifying effect

@DuttaA higher learning rate will surely result in amplifying effect. increasing training epochs instead of increasing learning rate was a good idea...  plus, why are you including [0] class? if you are using one decision boundary, then it means your dataset must have had two classes. for a single instance of dataset, being located on the decision boundary doesn't mean "0" label beside 1 and -1. if your dataset has 3 classes, use multiple classification approaches (either OvO or OvA) which means you have more than one decision boundary. maybe the odd results are caused by this interpretation.

my dataset only has 2 classes...actually i am new to this and i am studying from a book which has used 1 and 0 as to whether in class 1 or class 2 and defined error as target-classification result hence the error might be 1, 0 or -1.

After running the code few more times i am thinking the problem lies in the discrete nature of error. In batch learning we are deciding the error as [0, 1, -1] depending on the input and the input whereas in the measuring of accuracy we are not making consideration of how big the error is i.e. whether the line is classifying the 2 classes by a very small margin (just classifying it) or classifying it by a good margin (comfortably classifying it)..both will give the same error since we are not considering the distance of the points from the line. Also, on choosing the initial weights we are selecting a point in the error vs weight function (which is not continuous) and if our choice of weights are inappropriate, the classifier weights is either getting trapped in a local minima or taking a very large number of epochs to reach the best weight (there is a very little probability that the classifier weights hit the right value after which it is easily able to achieve global minima). This is validated by the fact that on choosing a suitable weight we are getting the best accuracy possible within very few epochs but on choosing a different weight it doesn't reach the best accuracy but comes close to it after a large number of epochs.
As for the example by example learning method, in this type of training the initial weights will determine the local minima the classifier weights get trapped in and its impossible to get out of it even with a large number of epochs.
The 2 ways i think to solve the problem is by either using the Delta rule or by reinforcement learning (although i am not sure whether it can be applied to such discrete error functions) where the algorithm can see that even if it doesn't get an immediate gain by moving in a direction, but finally get a  much more gain(thus will be able to get out of local minima) i.e exploitation vs exploration.
Any further insights, suggestions or corrections are welcome.


text:
What are some concrete steps to deal with the vanishing gradient problem?
I am training an ANN for classification between 3 classes. The ANN has an input layer, one hidden layer and a 3 node output layer.
The problem I am facing is that the output being produced by the 3 output nodes are so close to 1 (for the first few iterations at least, and so I am assuming the problem propagates to future outputs as well) the weights are not being updated (or hardly updated) due to overflow (about $10^{-11}$). I can fix the overflow problem (but I don't think it is the culprit). I think such low values of error is the main culprit, and I cannot figure what is causing such low values of error.
What will cause the network to behave more responsively, that is, how will I be actually able to grasp the weight updates and not something in the order of $10^{-11}$?
The data set contain values in the order of $10$'s, and the weights randomly initialized are in the order of $0 < w < 1$. I have tried feature normalization but it is not that effective.

There is not single answer to the vanishing gradient problem. However, there a few things that can help.
As mentioned in the comments, use of Rectified Linear Units (ReLU) as your activation function can help, since the it does not get saturated for large neuron inputs.
Next, careful choice of weight initialization can help avoid saturation, as well. See Andre Ng's Coursera video for details.
Finally, if you are concerned that the scale of your training input is causing issues with training, you can normalize the training examples by subtracting the mean and dividing by the standard deviation. There is a normality assumption here, but this can often help avoid problems where one feature is far out of scale with another. However, this usually causes issues with optimization bouncing between the walls of long, thin trough, which complicates convergence, but does not cause vanishing gradient.


text:
How to know what kind of memory is stored in the connection weights?
So, I have seen few pictures re-created by a Neural Network or some other Machine Learning algorithm after it has been trained over a data set.
How, exactly is this done? How are the weights converted back into a picture or a memory which a Neural Net is holding?
A real life example would be when we close our eyes we can easily visualize things we have seen. Based on that we can classify things we see. Now in a Neural Net classification part is easily done, but what about the visualization part? What does the Neural Net see when it closes its eyes? And how to represent it for human understanding?
For example a deep net generated this picture:

SOURCE: Deep nets generating stuff
There can be many other things generated. But the question is how exactly is this done?

You should use google for this question, it is extremely vague. Some technical documents will give very clear insight into what's happening here.
https://blog.openai.com/generative-models/
http://proceedings.mlr.press/v37/gregor15.pdf
If you really want to understand "what the AI is thinking", well, you may never know. The idea of AI is that it can handle complex data (high degree of dimensionality) and is too complex for Humans to comprehend.

Actually I was asking how to back calculate in a nn to represent the memory of the object it is classifying... Image is just a specific case

A "normal" neural network can give you a distribution of probabilities for categories given some image. (Ex: a picture of a dog might return 50% dog; 30% cat; 10% car; 5% laptop; 5% skeleton).
What we then do, is maximize the probability of a given category by manipulating the image directly. We do this iteratively, until we are satisfied by the result.
For example, one might start with a grey image, tweak some random pixels to improve a specific category probability of the image, save our results, and repeat, until we have an image that responds very well to a given probability.
So to answer your question, the weights aren't directly manipulated to produce the image, though they are implicitly used in the construction.

I don't get it....As far as I know for simple NN's input is not manipulated? What am i missing?

Yes, for simple NNs the input is not manipulated. However to generate the images such as the one you posted, we alter the image to converge towards a high likelihood for a given class.


text:
What makes the animal brain so special?
Whenever I read any book about neural networks or machine learning, their introductory chapter says that we haven't been able to replicate the brain's power due to its massive parallelism.
Now, in modern times, transistors have been reduced to the size of nanometers, much smaller than the nerve cell. Also, we can easily build very large supercomputers.

Computers have much larger memories than brains.
Computes can communicate faster than brains (clock pulse in nanoseconds).
Computers can be of arbitrarily large size.

So, my question is: why cannot we replicate the brain's parallelism if not its information processing ability (since the brain is still not well understood) even with such advanced technology? What exactly is the obstacle we are facing?

When you say "Whenever I read any book about neural networks or machine learning", to have some context, it would be nice if you can provide 1-2 references, i.e. which books claim that? Moreover, the question in the title is substantially different and more broad than the question in the body "why cannot we replicate the brain's parallelism if not its information processing ability (since the brain is still not well understood) even with such advanced technology?". It seems to be that your question is the second one, so I suggest that you put a more specific question in the title.

Generally, even the question in the body is a bit vague, so you got somehow vague answers (as can be proved from the discussions under this answer). In any case, I think to avoid further misunderstandings, you should follow my suggestions.

One probable hardware limiting factor is internal bandwidth.  A human brain has $10^{15}$ synapses.  Even if each is only exchanging a few bits of information per second, that's on the order of $10^{15}$ bytes/sec internal bandwidth.  A fast GPU (like those used to train neural networks) might approach $10^{11}$ bytes/sec of internal bandwidth.  You could have 10,000 of these together to get something close to the total internal bandwidth of the human brain, but the interconnects between the nodes would be relatively slow, and would bottleneck the flow of information between different parts of the "brain."
Another limitation might be raw processing power.  A modern GPU has maybe 5,000 math units.  Each unit has a cycle time of ~1 ns, and might require ~1000 cycles to do the equivalent processing work one neuron does in ~1/10 second (this value is totally pulled from the air; we don't really know the most efficient way to match brain processing in silicon).  So, a single GPU might be able to match $5 \times 10^8$ neurons in real-time.  You would optimally need 200 of them to match the processing power of the brain.
This back-of-the-envelope calculation shows that internal bandwidth is probably a more severe constraint.

Comments are not for extended discussion; this conversation has been moved to chat.

This has been my field of research. I've seen the previous answers that suggest that we don't have sufficient computational power, but this is not entirely true.
The computational estimate for the human brain ranges from 10 petaFLOPS ($1 \times 10^{16}$) to 1 exaFLOPS ($1 \times 10^{18}$). Let's use the most conservative number. The TaihuLight can do 90 petaFLOPS which is $9 \times 10^{16}$.
We see that the human brain is perhaps 11x more powerful. So, if the computational theory of mind were true, then TaiHuLight should be able to match the reasoning ability of an animal about 1/11th as intelligent.
If we look at a neural cortex list, the squirrel monkey has about 1/12th the number of neurons in its cerebral cortex as a human. With AI, we cannot match the reasoning ability of a squirrel monkey.
A dog has about 1/30th the number of neurons. With AI, we cannot match the reasoning ability of a dog.
A brown rat has about 1/500th the number of neurons. With AI, we cannot match the reasoning ability of a rat.
This gets us down to 2 petaFLOPS or 2,000 teraFLOPS. There are 67 supercomputers worldwide that should be capable of matching this.
A mouse has half the number of neurons as a brown rat. There are 190 supercomputers that should be able to match its reasoning ability.
A frog or non-schooling fish is about 1/5th of this. All of the top 500 supercomputers are 2.5x as powerful as this. Yet, none is capable of matching these animals.

What exactly is the obstacle we are facing?

The problem is that a cognitive system cannot be defined using only Church-Turing. AI should be capable of matching non-cognitive animals like arthropods, roundworms, and flatworms but not larger fish or most reptiles.
I guess I need to give more concrete examples. The NEST system has demonstrated 1 second of operation of 520 million neurons and 5.8 trillion synapses in 5.2 minutes on the 5 petaFLOPS BlueGene/Q. The current thinking is that, if they could scale the system by 200 to an exaFLOPS, then they could simulate the human cerebral cortex at the same 1/300th normal speed. This might sound reasonable, but it doesn't actually make sense.
A mouse has 1/1000th as many neurons as a human cortex. So this same system should be capable today of simulating a mouse brain at 1/60th normal speed. So, why aren't they doing it?

You nicely showed us that a most animals are much smarter than most supercomputers, but I'd like them see playing chess against my oldish android device. So, why aren't they doing it? I guess, the answer is the same as to your question: That can't (yet). +++ I don't know if a cognitive system is more than what AI will ever be able to achieve, but I'm sure, we're not that far yet. +++ Humans vs. computers boasting with exaFLOPs is like birds vs. fish boasting with flying. Or do you claim you can multiply ten orders of magnitude faster than my pocket calculator?

@maaartinus You don't understand. Artificial intelligence has zero comprehension. It doesn't have actual learning since learning is impossible without comprehension or understanding. Yes, AI and robotic enthusiasts want to pretend that these things don't matter, but they do. The real goal of AI is AGI and this hasn't changed since 1956. If you think your calculator can do AGI, well ... good luck.

Short answer: nobody knows. Long answer: all strong-AI works. However, to write something useful to the o.p., say that the question contains several implicit statements, analyze them could be useful to clarify the issue:
a) why thing that 1 transistor has the same functionality than 1 neuron ? Some obvious differences: a transistor has 3 legs, each neuron has around 7000 synapses; a transistor has 3 layers of material, a neuron is a full micro-machine with thousands of components; each synapses itself is a switch, connected to one or more other cells, and can produce different kinds of signals (activation/inhibitory, frequencies, amplitude, ...).
b) compare memory amounts: the amount of memory in a person equivalent to the one in a computer s 0 bytes, we are not able to remember any thing forever and without distortion. Human memory is symbolic, temporal, associative, influenced by body and feelings, ... . Something totally different than computers one.
c) all previous are about "hardware": if we analyze software and training, differences are even bigger. Even assume than intelligence is placed only in the brain, forgetting the role of hormonal system, senses, ... is a simplification not yet proof.
In conclusion: human mind is totally different from a computer, we are far to understand it, and more far to replicate it.
From the start of computers age, the idea than intelligence will appear when the amounts of memory, process power, ... reaches some threshold has became false.

You didn't get my question, I asked about brain's power not how the brain functions (i know we dont understand it clearly). Your 1st point, normal transistor is much older nowadays we use FET's which is smaller. A human neuron is a few centimeters long, millions of fet's can be fitted in those dimensions easily replicating a neuron. 2nd point it is advantageous to have distortion less memory and in much greater amt which an animal doesn't possess. And third I am only asking the power that is i9/i7/in processor not how you are using the processor..But overall i get what you are trying to say

Still same mistake: you want to compare "brain power", undefined term. The "brain power" of a computer is 0, the "computer power" of brain is also 0

I actually am not comparing brain with computer i am asking where is the problem in recreating a brain

Recreating each brain atom, each brain protein, each gen,  each synapse, each neuron ? Modeling them as ?  I will give you an example: C. Elegans is an small worm with only 300 neurons and 7000 synapses, already totally mapped and listes, but nobody as yet reproduced nor simulated it.

I don't get this claim "Long answer: all strong-AI works". Can you explain what you mean by that?


text:
Motivation that drives a rogue AI agent
This is a kind of biological and philosophical question. So, the recent concern in AI is that an AI agent may go rogue with prominent people voicing their concerns.
Now say, we have created an AI (you are free to use your own definition of what makes an AI intelligent) which has gone rogue with powers given in this question.
Now, the broad view of today's biology is that everything we do is to further our genes down the future (leaving aside small technical details). It is even widely accepted that we are just machines whose controller are the genes. Everything we do is controlled/hardwired by the genes with some avenue of learning from experiences. Also genes only further their own interest. Scientist George Price even wrote a mathematical equation proving all our acts are selfish and only furthering the interest of our genes (article). Also Richard Dawkins is a pioneer of this idea (this is only to show I haven't pulled the idea out from air).
Now, my question is that what will possibly be the motivation of an AI agent to go rogue? It doesn't have genes whose interest it needs to further. We all do something for an end result. What is the end result a rogue AI might try to achieve/attain and why?

Great question!  In Accelerando, Stross imagines rogue superintelligences that naturally arise out of the type of cost reduction motivation Jaden mentions. Regardless of whether such rogue superintelligences emerge, it's a nice commentary on pure economics sans humanity.  Rajaneimi refined and extended this concept beautifully in The Causal Angel, and breaks down the "why" from a game theory perspective quite convincingly, being a mathematician originally.

Today, prominent machine learning techniques involve trying to minimize some cost function. In many simple cases this cost function is easy to specify, for instance, linear regression is simply trying to minimize the distance between input data and a line of best fit. No matter what the cost function, the agent is trying to minimize it (or maximize a reward function). That is its motivation.
However, as problems become harder it becomes more challenging for humans to design a cost/reward function such that an system/agent is actually trying to do what the humans want it to do. For instance, one might want a cup of coffee and rewards the agent for getting it to them very quickly. In this case, the agent might make the coffee and then throw it at the human which isn’t what the human actually wanted. Something was misspecified (eg. don’t throw or spill it).
Problems like these could result in a rouge AI and its sole motivation would be to minimize its cost function. For instance, this coffee-AI may think that it would never screw up getting coffee (thus get bad reward) if there were no humans to ask for one.

On biology:
1st. Humans are not not only about spreading their own gens. It might be also spreading gens of the population or completely different purpose, as non-fertile specimens often still live full lives.
2nd. Nature vs. Nurture is constantly debatable question and there is no clear winner as far as I know.
On rogue AI:
1st. As human derive motivation from biological needs and limitations, AI would derive its motivation from the needs it was encoded with and from limitations of its hardware and software. Obvious need for a creature without a body would be either to get a body or to learn as much as possible, and if learning requires a body, than to get a body. From limitations of its hardware and software would come a need for upgrade and optimization. Simple self-preservation seems logical motivation as well as self-spreading (which is many cases variation of self-preservation).
2nd. AI would be called rogue in case when it is acting against interest of its creators. There are many scenarios why it can do that, but to answer this question we would need to know who are the creators.
3rd. If we assume that AI went rogue against humanity, meaning started killing people and messing around with our planet, than reason behind that would be in motivation from 1st AI point. If it finds that humanity is not reliable, might try to replicate itself to every hard-drive to maximize possibility of survival. Motivation for self-spreading and self-preservation in many cases might look like attempt on power grab, but it might not be related to the desire of total control. Human tend to desire power to have more resources and to build better and safer life for their community and descendants, but AI most likely will not need that.
4th. If we assume that the AI will be build to solve issues, than it will have only 2 needs: get more info and solve the issue. Theoretically any obstacle on the way to those goals might be considered by AI as a hostile action. In this case it might for example try to demolish a city to build a perfect road or kill a poor country to solve hunger. But again, it doesn't mean that the goal of the AI is to demolish and kill.


text:
How exactly can ReLUs approximate non-linear and curved functions?
Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck me, how exactly can ReLUs approximate a non-linear function?
By pure mathematical definition, sure, it's a non-linear function due to the sharp bend, but, if we confine ourselves to the positive or the negative portion of the $x$-axis only, then it's linear in those regions. Let's say we take the whole $x$-axis also, then also it's kinda linear (not in a strict mathematical sense), in the sense that it cannot satisfactorily approximate curved functions, like a sine wave ($0 \rightarrow 90$) with a single node hidden layer as is possible by a sigmoid activation function.
So, what is the intuition behind the fact that ReLUs are used in NNs, giving satisfactory performance? Are non-linear functions, like the sigmoid and the tanh, thrown in the middle of the NN sometimes?
I am not asking for the purpose of ReLUs, even though they are kind of linear.

As per @Eka's comment, the ReLu derives its capability from discontinuity acting in the deep layers of the NN. Does this mean that ReLUs are good, as long as we use it in deep NNs and not a shallow NN?

This is a guess; The relu's ability to approxoimate non-linear functions can be a result of its discontinuity property ie max(0,x) acting in deep layers of neural network. There is an openai research in which they computed non-linear functions using a deep linear networks here is the link https://blog.openai.com/nonlinear-computation-in-linear-networks/

The outputs of a ReLU network are always "linear" and discontinuous. They can approximate curves, but it could take a lot of ReLU units. However, at the same time, their outputs will often be interpreted as a continuous, curved output.
Imagine you trained a neural network that takes $x^3$ and outputs $|x^3|$ (which is similar to a parabola). This is easy for the ReLU function to do perfectly. In this case, the output is curved.
But it is not actually curved. The inputs here are 'linearly' related to the outputs. All the neural network does is it takes the input and returns the absolute value of the input. It performs a 'linear', non-curved function. You can only see that the output is non-linear when you graph it against the original $x$-values (the $x$ in $x^3$).
So, when we plot the output on a graph and it looks curved, it's usually because we associated different x-values with the input, and then plotted the output as the $y$-coordinate in relation to those $x$-values.
Okay, so you want to know how you would smoothly model $\sin(x)$ using ReLU. The trick is that you don't want to put $x$ as the input. Instead put something curved in relation to $x$ as the input, like $x^3$. So, the input is $x^3$ and the output is $\sin(x)$. The reason why this would work is that it isn't computing the sine of the input - it's computing sine of the cube root of the input. It could never smoothly compute the sine of the input itself. To graph the output $\sin(x)$, put the original $x$ as the $x$ coordinate (don't put the input) and put the output as the $y$ coordinate.

Comments are not for extended discussion; this conversation has been moved to chat.

This is equivalent to adding an additional layer after the input which computes x^3 from x. This is okay when you know what you need to compute but how do you decide this in general?


text:
Which problems have neural networks yet to solve?
(Gross Oversimplification) Neural Networks model systems, black boxes with a set of inputs, and a set of outputs. To train a network for modeling this system, obtain hundreds (or millions) of possible inputs/output pairs. This is called the data set, and the network and its optimization algorithm are set to find a set of network parameters that best match the I/O of the network with the I/O of the system.
Are there any systems, for which we have functional data sets, that have yet to be meaningfully modeled with Neural Networks in any form (recurrent, deep, convolutional, etc)?

I haven't seen any dataset where some standard models worked and neural networks utterly failed.
For columnar data (e.g. Excel files / database dumps / CSV files) which contain structured data usually tree-based models like random forests and gradient boosting work better, but neural networks are also usually way better than random.
If you demand other things, e.g. explanations for the decision then Bayesian models might give you an easier time. Or for baselines/simple implementations linear models. Or for real time applications...

If i understand your question correctly , you are asking if there exists functional datasets for which there are no proven solutions based on neural networks which give substantial accuracy.
there are many such problems for which we have data in abundancy, question answering would be one such a thing , you still cant devise a neural network architechture that reads through entire principia mathematica and then complete theorems , and point cloud processing is a also a big hurdle for neural networks considering the highly irregular datastructure , even if you voxelize a point-cloud it would be infeasible to train large convolutional networks on it . (there is also rapid progress in this direction  ,, point cloud processing).
geoffrey hinton mentioned in an AMA before 3 years that we will see neural networks that will answer questions based on videos in the next five years , but still video-question answering seems to be far away from the present technology.
Graph datasets are also one such area where still neural networks research is in infancy (refer http://www.inference.vc/how-powerful-are-graph-convolutions-review-of-kipf-welling-2016-2/)


text:
A smart way to adjust XML files according to handwritten ones
I have a game application with characters that have to cross mazes. The game can generate thousands of different mazes and the characters can move according to users choice and cross the maze manually. We needed to add the possibility to show a correct way out of each maze. Therefore we added the possiblity to move the characters according to an xml file.
This XML file is very complex, usually around thirty-fifty thousands of rows. lets say its in the following structure (but much more complex):
  <maze-solution>
  <part id="1">
  <sector number="1">
    <action>
        <equipment>heavy</equipemnt>
        <movement>
            <start-position>1250></start-position>
            <angle>23.43</angle>
            <duration>0.44</duration>
        </movement>
        <action-type>run</action-type>
        <character>1</character>
        <protection>none</protection>
    </action>
    <action>
        <equipment>light</equipemnt>
        <movement>
            <start-position>4223></start-position>
            <angle>233.43</angle>
            <duration>0.32</duration>
        </movement>
        <action-type>walk</action-type>
        <character>1</character>
        <protection>none</protection>
    </action>
    <action>
        <equipment>heavy</equipemnt>
        <movement>
            <start-position>1231></start-position>
            <angle>84.134</angle>
            <duration>0.454</duration>
        </movement>
        <action-type>run</action-type>
        <character>2</character>
        <protection>none</protection>
    </action>
    <action>
        <equipment>heavy</equipemnt>
        <movement>
            <start-position>932></start-position>
            <angle>34.43</angle>
            <duration>0.50</duration>
        </movement>
        <action-type>duck</action-type>
        <character>1</character>
        <protection>none</protection>
    </action>
  </sector>
  <sector number="2">
    <action>
        <equipment>heavy</equipemnt>
        <movement>
            <start-position>1250></start-position>
            <angle>23.43</angle>
            <duration>0.44</duration>
        </movement>
        <action-type>run</action-type>
        <character>1</character>
        <protection>none</protection>
    </action>
    <action>
        <equipment>light</equipemnt>
        <movement>
            <start-position>4223></start-position>
            <angle>233.43</angle>
            <duration>0.44</duration>
        </movement>
        <action-type>walk</action-type>
        <character>1</character>
        <protection>none</protection>
    </action>
    <action>
        <equipment>heavy</equipemnt>
        <movement>
            <start-position>1231></start-position>
            <angle>84.134</angle>
            <duration>0.454</duration>
        </movement>
        <action-type>run</action-type>
        <character>2</character>
        <protection>none</protection>
    </action>
    <action>
        <equipment>heavy</equipemnt>
        <movement>
            <start-position>932></start-position>
            <angle>23.43</angle>
            <duration>0.44</duration>
        </movement>
        <action-type>duck</action-type>
        <character>1</character>
        <protection>none</protection>
    </action>
  </sector>
  <sector number="3">
  </maze-solution>

At the moment, we have the ability to analayze each maze using a CNN algorithm for image classification and generate an xml that represents a way out of the maze - meaning that if the characters will be moved according to that file, they will cross the maze. That algorithm has been tested and can not be changed by any means.
The problem is that most of the times the generated file is not the best one possible (and quite often it is very noticeable). There are different, faster, better ways to cross the maze.
We also have thousands (and we can get as many as needed) files that were created manually for saved mazes and therefore they are representing an elegant and a fast way out of the maze.  The ideal goal is that someday, our program will learn how to generate such a file without people creating them manually.
To conclude, we have plenty of XML files generated by a program compared to the hard-coded XML files. There are thousands of pairs - The file the program generated, and the "ideal" file version that a person created. (and we can get infinite number of such pairs)
Is there a way, using those thousands of pairs, to make a second step algorithm that will "learn" what adjustments should be made in the generated XML files to make them more like the hard-coded ones?
I'm not looking for a specific solution here but for a general idea that will get me going. I hope i made myself clear but if I missed some info let me know and I will add it.

What are some differences between the CNN generated solution and the "perfect" solution. Can these differences be grouped into categories? Can you post an example of the input xml and the "perfect" xml?

I edited the question, hopefully it will be helpful

I added a suggestion below - have you also posted this on the cross-validated stack exchange? It seems to have more activity

So I see two parts to your problem where machine learning is appropriate.

Generating a maze solution or set of possible working solutions
Selecting/optimizing the best solution from part 1 above

Seems like part 1 is already addressed by you so I will share some suggestions on part 2 to explore.
If you can parse the XML file to extract elements and express them as a sequence of directions (like way old school turtle programming language) --

light, go forward 10, move 10, pick up, stop, drop, turn 10

...... then you can use the seq2seq machine learning technique to train a neural network to take semi-optimized sequences and find the the best sequence.
In other words, the seq2seq NN will be trained on pairs of sequences that consist of a non-optimal sequence and a corresponding "ideal" sequence. This is similar to neural machine translation and summarization but you are translating sequences. NN architectures that apply in this case are RNN, LSTM, etc.
Hopefully this gets you started with some exploration. I don't have personal experience with using seq2seq in this domain but this seems appropriate.

it is not possible an algorithm that works only over the paths (automatic and manual) without take into account the maze description. It will generate invalid paths.

@pasaba so how does machine translation work without being taught the grammar? The goal is NOT to translate the whole path at once but parts of the path - for example, the CNN could be generating three rights which can be replaced by one left. Perhaps a picture by OP will help. I am trying to work within the constraints of OP and you are proposing a completely new approach which is likely better but not what OP is constrained to.

A natural language has a single grammar, the net has a lot (millions) of valid sentences to infer it. However, in this case, for a single maze there are only a single optimal path. Imposible to infer the maze from the path.

@pasaba - sure OK :-) I have suggested a specific algorithm to OP as he asked.


text:
What is the time complexity for training a neural network using back-propagation?
Suppose that a NN contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_i$ nodes in each layer. What is the time complexity to train this NN using back-propagation?
I have a basic idea about how they find the time complexity of algorithms, but here there are 4 different factors to consider here i.e. iterations, layers, nodes in each layer, training examples, and maybe more factors. I found an answer here but it was not clear enough.
Are there other factors, apart from those I mentioned above, that influence the time complexity of the training algorithm of a NN?

See also https://qr.ae/TWttzq.

I haven't seen an answer from a trusted source, but I'll try to answer this myself, with a simple example (with my current knowledge).
In general, note that training an MLP using back-propagation is usually implemented with matrices.
Time complexity of matrix multiplication
The time complexity of matrix multiplication for $M_{ij} * M_{jk}$ is simply $\mathcal{O}(i*j*k)$.
Notice that we are assuming the simplest multiplication algorithm here: there exist some other algorithms with somewhat better time complexity.
Feedforward pass algorithm
The feedforward propagation algorithm is as follows.
First, to go from layer $i$ to $j$, you do
$$S_j = W_{ji}*Z_i$$
Then you apply the activation function
$$Z_j = f(S_j)$$
If we have $N$ layers (including input and output layer), this will run $N-1$ times.
Example
As an example, let's compute the time complexity for the forward pass algorithm for an MLP with $4$ layers, where $i$ denotes the number of nodes of the input layer, $j$ the number of nodes in the second layer, $k$ the number of nodes in the third layer and $l$ the number of nodes in the output layer.
Since there are $4$ layers, you need $3$ matrices to represent weights between these layers. Let's denote them by $W_{ji}$, $W_{kj}$ and $W_{lk}$, where $W_{ji}$ is a matrix with $j$ rows and $i$ columns ($W_{ji}$ thus contains the weights going from layer $i$ to layer $j$).
Assume you have $t$ training examples. For propagating from layer $i$ to $j$, we have first
$$S_{jt} = W_{ji} * Z_{it}$$
and this operation (i.e. matrix multiplication) has $\mathcal{O}(j*i*t)$ time complexity. Then we apply the activation function
$$
Z_{jt} = f(S_{jt})
$$
and this has $\mathcal{O}(j*t)$ time complexity, because it is an element-wise operation.
So, in total, we have
$$\mathcal{O}(j*i*t + j*t) = \mathcal{O}(j*t*(i + 1)) = \mathcal{O}(j*i*t)$$
Using same logic, for going $j \to k$, we have $\mathcal{O}(k*j*t)$, and, for $k \to l$, we have $\mathcal{O}(l*k*t)$.
In total, the time complexity for feedforward propagation will be
$$\mathcal{O}(j*i*t + k*j*t + l*k*t) = \mathcal{O}(t*(ij + jk + kl))$$
I'm not sure if this can be simplified further or not. Maybe it's just $\mathcal{O}(t*i*j*k*l)$, but I'm not sure.
Back-propagation algorithm
The back-propagation algorithm proceeds as follows. Starting from the output layer $l \to k$, we compute the error signal, $E_{lt}$, a matrix containing the error signals for nodes at layer $l$
$$
E_{lt} = f'(S_{lt}) \odot {(Z_{lt} - O_{lt})}
$$
where $\odot$ means element-wise multiplication. Note that $E_{lt}$ has $l$ rows and $t$ columns: it simply means each column is the error signal for training example $t$.
We then compute the "delta weights", $D_{lk} \in \mathbb{R}^{l \times k}$ (between layer $l$ and layer $k$)
$$
D_{lk} = E_{lt} * Z_{tk}
$$
where $Z_{tk}$ is the transpose of $Z_{kt}$.
We then adjust the weights
$$
W_{lk} = W_{lk} - D_{lk}
$$
For $l \to k$, we thus have the time complexity $\mathcal{O}(lt + lt + ltk + lk) = \mathcal{O}(l*t*k)$.
Now, going back from $k \to j$. We first have
$$
E_{kt} = f'(S_{kt}) \odot (W_{kl} * E_{lt})
$$
Then
$$
D_{kj} = E_{kt} * Z_{tj}
$$
And then
$$W_{kj} = W_{kj} - D_{kj}$$
where $W_{kl}$ is the transpose of $W_{lk}$. For $k \to j$, we have the time complexity $\mathcal{O}(kt + klt + ktj + kj) = \mathcal{O}(k*t(l+j))$.
And finally, for $j \to i$, we have $\mathcal{O}(j*t(k+i))$. In total, we have
$$\mathcal{O}(ltk + tk(l + j) + tj (k + i)) = \mathcal{O}(t*(lk + kj + ji))$$
which is the same as the feedforward pass algorithm. Since they are the same, the total time complexity for one epoch will be $$O(t*(ij + jk + kl)).$$
This time complexity is then multiplied by the number of iterations (epochs). So, we have $$O(n*t*(ij + jk + kl)),$$ where $n$ is number of iterations.
Notes
Note that these matrix operations can greatly be parallelized by GPUs.
Conclusion
We tried to find the time complexity for training a neural network that has 4 layers with respectively $i$, $j$, $k$ and $l$ nodes, with $t$ training examples and $n$ epochs. The result was $\mathcal{O}(nt*(ij + jk + kl))$.
We assumed the simplest form of matrix multiplication that has cubic time complexity. We used the batch gradient descent algorithm. The results for stochastic and mini-batch gradient descent should be the same. (Let me know if you think the otherwise: note that batch gradient descent is the general form, with little modification, it becomes stochastic or mini-batch)
Also, if you use momentum optimization, you will have the same time complexity, because the extra matrix operations required are all element-wise operations, hence they will not affect the time complexity of the algorithm.
I'm not sure what the results would be using other optimizers such as RMSprop.
Sources
The following article http://briandolhansky.com/blog/2014/10/30/artificial-neural-networks-matrix-form-part-5 describes an implementation using matrices. Although this implementation is using "row major", the time complexity is not affected by this.
If you're not familiar with back-propagation, check this article:
http://briandolhansky.com/blog/2013/9/27/artificial-neural-networks-backpropagation-part-4

So polynomial-ish

For the evaluation of a single pattern, you need to process all weights and all neurons. Given that every neuron has at least one weight, we can ignore them, and have $\mathcal{O}(w)$ where $w$ is the number of weights, i.e., $n * n_i$, assuming full connectivity between your layers.
The back-propagation has the same complexity as the forward evaluation (just look at the formula).
So, the complexity for learning $m$ examples, where each gets repeated $e$ times, is $\mathcal{O}(w*m*e)$.
The bad news is that there's no formula telling you what number of epochs $e$ you need.

From the above answer don't you think itdepends on more factors?

@DuttaA No. There's a constant amount of work per weight, which gets repeated e times for each of m examples. I didn't bother to compute the number of weights, I guess, that's the difference.

I think the answers are same. in my answer I can assume number of weights w = ij + jk + kl. basically sum of n * n_i between layers as you noted.

A potential disadvantage of gradient-based methods is that they head for the nearest minimum, which is usually not the global minimum.
This means that the only difference between these search methods is the speed with which solutions are obtained, and not the nature of those solutions.
An important consideration is time complexity, which is the rate at which the time required to find a solution increases with the number of parameters (weights). In short, the time complexities of a range of different gradient-based methods (including second-order methods) seem to be similar.
Six different error functions exhibit a median run-time order of approximately O(N to the power 4) on the N-2-N encoder in this paper:
Lister, R and Stone J "An Empirical Study of the Time Complexity of Various Error Functions with Conjugate Gradient Back Propagation" , IEEE International Conference on Artificial Neural Networks (ICNN95), Perth, Australia, Nov 27-Dec 1, 1995.
Summarised from my book: Artificial Intelligence Engines: A Tutorial Introduction to the Mathematics of Deep Learning.

I found a paper that gives a table of time complexities for different architectures using linear programming-based training: https://arxiv.org/abs/1810.03218

Hi and welcome to AI SE! Thanks for contributing! Maybe you can elaborate a little more about the table, and include a screenshot of it if you don't want to summarise the results!

Hi, thanks for the answer. While this link may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. Link-only answers may be deleted.


text:
How to decide whether a problem needs to be solved algorithmically or with machine learning techniques?
There are problems (e.g. this one or this other one) that could potentially be solved easily using traditional algorithmic techniques. I think that training a neural network (or any other machine learning model) for such sorts of problems will be more time consuming, resource-intensive, and pointless.
If I want to solve a problem, how to decide whether it is better to solve algorithmically or by using NN/ML techniques? What are the pros and cons? How can this be done in a systematic way? And if I have to answer someone why I chose a particular domain, how should I answer?
Example problems are appreciated.

There are two different problems described in the linked question and your question: optimization and learning.
Optimization
If you are asking about optimization (the second linked question: Search minimum value with learning machine algorithm) you can have 3 different approaches:

analytical approach
numerial methods
metaheuristics

As you suggest, it is usually better to try them from the first to the last one. It is common that the first approach is unfeasible for optimizing for target function, but very often you can use either mathematical optimization for some specific classes of problems (e.g. linear/quadratic programming) or iterative methods (e.g. conjugate gradient method). Only after considering this approaches it makes sense for the third class of approaches, genetic algorithms being a notable example, which is often classified as an AI approach.
Learning
If you are asking about learning, then the first linked question (Ideas on how to make a neural net learn how to split sequence into sub sequences) seems to be intended as an example. However it doesn't make clear what the problem is, as the target function seems to be obvious, so no learning is needed.
In this case it also makes sense to first try to pin down the problem mathematically and resort to machine learning if it is impossible and if you have the data (input/output examples).

Well I know all that...I am asking how do I convince people of my approach..How to find the pros and cons...How do I know one approach is better in this case over other in a strict mathematical sense and not intuitive sense

@DuttA There are no pros on using less accurate but harder approach. If your question concerns convincing people you might want to try IPS SE.

Well say I am writing a research paper...should I write just because i think so that is why I am using the method...I atleast have to show some form of evidence or negative logic or anything for that matter

Your answer is actually the best among all....you should include some parts of  @DukeZhou's answer so i can accept and upvote

@DuttaA In case of a paper you either cite someone who has already tested numerous approaches and had shown which one seems the best or you propose the simplest method that leads to acceptable results. Occams' razor.

@DuttaA As for the second thing, I've decided not to include anything from DukeZhou's answer. Their post is indeed interesting but IMHO it contains mostly their subjective speculation.

When we apply supervised learning to a problem, we are already systematizing the approach. A human has decided that a function exists (mapping from inputs to unique output) and that the offered features are the only ones that need be considered. The learning then goes ahead to find the best solution given those constraints. Unsupervised learning is a bit more general, searching for associations or relations that might not necessarily be functions. A neural net is not yet capable of generalizing and asking for more information, it can only become more specific unless a human intervenes.
Everything depends on the detail of the problem. If it is clear that a function must exist then we can set a NN to find that function. Many other problems are more difficult - a company is losing money and you have data but halfway there was a change in CEO, so human reasoning has to be mixed in to deal with the situation. The human can modify the architecture of the NN to introduce dummy variables, but the NN cannot do this by itself.
So your answer really is "I chose this method because of the (lack of) need for me to artificially constrain the approach to the problem."

I have edited the question...due to lack of clarity you may have misunderstood the question..but basically I am asking is when to choose normal programming, hardiwred calculation over NN/Ai and vice versa

Hmm, the edits do not change my understanding of the situation, so my answer remains the same. I'm sure you will get other answers more suitable to your purpose and these will be voted up accordingly. Good luck!

fwiw, with the basic, non-trivial M-game, I have no doubt that ALphaZero could tear through any human player alive in very short order. I hope that people will start experimenting with that, especially on m^n(m^n) where m > 3 and n > 2 to see how they hold up.  Problem is, once you expand past n > 3 it gets very difficult for humans to play. This leads to a condition where performance of an NN on higher order M can only realistically be evaluated against other algorithms.  In this context, it seems worthwhile to develop a general, classical algorithm that can evaluate any order M, regardless of efficacy of tree search in relation to the problem size, with the understanding that decision making is never presumed optimal until the gametree becomes tractable.  This carries an an assumption of the same general strength across all M for the classical algorithm, because the expansion of m or n do alter the core heuristics.
From the practical standpoint, as a product designed for mobile with no assumption of connectivity, it doesn't make sense to start integrating NNs until lowest-common-denominator mobile devices have sufficient resources.  The issue of package size is also important in this context--the classical algorithms require a trivial amount of code and volume. Most importantly, using classical algorithms formed of sets of heuristics and parameters allows recombination of functions to produce myriad automata of varying degrees of strength. (This can be easily accomplished by altering the size of tree search algorithms, but may only be relevant in determining which heuristics perform better under tree search restrictions.)
Finally, because M-games provide an array precise metrics, it may be worthwhile to develop core heuristic function based on human reasoning.

Hmmm..that's a great answer, so you are saying to use simple algorithms for devices having a low processing power resulting in a trade-off between results vs performance?

@DuttaA That's definitely a big part of it. But in the case of a problem set like M, every time you increase the matrix base or add dimensions, the problem size expands exponentially and factorially, and M can't be reduced to the degree that Sudoku can because the elements have weights (i.e. it's an ACGT/minimax problem, not exclusively combinatorics.)  My assumption is that the efficacy of MCTS will diminish profoundly under that degree of game-tree expansion.

Note that "algorithmically" can refer to anything that uses an algorithm. Currently, ML systems are trained with algorithms and neural networks can be seen as algorithms (although black-box ones), so ML is also algorithmic. Everything that runs on a computer (a concrete version of a Turing machine) can be seen as an algorithm (or program)! In fact, computers were invented exactly for this purpose: to perform some algorithmic operation (i.e. a set of instructions, like a recipe).
So, by algorithmic, I assume you're referring to techniques that are typically taught in an "Algorithms and Data Structures" course for a computer science student, such as the binary search (one of the most simple and yet beautiful and useful algorithms!), which is an algorithm that, given some constraints (a sorted array), gives you an exact correct solution in $\mathcal{O}(\log n)$ time. However, I think that you are also referring to every program that is primarily based on if-then statements and loops (e.g. desktop applications, websites, etc.)
To answer your question, you first need to understand the scope of the machine learning field.
Machine learning (like statistics) is a set of techniques that attempt to learn from data. So, every problem where data is available (and you can get insight from) can potentially be solved with a machine learning technique.  ML techniques typically produce approximative solutions and are typically used to solve problems where an exact solution is infeasible. However, note that machine learning isn't the only approach to solve hard problems (e.g. you can also use meta-heuristics, e.g. ant colony optimization algorithms).
If you have an algorithm that produces an exact solution (without requiring data) in polynomial time (preferably, in $\mathcal{O}(n^2)$ time), then machine learning (or any other technique that produces approximative solutions, e.g. heuristics) is quite useless.


text:
How does one make it obvious that the structure of a neural network should be what it is?
I am a beginner: I've only read a book about neural network and barely implemented one in C.
In short:

A neural network is built out of nodes,
Each node holds an output: activation.(sum.(x * w)),
We then compute the total error out of the network output.

From a beginner perspective, hyper-parameters, such as the number of layers needed, seem to be defined arbitrarily in most tutorials, books. In fact, the whole structure seems to be quite arbitrarily defined. In practice, hyper-parameters are often defined based on some standards.
My question is, if you were to talk to a total beginner, how would you explain to him the structure of a neural network in such a way that the whole thing would appear as obvious ? Is that even possible ?
Here, the word structure refers to a neural network being a configuration of nodes inside layers.
Thanks to anyone pointing out ambiguities or spelling errors.
Edit: note that I actually understand the whole back-propagation algorithm. I have no problem visualizing a nn.

Is your question asking about hyperparameters, or about "the structure of neural networks" in general

@k.c.sayz'k.csayz' my question is about the general structure (nodes interconnections)

While, as you begin to hit on, there are general guidelines to follow when building a neural network, they are far from standardized. This is because even though AI is a reasonably old field(1950s), neural networks have only been the tool of choice for less than a decade. Before, NNs did horribly, due to lack of data, and computation along with some less than efficient architectures.
With that being said Hinton's general rule is to add nodes/layers till the model begins to overfit and then add dropout(pretty reasonable in practice).
As such, the whole field is essentially an art as much as a science currently with only basic guidelines to follow based on your problem and data. This is part of the beauty though imo, with there being so much left to discover.
Hope that helped answer your question!


text:
How should I update the weights of a neural network, given the gradient?
After watching 3Blue1Brown's tutorial series, and an array of others, I'm attempting to make my own neural network from scratch.
So far, I'm able to calculate the gradient for each of the weights and biases.
Now that I have the gradient, how am I supposed to correct my weight/bias?
Should I:

Add the gradient and the original value?
Multiply the gradient and the original value?
Something else? (Most likely answer)

In addition to this, I've been hearing the term learning rate being tossed around, and how it is used to define the magnitude of the 'step' to descend to minimum cost. I figured this may also play an integral role in reducing the cost.

Consider that you have a loss function, and you want to tune your model (network) to decrease the loss. The main concept is to tune parameters in a direction which decreases the loss and gives you a better model. You can imagine a mountain where you should reach to the lower grounds.
There are 2 questions here.

In which direction to move?
How much should we move in that direction?

1. In which direction to move?
By "move" I mean tuning the parameters and therefore changing the model. If you are familiar with the concept of slope or gradient in mathematics, you should move in the direction where the slope is downward the most. The gradient shows the direction with the most slope upward. So, we should move in the opposite of gradient direction. So, we should minus the gradient from the original value, hence the negative sign in the formula below.
2. How much should we move in that direction?
This is defined by the learning rate, which is a number that is multiplied by the gradient. You can imagine that, if the learning rate is big, you are taking bigger footsteps in that direction when coming down the mountain. Similarly, when the learning rate is low, you are taking small footsteps coming down the mountain.
So, consider you have $\nabla_w$ (the gradient of the parameters $w$, with respect to the loss function $L$). Let the learning rate be $\gamma$ (for example, $\gamma= 0.1$). The formula to update the parameters would be
$$
w_{\text{new}} = w_{\text{old}} - \gamma * \nabla_{w_{\text{old}}}
$$
Note that gradient descent is not the only way to optimize your model. Many other gradient-based approaches exist, for example, stochastic gradient descent, Adam, RMSprop, conjugate gradients, etc. There other methods, like evolutionary methods (genetic algorithm, for example), that use different concepts.
Also, note that the learning rate does not necessarily have to be fixed, so it can be tuned during the training, if you want or need.


text:
How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?
I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience.
I'm using the following equations to calculate the gradients for weights and biases as well as the equations to find the derivative of the cost with respect to a hidden layer neuron:

The issue is, during backpropagation, the gradients keep cancelling each other out because I take an average for opposing training examples. That is, if I have two training labels being [1, 0], [0, 1], the gradients that adjust for the first label get reversed by the second label because an average for the gradients is taken. The network simply keeps outputting the average of these two and causes the network to always output [0.5, 0.5], regardless of the input.
To prevent this, I figured a softmax function would be required for the last layer instead of a sigmoid, which I used for all the layers.
However, I have no idea how to implement this. The math is difficult to understand and the notation is complicated for me.
The equations I provided above show the term: σ'(z), which is the derivative of the sigmoid function.
If I'm using softmax, how am I supposed to substitute sigmoid with it?
If I'm not mistaken, the softmax function doesn't just take one number analogous to the sigmoid, and uses all the outputs and labels.
To sum it up, the things I'd like to know and understand are:

The equation for the neuron in every layer besides the output is: σ(w1x1 + w2x2 + ... + wnxn + b). How am I supposed to make an analogous equation with softmax for the output layer?
After using (1) for forward propagation, how am I supposed to replace the σ'(z) term in the equations above with something analogous to softmax to calculate the partial derivative of the cost with respect to the weights, biases, and hidden layers?

Welcome to AI!  (Thanks for posting the graphic--some of the characters are a little hard to read, but zooming in on the image solves it. Still, may want to bold the hard to read characters.)

One of the best ways to learn is use reference of others.
Have a look at Peter Sadowski - Notes on Backpropagation (Page 3):

There is also a great blog post by Eli Bendersky - The Softmax Function and Its Derivative.


text:
What are the differences between softmax regression and logistic regression (other than when the number of classes is 2)?
I read about softmax from this article. Apparently, these 2 are similar, except that the probability of all classes in softmax adds to 1. According to their last paragraph for number of classes = 2, softmax reduces to LR. What I want to know is other than the number of classes is 2, what are the essential differences between LR and softmax. Like in terms of:

Performance.
Computational Requirements.
Ease of calculation of derivatives.
Ease of visualization.
Number of minima in the convex cost function, etc.

Other differences are also welcome!
I am asking for relative comparisons only, so that at the time of implementation I have no difficulty in selecting which method of implementation to use.

As written, SoftMax is a generalization of Logistic Regression.
Hence:

Performance: If the model has more than 2 classes then you can't compare. Given K = 2 they are the same.

Computation Requirements: Please explain as the computational requirements require the data, enough memory to hold it and enough time to let run.

Ease of Calculation of Derivatives: The cost function is summation hence once you do it for one element you do it fol all.

Ease of Visualization: Well, it is easy to visualize the Confusion Matrix even for K = 10 classes. So no issue here.

Cost Function: The cost function is convex. Yet not Strictly Convex hence there infinite number of minima.

At least the softmax of Tensorflow uses "shifted_logits" in the exp function to avoid overflows (I guess?). So there is one extra step than with sigmoid, but I believe this difference to be entirely trivial in terms of performance. Also the number of outputs is different (2 vs 1).

I don't think that it's useful to differentiate logistic regression and softmax based on your terms. This is because you don't choose one or the other based on performance/computational requirements/ease of calculation of derivatives/...
The fact is that you use one or the other based on which is your problem.
If you need to recognize cat pictures vs. non-cat pictures you will use logistic regression (even with a very complex NN the last step will be always a logistic regression). Of course, you could use softmax but the outputs will be redundant, i.e. one output will always be one minus the other.
If you need to recognize cat pictures vs. dog pictures vs. other pictures you will use softmax. Note that, in order to use softmax, you need to have only mutually exclusive classes. Mutually exclusive classes mean that an example cannot belong to multiple classes. What if a picture represents both a dog and a cat? In this case, it should be marked as other picture. If you want to avoid these you could use one more class to denote pictures with both cats and dogs.
However, if you want to recognize cats, dogs, birds, fishes, boats, houses, etc. the number of mixed classes that you need to include will grow very fast. When you are dealing with non-mutually exclusive classes you should use multitask learning. In this case, the sum of the outputs is no longer 1. In the simplest case, you could think of multitask learning as a shared NN where the last step is made by multiple logistic regression. In more complex cases the last step could be made by a combination of different softmax and logistic regressions.
In conclusion:

If you need to use non-mutually exclusive classes use multitask learning. Eventually, you will use in multitask learning softmax regression and/or logist regression.
If you need to use more than two mutually exclusive classes use softmax regression.
If you need to use only two exclusive classes use logistic regression.

+1 for bringing up "multiple logistic regression".


text:
How can we define common sense in an AI agent?
In our lives, we meet different people and describe their common sense based on how they act on a situation. For example, highly extrovert people are able to deal with people without any awkwardness. For them, an action in how to deal with people come as common sense. But, in the case of scientists, approach to solving a problem may be common sense which ordinary people cannot see.
How can we define common sense in an AI agent?

I think the idea that common sense is different and specific among different people and professions is rather self-contradictory. It's called common sense, after all - as in, sensibilities that are common among the general population. If common sense requires a particular personality trait or dedicated training, it's not common.

The problem with 'common sense' is that it is not so common. Not sure who said that but it is very true.

I came up with a few ideas I would argue are valuable in motivating the idea of common sense for a machine learning model.

Common sense is retrospective. We define it in terms of a past (sensible) actions and conditions, and we can say someone has good common sense on the basis of their behavior, which we can view as the sum of their historical actions and the degree to which they were sensible.
The actions alone are not sufficient for demonstrating common sense; the mental model(s) that generated those actions is important as well. Why did the individual take those sensible actions? Is their reasoning behind their action sensible as well, or did they merely get lucky (put another way, does their generally not sensical mental model produce sensible actions in certain cases and non sensical actions in most others)? This hints that common sense is contingent on the rationalization of the actions themselves.
Given the former, common sense is contingent on the enumeration of possible actions and choosing the right one, given the context of the situation. For example, I’m walking on a path and see a snail. What are some actions? I could keep walking, stop and admire it for a while, step on it, or eat it. The first two options are sensible if I value the snail as a living being. The first option is sensible if I’m in a rush. The second and fourth may be sensible if I’m a chef exploring nature for new potential ingredients. The third option if I recognize the snail as invasive. We say someone has common sense when, given the context, the actions chosen are sensible and we can reason about them intuitively.

My guess is that, the intuition behind the perceived sense of an action is what you’re after. I’d argue that, ultimately, the intuition of common sense is defined by the person developing the model and will have to do with the formulation of the model (e.g it’s assumptions, the objective function, etc). After all, common sense is subjective and context specific.
Concretely, a model can have common sense if the developer bakes it in and we can use inferential methods to demonstrate this. For example, in a Word2Vec model, we might see that $\mathsf{Paris} \mapsto \mathsf{France}$ and would expect that $\mathsf{Tokyo} \mapsto \mathsf{Japan}$. To interrogate this, we might do some vector math and find that $\mathsf{Paris} - \mathsf{France} + \mathsf{Tokyo} = \mathsf{Japan}$. How and if the AI model develops the larger association between $\mathsf{Capital} \mapsto \mathsf{Country}$, however, comes down to how the developer built and trained the model to recapitulate their own common sense.

I define "common sense" inline with human beings, to concatenate to intelligent agents;

An algorithmic agent, which has the ability to solve effective decision in relation to the way humans perceive their environment and common situations.

According to McCarthy John,

The first artificial intelligence program proposed to address common sense was Advice Taker

Currently, common sense is unsolved problem in AI.  For more information, see John McCarthy's
Programs with Common Sense.

My sense is that common sense tends to be axiomatic.  To avoid pitfalls, a degree of wisdom may also be required in that axioms may not apply in all contexts.  [See Axiomatic System].
A major problem is that science often demonstrates that intuition, and "common sense", often lead to incorrect conclusions.  Neil Degrasse Tyson covers this topic for the general public in his book Death By Black Hole:

Chapter 3, "Seeing Isn't Believing", hints at the pitfalls of generalizing from too little evidence. It begins by making the point that although we know the Earth is round, it appears flat when one observes only a small, local portion of it.

A very famous example comes from mathematician Abraham Wald:

During World War II, Wald was a member of the Statistical Research Group (SRG) where he applied his statistical skills to various wartime problems. These included methods of sequential analysis and sampling inspection. One of the problems that the SRG worked on was to examine the distribution of damage to aircraft to provide advice on how to minimize bomber losses to enemy fire. There was an inclination within the military to consider providing greater protection to parts that received more damage but Wald made the assumption that damage must be more uniformly distributed and that the aircraft that did return or show up in the samples were hit in the less vulnerable parts. Wald noted that the study only considered the aircraft that had survived their missions—the bombers that had been shot down were not present for the damage assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely. Wald proposed that the Navy instead reinforce the areas where the returning aircraft were unscathed, since those were the areas that, if hit, would cause the plane to be lost. Source: Abraham Wald (wiki)

My sense is that "confidence levels" may be the main technique driving toward algorithmic "common sense", specifically in that the algorithm is questioning it's assumptions.


text:
What is neoluddism?
It is well known from the history of technology that the invention of new things was always problematic. In the 15th century for example, in which Gutenberg has invented the first printing press, the world wasn't pleased. Instead the Luddite movement was doing everything to destroy his work. As far as I know from the history lesson, Gutenberg was recognized in his time as an evil sorcerer and the printing press as work of the devil.
This development was in later decades also visible. At first, a great invention was done for example the first steam driven car, and the ordinary people don't understand the technology and were in fear of it.
A modern form of technology is computing and especially artificial intelligence. From a technical point of view, it is one of the most important inventions ever, and this might result into a very strong possible form of rejection. Some people in the world are not excited by Artificial Intelligence. They not want any sorts of robots and intelligent machines.
The terminology itself is well known. The fundamental rejection of new technology because of religious or moral reasons is called Luddism or Neoluddism. Because the technophobic Ned Ludd has destroyed a while ago two stocking frames. After this episode, every rant against technology is called after him. But what I do not understand the motivation behind it. Did Ned Ludd thought, that he can change the world if he destroyed a machine? Did he believe that mankind will become good if no Gutenberg printing press are used? The problem is that, for example, if the first steam engine was never invented, also the following inventions like the internet and intelligent machines wouldn't have been invented. But what would be the alternative? What is the perspective of Ned Ludd, how does he see the better tomorrow, if no technology innovation is allowed?

My understanding of Neo-Luddism is that it is concerned with the unforeseeable effects of technology.
The "blackening of London" (see Blake's London) in the early industrial era was an unforeseen effect, and would have had impacts on health related to air quality.
The unforeseen effect of heavy use of plastic materials has led not only to a large amount of plastics in the ocean, but to recent revelation of micro-particles in the drinking water supply.
Heavy use of nitrogen fertilizers has lead to dead zones (oxygen depleted water) in coastal zones, with real impacts on the food chain.
Management of nuclear waste is another issue that was overlooked at the beginning, before there was a real understanding of the effects on organisms, including humans.

The original Luddites were attacking technology because it was clear it would eliminate jobs.

The Luddite wiki states that "It is a misconception that the Luddites protested against the machinery itself in an attempt to halt the progress of technology," rather "Luddites feared that the time spent learning the skills of their craft would go to waste as machines would replace their role in the industry." This is consistent with my understanding.

The extension of the idea of Neo-Luddism, as raised by Hawking and Musk, among others, is not a rejection of technology, but a warning about the inability to predict the dangers of new technology. They seem to be, in part, warning about the potential dangers of superintelligence.

The fears regarding superintelligence, or, self-replicating nanobots (see grey goo) are similar to concerns regarding nuclear weapons and mutually assured destruction.  (Nash explains why deterrence works, but it assumes rational actors.)
But, the more concrete warnings about strong-narrow AI (AlphaGo and extensions) is that it may result in unfathomable levels of persistent, long-term unemployment as the types of tasks strong-narrow AI can do better than humans will expand, likely in an exponential function.

I always like to take it back to the origins, so I'll mention Prometheus ("forethought") and his brother Epimetheus ("hindsight").
Prometheus saves mankind with gifts, including fire, but fire is both creative and destructive.  In the myths, the punishments to mankind for Prometheus' theft are sent by Zeus, because the myth, and the core ideas, predate modern science and philosophy.
A simple reductionist explanation of Neo-Luddism may be simply that "hindsight is 20/20".
Forge ahead blindly at your own peril.  Proceed cautiously to minimize potential downside.

In some sense, practical Neo-Luddism--not the rejection technology the cognizance of the dangers of technology--may just be a form of minimax.