text:
The Ackermann function
The Ackermann function is notable for being the one of the simplest examples of a total, computable function that isn't primitive recursive.
We will use the definition of \$A(m,n)\$ taking in two nonnegative integers where
$$\begin{align}
A(0,n) & = n+1 \\
A(m,0) & = A(m-1,1) \\
A(m,n) & = A(m-1,A(m,n-1))
\end{align}$$
You may implement

a named or anonymous function taking two integers as input, returning an integer, or
a program taking two space- or newline-separated integers on STDIN, printing a result to STDOUT.

You may not use an Ackermann function or hyperexponentiation function from a library, if one exists, but you may use any other function from any other library. Regular exponentiation is allowed.
Your function must be able to find the value of \$A(m,n)\$ for \$m \le 3\$ and \$n \le 10\$ in less than a minute. It must at least theoretically terminate on any other inputs: given infinite stack space, a native Bigint type, and an arbitrarily long period of time, it would return the answer. Edit: If your language has a default recursion depth that is too restrictive, you may reconfigure that at no character cost.
The submission with the shortest number of characters wins.
Here are some values, to check your answer:
$$\begin{array}{c|cccccccccc}
A & n = 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 \\
\hline
m=0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 \\
1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 \\
2 & 3 & 5 & 7 & 9 & 11 & 13 & 15 & 17 & 19 & 21 & 23 \\
3 & 5 & 13 & 29 & 61 & 125 & 253 & 509 & 1021 & 2045 & 4093 & 8189 \\
4 & 13 & 65533 & \text{big}
\end{array}$$

How has this not been asked before??

Tried in Python, got ~60 bytes and a lot of stack overflows. Decided not to post as anything that can only calculate m <= 2 is not very useful :D

I think it'd be more fun to make this [tag:fastest-code]

Downvoting because there's no challenge here. The obvious answer -- just naively implementing the function exactly according to its definition -- is always going to be the best answer. So the question is just "Which language has the least number of characters in the obvious expression of Ackermann's function?" The true winner is the programming language, not the person who wrote the obvious program in it.

I agree that this challenge would be more interesting as a fastest code, currently this challenge seems to be mostly, how can I implement the ackerman function in as little characters without much consideration to speed.
Instead of how can I make use of any patterns in the ackerman function to speed it up

What if my language's recursion limit is too low to compute A(3,8) and above as naively as the others did? Do I have to come up with a non-recursion solution, or can I also just "assume infinite stack space" in these cases? I'm fairly certain, it would terminate within a minute.

@DavidRicherby "The obvious answer [...] is always going to be the best answer." This is not true of all languages. I feel a little dirty for only having an example in my home language, but there are multiple ways to express Ackermann and in some languages you can get savings by using that fact. This was my intention for the challenge.

@Sp3000 A problem with fastest-code is what counts as cheating. My SO answer to a related question at http://stackoverflow.com/a/12188327/400547 used shortcuts for m < 3 and one could add return BigInteger.Pow(2, n + 3) - 3 for m == 3 too. Does that count as cheating?

Pyth, 19
DaGHR?atG?aGtHH1GhH

Defines a, which works as the Ackermann function. Note that this requires a higher recursion depth than the official pyth compiler allowed up until today to compute a 3 10, so I increased the recursion depth. This is not a change to the language, just to the compiler.
Test:
$ time pyth -c "DaGHR?atG?aGtHH1GhH           ;a 3 10"
8189

real    0m0.092s
user    0m0.088s
sys     0m0.000s

Explanation:
DaGH                     def a(G,H):
    R                    return
    ?          G                (if G:
     atG                              (a(G-1,
        ?    H                               (if H:
         aGtH                                      a(G,H-1)
              1                               else:1)
                hH               else:H+1)

Essentially, it first conditions on the truth value of G whether to recurse or return H+1. If it is recursing, the first argument is always G-1, and it conditions on the truth value of H whether to use a(G,H-1) as the second argument, or to use 1 as the second argument.

In modern Pyth (I assume this was added after this challenge) I think you can more or less change DaGHR to M and a to g. (Did the order of arguments for ? change?)

@Mauris Yes, you can use M instead, and yes, ? argument order changed. It's now condition, true, false. It was true, condition, false.

@Lynn Fun Pyth history facts: This question actually influenced isaacg to change (at least) two things about Pyth: the recursion limit and the change to M!

Haskell, 35
0%n=1+n
m%n=iterate((m-1)%)1!!(n+1)

this defines the operator function %.
this works by noticing that m%n (where a is the ackerman function) for nonzero m is (m-1)% applied n+1 times to 1. for example, 3%2 is defined as 2%(3%1) which is 2%(2%(3%0)), and this is 2%(2%(2%1))

too bad I can't use 0%n instead of n+1 because of precedence

What am I doing wrong?

wow, this has been wrong for so long, and nobody, including myself, noticed? good work. so it seems I mistakenly copied the first version of the answer, which was faulty, maybe by mistake or because I thought it worked.

Binary lambda calculus, 54 bits = 6.75 bytes
Hexdump:
00000000: 1607 2d88 072f 68                        ..-../h

Binary:
000101100000011100101101100010000000011100101111011010

This is λm. m (λg. λn. g (n g 1)) (λn. λf. λx. f (n f x)), where all numbers are represented as Church numerals.

GolfScript (30)
{1$1>{1\){1$(\A}*\;}{+)}if}:A;

Online demo
Without the 1> (which special-cases A(1, n)) it takes 9 minutes to compute A(3, 10) on the computer I've tested it on. With that special case it's fast enough that the online demo takes less than 10 seconds.
Note that this isn't a naïve translation of the definition. The recursion depth is bounded by m.
Dissection
{             # Function boilerplate
  1$          # Get a copy of m: stack holds m n m
  1>{         # (Optimisation): if m is greater than 1
    1         #   Take this as the value of A(m, -1)
    \){       #   Repeat n+1 times:
              #     Stack: m A(m, i-1)
      1$(\    #     Stack: m m-1 A(m, i-1)
      A       #     Stack: m A(m, i)
    }*
    \;        #   Lose that unwanted copy of m
  }{          # Else
    +)        #   A(m in {0, 1}, n) = m + n + 1
  }if
}:A;          # Function boilerplate

In CJam, you wouldn't need 1>. After removal (and changing if to ?), computing 3 10 A takes 110 seconds with the online interpreter and six seconds with the Java interpreter.

J - 26 char
($:^:(<:@[`]`1:)^:(0<[)>:)

There is an alternate, more functional definition of Ackermann:
Ack 0 n = n+1
Ack m n = Iter (Ack (m-1)) n
Iter f 0 = f 1
Iter f n = f (Iter f (n-1))

It so happens that Iter is very easy to write in J, because J has a way of passing in the m-1 to Ack and also to define the initial value of Iter to be 1. Explained by explosion:
(                      >:)  NB. increment n
                ^:(0<[)     NB. if m=0, do nothing to n+1; else:
   ^:                       NB. iterate...
($:                      )  NB.   self ($: is recursion)
     (<:@[     )            NB.   with left arg m-1
          `]                NB.   n+1 times
            `1:             NB.   starting on 1

This relies on what J calls the gerund form of ^:—basically a way to have more control over all the bounds in a tacit (point-free) fashion.
At the REPL:
   3 ($:^:(<:@[`]`1:)^:(0<[)>:) 3
61
   ack =: ($:^:(<:@[`]`1:)^:(0<[)>:)
   (i.4) ack"0 table (i.11)
+-----+------------------------------------------+
|ack"0|0  1  2  3   4   5   6    7    8    9   10|
+-----+------------------------------------------+
|0    |1  2  3  4   5   6   7    8    9   10   11|
|1    |2  3  4  5   6   7   8    9   10   11   12|
|2    |3  5  7  9  11  13  15   17   19   21   23|
|3    |5 13 29 61 125 253 509 1021 2045 4093 8189|
+-----+------------------------------------------+
   6!:2 '3 ($:^:(<:@[`]`1:)^:(0<[)>:) 10'  NB. snugly fits in a minute
58.5831

We need to define ack by name to be able to put it in a table, because $: is a horrible, ugly beast and lashes out at anyone who attempts to understand it. It is self-reference, where self is defined as the largest verb phrase containing it. table is an adverb and so would love to become part of the verb phrase if you give it the chance, so you have to trap $: in a named definition to use it.

Edit: 24 char?
Years later, I found a solution which is two characters shorter.
(0&<~(<:@#~$:/@,1:^:)>:)

It's a lot slower, though: 3 ack 8 takes over a minute on my machine. This is because (1) I use a fold / instead of iteration, so J probably has to remember more things than usual, and (2) while 0&<~ performs the same calculation as (0<[), it actually gets executed n+1 times before taking the recursive step when invoking m ack n—0&< happens to be idempotent, so it doesn't ruin the calculation, but n gets big fast and ack is highly recursive.
I am doubtful that a more powerful machine could push the new code under a minute, because this is a computer where the old code can find 3 ack 10 in less than 15 seconds.

JavaScript, ES6, 41 34 bytes
f=(m,n)=>m?f(m-1,!n||f(m,n-1)):n+1

Run this in a latest Firefox Console and it will create a function called f which you can call with different values of m and n like
f(3,2) // returns 29

OR
try the code below in a latest Firefox

f=(m,n)=>m?f(m-1,!n||f(m,n-1)):n+1

B.onclick=_=>alert(f(+M.value, +N.value))
#M,#N{max-width:15px;border: 1px solid;border-width:0 0 1px 0}

f(<input id=M />,<input id=N />)

<button id=B>Evaluate</button>

Testing this with 0,1 in Chrome gives no result.

Pl read, this only works in latest Firefox due to ES6

Wow... we have 4 practically identical JS solutions, all at 34 bytes. I've never seen that before.

Python 2.7.8 - 80, 54, 48, 46 45
A=lambda m,n:m and A(m-1,n<1or A(m,n-1))or-~n

(Credits to xnor!)
More readable, but with 1 more character:
A=lambda m,n:n+(m<1or A(m-1,n<1or A(m,n-1))-n)

Not that I had to set sys.setrecursionlimit(10000) in order to get a result for A(3,10). Further golfing using logical indexing did not work due to the dramatically growing recursion depth.

I get a syntax error on the 1else. The start letter e causes trouble for the parser because numbers can be written like 1e3.

Saved a few chars switching to and/or: A=lambda m,n:m and A(m-1,n<1or A(m,n-1))or-~n

@xnor: Thanks for the tip! See this discussion for the parsing issue issue. Python 2.7.8 accepts 1else, most other versions don't.

Thanks for pointer about 1else; it lets me squeeze out a char here and probably other places. But darn is it version-specific! Python 2.7.4 doesn't allow it. Are you using an online version with 2.7.8 or would I have to download it?

@xnor: It's an offline installation. http://ideone.com/, e.g., fails to parse 1else as well.

@FryAmTheEggman See my answer for how to use D and how to use Pyth's ternary.

@FryAmTheEggman The two more changes needed to save 2 more characters are: 1. change <Z1 to !Z - we just want to test whether Z is 0, so using a not saves a character. 2. Use ? instead of |&, and move the conditional, Y in this case, in between the true case and the false case (just before hZ).

C - 41 bytes
Nothing to it--the small limits mean that all the required values can be calculated in less than 1 second by naively following the function definition.
A(m,n){return!m?n+1:A(m-1,n?A(m,n-1):1);}

int main()
{
    int m,n;
    for(m = 0; m <= 3; m++)
    for(n = 0; n <= 10; n++)
    printf("%d %d %d\n", m,n,A(m,n));
    return 0;
}

Javascript ES6 (34)
a=(m,n)=>m?a(m-1,n?a(m,n-1):1):n+1

Implementation:

a=(m,n)=>m?a(m-1,n?a(m,n-1):1):n+1
td[colspan="2"] input{width: 100%;}

m=	<input id="m" type="number" value="0" />

n=	<input id="n" type="number" value="0" />

<input type="button" value="Calculate!" onclick="document.getElementById('out').value=a(document.getElementById('m').value, document.getElementById('n').value)" />

<input id="out" disabled="disabled" type="text" />

Coq, 40
nat_rec _ S(fun _ b n=>nat_iter(S n)b 1)

This is a function of type nat -> nat -> nat.  Since Coq only allows the construction of total functions, it also serves as a formal proof that the Ackermann recurrence is well-founded.
Demo:
Welcome to Coq 8.4pl6 (November 2015)

Coq < Compute nat_rec _ S(fun _ b n=>nat_iter(S n)b 1) 3 10.
     = 8189
     : nat

Note: Coq 8.5, released after this challenge, renamed nat_iter to Nat.iter.

JavaScript (ES6) - 34
A=(m,n)=>m?A(m-1,!n||A(m,n-1)):n+1

And a test:
> A=(m,n)=>m?A(m-1,!n||A(m,n-1)):n+1;s=new Date().getTime();console.log(A(3,10),(new Date().getTime() - s)/1000)
8189 16.441

Jelly, 13 bytes
+‘ð;x’ß@ƒ1ðỊ?

Try it online!
Again uses proud haskeller's approach: A(m+1,n) is A(m,?) applied n+1 times to 1.
The base case should've been simply A(0,n) = n+1, but unfortunately it (the code at the bottom) times out badly, so I had to use A(m,n) = m+n+1 if m <= 1 instead.
How it works
+‘ð;x’ß@ƒ1ðỊ?    Dyadic link. Left = m, Right = n
AAðBBBBBBBðC?    If C, do A; otherwise, do B
           Ị     If m <= 1...
+‘               Return m+n+1
   ;x’ß@ƒ1       Otherwise do this:
   ;x            Concatenate m to n copies of m (which gives n+1 copies of m)
     ’           Decrement each number (n+1 copies of m-1)
        ƒ1       Reduce with seed 1...
      ß@         This function, args swapped (since new m is on the right side)

Jelly, 12 bytes, times out on A(3,9)
;x’ß@ƒ1ð‘}ḷ?

Try it online!

Mathematica, 46 bytes
0~a~n_:=n+1
m_~a~n_:=a[m-1,If[n<1,1,a[m,n-1]]]

Takes pretty much exactly a minute for a[3,10]. Note that Mathematica's default recursion limit is too small for a[3,8] and beyond (at least on my machine), but that can be fixed by configuring
$RecursionLimit = Infinity

Wow, so are you saying that JS is more than 25 times faster than Mathematica ?

@Optimizer At least when it comes to recursion... I guess part if it is that it has to figure out each time which definition to use, and If being a function is even slower.

With memoization, it takes 0.07 seconds.  I.e. m_~a~n_:=m~a~n=...

@MarkAdler That's a really nice way to do memoisation in Mathematica!

You can save a byte with an immediate assignment on the first line: 0~a~n_=n+1

Alternatively, a@0=#+1&;a@m_:=a[m-1]@If[#<1,1,a[m][#-1]]& would be 42 bytes but needs to be called as a[m][n]

I know this answer is several years old, but if any of the users in this thread are still active on this site, could you please explain how m_~a~n:=m~a~n=... achieves memoization in Mathematica? What is the ... supposed to be in this function definition?

@dirvine Just the regular value of the function, so a[m-1,If[n<1,1,a[m,n-1]]] in this particular example. How it works is that everything to the right of := is evaluated when the function is called. That is the assignment m~a~n=... which then gets stored as a special case that takes precedence over the more general definition next time the function is called with these values.

Haskell, 48 44 chars (36 for the list)
While not as short as the other Haskell solution, this one is notable because it expresses the Ackermann function as an infinite list, which I think is kinda neat.  The result is an infinite list (of infinite lists) such that at position [m,n] it holds the value A(m,n).
The infinite list itself:
iterate(tail.(`iterate`1).(!!))[1..]

As a function (to comply with the specification):
i=iterate;m%n=i(tail.(`i`1).(!!))[1..]!!m!!n

The formulation was derived by observing that the general/common case for the Ackermann function is to use the value to the left as an index in the row above.  The base case for this recursion (i.e. the leftmost column of a row, i.e. A(m,0)) is to use the second left-most value in the row above.  The base case for that recursion is the A(0,n) = n+1 case, i.e. the first row is [1..].
Thus, we get
let a0 = [1..]
let a1 = tail $ iterate (a0 !!) 1  -- 'tail' because iterate starts by applying
let a2 = tail $ iterate (a1 !!) 1  -- the function 0 times
-- etc

Then we simply add another level of iteration based on that pattern, and do some pointless juggling.

You could alias iterate to a single letter name i.e. i=iterate;ack=i ...

@proudhaskeller oh yeah, didn't think about that. Thanks! Borrowing your operator-name use as well.

Julia, 34 31 28 bytes
m\n=m>0?~-m\(n<1||m\~-n):n+1

This is a named anonymous function. It is a straightforward implementation of the recursive definition, abusing Julia's ability to redefine operators.
Try it online!

Python 3, 58 49 bytes
a=lambda m,n:m and a(m-1,n and a(m,n-1)or 1)or-~n

Try it online!
I didn't see a Python 3 answer (there was a Python 2), so I decided to fix that and put my own answer in here. (-(43+21) thanks to caird, -9 thanks to HN)

78 bytes

57 bytes

49 bytes

I was going to mention that n<1or would save even more bytes but then it becomes identical to the aforementioned Python 2 answer...

Racket 67
(define(a m n)(if(= m 0)(+ n 1)(a(- m 1)(if(= n 0)1(a m(- n 1))))))

Javascript with lambdas, 34
A=(m,n)=>m?A(m-1,n?A(m,n-1):1):n+1

A tipical answer, can't make anything shorter.

Tiny Lisp, 70  (out of competition)
This runs out of competition, as the language is newer than the question, and it also doesn't succeed to run the (A 3 10) as required in the question, due to a stack overflow.
(d A(q((m n)(i m(i n(A(s m 1)(A m(s n 1)))(A(s m 1)1))(s n(s 0 1))))))
This defines a function A which calculates the Ackermann function.
Formatted:
(d A
   (q( (m n)
       (i m
          (i n
             (A (s m 1)
                (A m
                   (s n 1)
                 )
              )
             (A (s m 1)
                1
              )
           )
          (s n
             (s 0 1)
           )
        )
    ) )
 )

We are using all builtin macros (d (define) and q (quote) and i (if)) and one builtin function (s – subtract) here.
i executes its true part when the condition is a number > 0 (and otherwise the false part), so we don't have to do an explicit comparison here.
s is the only arithmetic operation available, we use it for the n-1/m-1, as well as as (s n (s 0 1)) for n+1.
Tiny lisp is using tail recursion optimization, but this only helps for the outer A call in the result, not for the A(m, n-1) call which is used for the parameters.
With my tiny lisp implementation in Ceylon on the JVM, it works up to (A 3 5) = 253, but it seems to break down when trying to calculate (A 2 125) directly (which should give the same result). If calculating that after (A 3 4) = 125, the JVM seems to got to optimize the functions enough to inline some intermediate function calls in my interpreter, allowing more recursion depth. Strange.
The reference implementation gets up to (A 3 5) = 253 and also (A 2 163) = 329, but doesn't succeed (A 2 164), and therefore even less (A 3 6) = (A 2 253).

this might  be competitive save for the whitespace and parenthesis ;)

brainfuck, 90 bytes
>>>>+>,>,<<[>[>[-[->>>+<<<]<[->+>>+<<<]>-[-<+>]>+>>>>>]<[->+>>]]<[>>+[-<<<+>>>]<<-]<<<]>>.

Try it online!
Assumes an implementation with arbitrary sized cell size, with IO as numbers. -6 bytes if you don't mind using negative cells.
Finishes in about 30 seconds for 3,8 in the linked interpreter, provided you tick the correct settings. Type inputted numbers prepended with \s, e.g. 3,9 is \3\9.

Ral, 81 61 bytes
,,/:11+-111:++:+:++:+:+?/0=:1/-/1/10*-0*1:+?+0=11:+?1++/1:+?.

Try it online!
Commented
Code                     Stack after        Description
,,                       0, m, n              Input m and n
                                            Loop:
/:11+-111:++:+:++:+:+?   r, n, m              Jump to SmallM if m < 2
/0=:1/-/1/10*-           r, m-1, 1, m, n-1    Perform [m, n] => [m-1, 1, m, n-1]
0*1:+?                   r, m-1, 1, m, n-1    Jump to Loop if n>0
+0=                      r, m-1, 1            Remove the last two elements
11:+?                    r, m-1, 1            Jump to Loop
                                            SmallM:
1++                      n+m+1, r             Perform [m, n] => [n+m+1]
/1:+?                    n+m+1                Jump to Loop if r>0
.                                             Print n+m+1

Explanation
Because there are no built in function calls (or recursion) in Ral, we need to simulate the stack frame. One way to do that is as follows:

Start with the stack [m, n]
Repeatedly apply the following rules, until a single element remains:

[..., 0, n] => [..., n+1]
[..., m, 0] => [..., m-1, 1]
[..., m, n] => [..., m-1, m, n-1]

Print the remaining element

However, since there is no way to know the current length of the stack in Ral, an additional flag is stored to test if the recursion should continue, resulting in the following rules:

[..., 0, 0, n] => Print n+1 and exit
[..., 1, 0, n] => [..., n+1]
[..., m, 0] => [..., m-1, 1]
[..., m, n] => [..., m-1, 1, m, n-1]

K (ngn/k), 16 bytes
{x/[;1]1+}/[;1+]

Try it online!
Same structure as my K answer for fast growing hierarchy, and happens to be the same length too. Can compute f(0..3)(0..10) in a couple seconds.
Again uses "A(m+1,n) is A(m,?) applied n+1 times to 1."
{x/[;1]1+}/[;1+]    input: m, returns function that takes n
{        }/[;1+]    iterate m times on the function 1+ (increment):
       1+             create a function that increments n and then
 x/[;1]               run `x/[n+1;1]` == `(n+1)x/1`
                      == iterate function x n+1 times to 1

APL, 31
{⍺=0:⍵+1⋄⍵=0:1∇⍨⍺-1⋄(⍺-1)∇⍺∇⍵-1}

Pretty straightforward. Uses the ⍨ character once to save one byte by reversing arguments. Takes m as the left argument and n as the right argument.
TryAPL.org

R - 54 52
I've used this as an excuse to try and get my head around R, so this is probably really badly done:)
a=function(m,n)"if"(m,a(m-1,"if"(n,a(m,n-1),1)),n+1)

Example run
> a(3,8)
[1] 2045

I get a stack overflow for anything beyond that
T-SQL- 222
I thought I would try to get T-SQL to do it as well.  Used a different method because recursion isn't that nice in SQL.  Anything over 4,2 bombs it.
DECLARE @m INT=4,@n INT=1;WITH R AS(SELECT 2 C, 1 X UNION ALL   SELECT POWER(2,C),X+1FROM R)SELECT IIF(@m=0,@n+1,IIF(@m=1,@n+2,IIF(@m=2,2*@n+3,IIF(@m=3,POWER(2,@n+3)-3,IIF(@m=4,(SELECT TOP(1)C FROM R WHERE x= @n+3)-3,-1)))))

for your R submisson, looks like you don't need the {} although there's no helping the stack overflow limit, since R doesn't have TCO...

@Giuseppe thanks ... in my defense, I was new to it then :)

Go, 260 243 240 122 bytes
I didn't see that the question allowed anon funcs.
far from competitive but i'm learning this language and i wanted to test it out.

func (m,n int)int{r:=0
switch{case m==0&&n!=0:r=n+1
case m!=0&&n==0:r=a(m-1,1)
case m!=0&&n!=0:r=a(m-1,a(m,n-1))}
return r}
use it like go run ack.go and then supply two numbers, m and n. if m>4 or n>30, execution time may be in excess of half a minute.
for m=3 n=11:

$ time go run ack
16381
real    0m1.434s
user    0m1.432s
sys     0m0.004s

edit: saved total 17 bytes by switching to switch over if/else and dot-imports

You can do even better! switch 0 {case m:r=n+1 case n:r=a(m-1,1) default:r=a(m-1,a(m,n-1))} Go's switch statement is so beautifully flexible!

@EMBLEM Thanks, it's been so long since I've written a line of Go, but I'm glad to see there are other Go-golfers about :D

Ceylon, 88 87 85
alias I=>Integer;I a(I m,I n)=>m<1then n+1else(n<1then a(m-1,1)else a(m-1,a(m,n-1)));

This is a straightforward implementation. Formatted:
alias I => Integer;
I a(I m, I n) =>
        m < 1
        then n + 1
        else (n < 1
            then a(m - 1, 1)
            else a(m - 1, a(m, n - 1)));

The alias saves just one byte, without it (with writing Integer instead of I) we would get to 86 bytes. Another two bytes can be saved by replacing == 0 by < 1 twice.
With the default settings of ceylon run, it will work up to A(3,12) = 32765 (and A(4,0) = 13), but A(3,13) (and therefore also A(4,1)) will throw a stack overflow error. (A(3,12) takes about 5 seconds, A(3,11) about 3 on my computer.)
Using ceylon run-js (i.e. running the result of compiling to JavaScript on node.js) is a lot slower (needs 1 min 19 s for A(3,10)), and breaks already for A(3, 11) with a »Maximum call stack size exceeded« (using default settings) after running for 1 min 30 s.

Ceylon without recursion, 228
As a bonus, here is a non-recursive version (longer, of course, but immune to stack overflows – might get an out-of-memory error at some point).
import ceylon.collection{A=ArrayList}Integer a(Integer[2]r){value s=A{*r};value p=s.addAll;while(true){if(exists m=s.pop()){if(exists n=s.pop()){if(n<1){p([m+1]);}else if(m<1){p([n-1,1]);}else{p([n-1,n,m-1]);}}else{return m;}}}}

Formatted:
import ceylon.collection {
    A=ArrayList
}

Integer a(Integer[2] r) {
    value s = A { *r };
    value p = s.addAll;
    while (true) {
        if (exists m = s.pop()) {
            if (exists n = s.pop()) {
                if (n < 1) {
                    p([m + 1]);
                } else if (m < 1) {
                    p([n - 1, 1]);
                } else {
                    p([n - 1, n, m - 1]);
                }
            } else {
                // stack is empty
                return m;
            }
        }
    }
}

It is quite slower on my computer than the recursive version: A(3,11) takes 9.5 seconds, A(3,12) takes 34 seconds, A(3,13) takes 2:08 minutes, A(3,14) takes 8:25 minutes. (I originally had a version using lazy iterables instead of the tuples I now have, which was even much slower, with the same size).
A bit faster (21 seconds for A(3,12)) (but also one byte longer) is a version using s.push instead of s.addAll, but that needed to be called several times to add multiple numbers, as it takes just a single Integer each. Using a LinkedList instead of an ArrayList is a lot slower.

UGL, 31 30 bytes
iiRuldr%l%lR$d%rd:u%d:%+uRu:ro

Input separated by newlines.
Try it online!
(It has been implemented as a standard example in the interpreter.)

Pyth, 15 bytes
M?GgtG?HgGtH1hH

Try it online! (sample usage of the function g3T added, which means g(3,10))

Tcl, 67 bytes
proc tcl::mathfunc::A m\ n {expr {$m?A($m-1,$n?A($m,$n-1):1):$n+1}}

Try it online!

Tcl, 77 bytes
proc A m\ n {expr {$m?[A [expr $m-1] [expr {$n?[A $m [expr $n-1]]:1}]]:$n+1}}

Try it online!

In the online compiler it fails to run due to time-out, but in a local Tcl interpreter it runs well. I profiled of each root call to A function, to see how much time the calculation took for each pair {m,n} subject to be tested:
m=0, n=0, A=1, time=3.5e-5 seconds
m=0, n=1, A=2, time=2e-6 seconds
m=0, n=2, A=3, time=8e-6 seconds
m=0, n=3, A=4, time=1e-6 seconds
m=0, n=4, A=5, time=2e-6 seconds
m=0, n=5, A=6, time=1e-6 seconds
m=0, n=6, A=7, time=1e-6 seconds
m=0, n=7, A=8, time=1e-6 seconds
m=0, n=8, A=9, time=1e-6 seconds
m=0, n=9, A=10, time=0.0 seconds
m=0, n=10, A=11, time=1e-6 seconds
m=1, n=0, A=2, time=4e-6 seconds
m=1, n=1, A=3, time=6e-6 seconds
m=1, n=2, A=4, time=1e-5 seconds
m=1, n=3, A=5, time=1.2e-5 seconds
m=1, n=4, A=6, time=1.5e-5 seconds
m=1, n=5, A=7, time=2e-5 seconds
m=1, n=6, A=8, time=2e-5 seconds
m=1, n=7, A=9, time=2.6e-5 seconds
m=1, n=8, A=10, time=3e-5 seconds
m=1, n=9, A=11, time=3e-5 seconds
m=1, n=10, A=12, time=3.3e-5 seconds
m=2, n=0, A=3, time=8e-6 seconds
m=2, n=1, A=5, time=2.2e-5 seconds
m=2, n=2, A=7, time=3.9e-5 seconds
m=2, n=3, A=9, time=6.3e-5 seconds
m=2, n=4, A=11, time=9.1e-5 seconds
m=2, n=5, A=13, time=0.000124 seconds
m=2, n=6, A=15, time=0.000163 seconds
m=2, n=7, A=17, time=0.000213 seconds
m=2, n=8, A=19, time=0.000262 seconds
m=2, n=9, A=21, time=0.000316 seconds
m=2, n=10, A=23, time=0.000377 seconds
m=3, n=0, A=5, time=2.2e-5 seconds
m=3, n=1, A=13, time=0.000145 seconds
m=3, n=2, A=29, time=0.000745 seconds
m=3, n=3, A=61, time=0.003345 seconds
m=3, n=4, A=125, time=0.015048 seconds
m=3, n=5, A=253, time=0.059836 seconds
m=3, n=6, A=509, time=0.241431 seconds
m=3, n=7, A=1021, time=0.971836 seconds
m=3, n=8, A=2045, time=3.908884 seconds
m=3, n=9, A=4093, time=15.926341 seconds
m=3, n=10, A=8189, time=63.734713 seconds

It fails for the last pair {m,n}={3,10}, as it takes a very little more than one minute.
For higher values of m, it will be needed to increase the recursionlimit value.

I coult get it shorter to 65 bytes, but it will not meet the question's requirement "Your function must be able to find the value of A(m,n) for m ≤ 3 and n ≤ 10 in less than a minute.". Without the {} it will timeout on TIO and not do the demo of the last two entries.
Tcl, 65 bytes
proc tcl::mathfunc::A m\ n {expr $m?A($m-1,$n?A($m,$n-1):1):$n+1}

Try it online!

Io, 53 bytes
Yet another recursion solution.
a :=method(m,n,if(m<1,n+1,a(m-1,if(n<1,1,a(m,n-1)))))

Try it online!

C (gcc), 36 bytes
A(m,n){m=m?A(m-1,n?A(m,n-1):1):n+1;}

Naive strategy with a little bit of optimization.
Try it online!

Verilog (Icarus Verilog), 81 bytes
Based on the C answer. Verilog is not really intended for this kind of thing - Yosys crashes on this - but it's possible with Icarus.
function automatic[15:0]a;input[15:0]m,n;a=(m==0)?n+1:(n==0)?a(m-1,1):a(m-1,a(m,n-1));endfunction

Testbench:
module test();

function automatic[31:0]a;input[31:0]m,n;a=m?a(m-1,n?a(m,n-1):1):n+1;endfunction

initial begin
    $display("%d", a(3,10));
end

endmodule

Icarus is really not intended for this kind of constant abuse, though:
$ time ./a.out
 8189

real    0m55.422s
user    0m55.375s
sys     0m0.063s

Haskell, 47 bytes
a 0 n=n+1
a m 0=a(m-1)1
a m n=a(m-1)(a m (n-1))

(Run using ghci <filename>)
Almost directly copies from the problem definition (thank you pattern matching!).

You may want to include a TIO link so everyone can test your submission?

I think a(m-1)(a m (n-1)) can be a(m-1).a m$n-1.

APL (Dyalog Unicode), 22 bytes
{0=⍺:1+⍵⋄1∇⍣(⍵+1)⍨⍺-1}

Try it online!
A port of proud haskeller's Haskell answer. Uses the fact that Ack(m,n) is equal to Ack(m-1,?) applied n+1 times to 1.
How it works
{0=⍺:1+⍵⋄1∇⍣(⍵+1)⍨⍺-1}
{                    }  ⍝ ⍺←m, ⍵←n
 0=⍺:1+⍵⋄               ⍝ If m = 0, return 1 + n
          ∇      ⍨⍺-1   ⍝ Apply (m-1)∇ (∇ is recursive call)
           ⍣(⍵+1)       ⍝ n+1 times
         1              ⍝ to the initial value 1

MMIX, 56 bytes (14 instructions)
5A000003 23000101 F8010000 27000001 5A010003 E3010001 F1FFFFFA
FE020004 23040001 27050101 FB03FFF6 F6040002 C9010003 F1FFFFF3

Disassembly and explanation
ack PBNZ  $0,0F     // if(!m) {
    ADDU  $0,$1,1   //   m = n + 1
    POP   1,0       //   return m }
0H  SUBU  $0,$0,1   // m = m - 1
    PBNZ  $1,0F     // if(!n) {
    SETL  $1,1      //   n = 1
    JMP   ack       //   return ack(m,n) }
0H  GET   $2,rJ     // x = retaddr (to avoid trampling)
    ADDU  $4,$0,1   // m' = m + 1
    SUBU  $5,$1,1   // n' = n - 1
    PUSHJ $3,ack    // a = ack(m',n')
    PUT   rJ,$2     // retaddr = x
    SET   $1,$3     // n = a
    JMP   ack       // return ack(m,n)

There's hefty use here of tail recursion optimization, not that that'll help much (four words of stack are used here for every time we don't tail recurse, and the stack segment only holds \$2^{58}\$ words).

Jelly, 17 bytes
’çç’}¥ɗ’ç1Ɗṛ?‘}ḷ?

Try it online!
Requires a higher recursion limit for \$A(3, 10)\$ than the Jelly interpreter has, but finishes within 1 minute when I adjust the limit locally
Full program, due to some bugs with ß.
How it works
’çç’}¥ɗ’ç1Ɗṛ?‘}ḷ? - Main link A(m, n). Takes m on the left and n on the right
                ? - If:
               ḷ  -   Condition: m is non-zero
                  -   Then:
            ?     -     If:
           ṛ      -       Condition: n in non-zero
      ɗ           -       Then:
’                 -         m-1
     ¥            -         Group the previous 2 links into a dyad f(m, n):
   ’}             -           n-1
  ç               -           A(m, n-1)
 ç                -         A(m-1, A(m, n-1)
          Ɗ       -       Else:
       ’          -         m-1
        ç1        -         A(m-1, 1)
              }   -   Else:
             ‘    -     n+1

I wonder if it's possible to adjust the limit via the python eval command

@Razetime Unfortunately, the problem on TIO is that Jelly segfaults if it recurses too much: Try it online!, whereas it doesn't when run locally

J : 50
>:@]`(1$:~<:@[)`(<:@[$:[$:_1+])@.(0>.[:<:@#.,&*)M.

Returns in a fraction of a second for 0...3 vs 0 ... 10:
   A=:>:@]`(1$:~<:@[)`(<:@[$:[$:_1+])@.(0>.[:<:@#.,&*)M.
   timespacex 'res=:(i.4) A"0 table (i.11)'
0.0336829 3.54035e6
   res
┌───┬──────────────────────────────────────────┐
│A"0│0  1  2  3   4   5   6    7    8    9   10│
├───┼──────────────────────────────────────────┤
│0  │1  2  3  4   5   6   7    8    9   10   11│
│1  │2  3  4  5   6   7   8    9   10   11   12│
│2  │3  5  7  9  11  13  15   17   19   21   23│
│3  │5 13 29 61 125 253 509 1021 2045 4093 8189│
└───┴──────────────────────────────────────────┘

PS: the "0 serves to make A work on each single element, instead of gobbling up the left and right array, and generating length errors. But it's not needed for eg. 9 = 2 A 3 .

http://codegolf.stackexchange.com/a/40174/48934 You have been beaten!

Haskell: 81 69 bytes
a::Int->Int->Int
a 0 n=n+1
a m 0=a (m-1) 1
a m n=a (m-1) a m (n-1)

a 3 10 takes about 45 seconds.

this is code golf, so you should try to have the shortest code possible. for example, remove unnecessary spaces and the explicit type

you're also missing the parens on the fourth line

Ruby, 65
h,a={},->m,n{h[[m,n]]||=m<1?(n+1):(n<1?a[m-1,1]:a[m-1,a[m,n-1]])}

Explanation
This is a pretty straightforward translation of the algorithm given in the problem description.

Input is taken as the arguments to a lambda. Two Integers are expected.
For speed and avoiding stack-overflow errors, answers are memoized in the Hash h. The ||= operator is used to calculate a value that wasn't previously calculated.

a[3,10] is calculated in ~0.1 sec on my machine.
Here's an ungolfed version
h = {}
a = lambda do |m,n|
  h[[m,n]] ||= if m < 1
    n + 1
  elsif n < 1
    a[m-1,1]
  else
    a[m-1,a[m,n-1]]
  end
end

a[3,10] throw a SystemStackError on my machine...

Golf nitpicks: You could change m<1?(n+1):(n<1?a[m-1,1]:a[m-1,a[m,n-1]]) to m<1?n+1:a[m-1,n<1?1:a[m,n-1]]

Mouse-2002, 99 83 bytes
$Y1%j:j.0=m:2%k:k.0=n:m.n.>[k.1+!|m.n.<[#Y,j.1-,1;|m.n.*0=[#Y,j.1-,#Y,j.,k.1+;;]]]@

Java, 274 bytes
import java.math.*;class a{BigInteger A(BigInteger b,BigInteger B){if(b.equals(BigInteger.ZERO))return B.add(BigInteger.ONE);if(B.equals(BigInteger.ZERO))return A(b.subtract(BigInteger.ONE),BigInteger.ONE);return A(b.subtract(BigInteger.ONE),A(b,B.subtract(BigInteger.ONE)));}}

It calculates A(3,10) in a few seconds, and given infinite memory and stack space, it can calculate any combination of b and B as long as the result is below 22147483647-1.

I know it's been a while, but you can golf this to 185 bytes: import java.math.*;BigInteger A(BigInteger b,BigInteger B){return b.equals(B.ZERO)?B.add(B.ONE):B.equals(B.ZERO)?A(b.subtract(B.ONE),B.ONE):A(b.subtract(B.ONE),A(b,B.subtract(B.ONE)));}

Erlang (escript), 53 bytes
I tried to put the pattern matching in an if statement; turns out that makes the program longer...
a(0,N)->N+1;a(M,0)->a(M-1,1);a(M,N)->a(M-1,a(M,N-1)).

Try it online!

Japt, 12 bytes
?ßUÉ!V´ªß:VÄ

Try it
?ßUÉ!V´ªß:VÄ     :Implicit input of integers U=m & V=n
?                :If U is not 0
 ß               :Recursive call with arguments
  UÉ             :  U-1 and
    !V´          :  Logical NOT of postfix decremented V
       ª         :  Logical OR with
        ß        :  Recursive call with arguments U and the now decremented V
         :VÄ     :Else return V+1

Python 3.11, 96 bytes
def a(m,n):
    if m==0:return n+1
    elif n==0:return a(m-1,1)
    else:return a(m-1,a(m,n-1))

Ungolfed:
def a(m, n):
    if m == 0:
        return n + 1
    elif n == 0:
        return a(m-1, 1)
    else:
        return a(m-1, a(m, n-1))

Pascal, 110 B
To prevent death-by-boredom victims, the boilerplate recursive definition has been omitted.
Using the boilerplate recursive definition as reference, you can save 1 byte in code length by doing the following:

Presume the m = 0 case the default case.
Do not wrap it in an if statement, just calculate A ≔ n + 1 (and possibly overwrite this value).
This necessitates a different expression for the m ≠ 0 ∧ n = 0 situation.
The following code uses sets: [n] − [m] = [0].
It means, take the integer set consisting of the single element n, remove from it all elements found in the second set ([m]) and compare the resulting set value against the set [0].
The fallback can be phrased as n × m > 0.
Recall that any product is zero as soon as at least one factor is zero.
Phrasing the fallback like this is necessary, because there are no else clauses.

function A(m,n:integer):integer;begin
A:=n+1;if[n]-[m]=[0]then A:=A(m-1,1);if n*m>0 then A:=A(m-1,A(m,n-1))end

NB:
The limits of integer are implementation-defined.
For example, the GNU Pascal Compiler implementation will require the ‑‑setlimit=8189 parameter (or larger) to work correctly for \$m \leq 3\$, \$n \leq 10\$ test cases (beside the fact of having sufficient memory).

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Find a representative submatrix
You have a matrix of size m x n.
Each cell in the matrix has a uniformly random integer value v, where 0 ≤ v < i, and i ≤ (m x n). (Meaning, the matrix contains a maximum of m x n distinct values, but it may have fewer.)
1) Write a function that accepts input values of m, n, i, and returns a matrix that meets the criteria above.
2) Write a second function that accepts the output of the first function as input, and returns the smallest contiguous submatrix that contains every value of v (by returning the submatrix itself, or its coordinates). By “smallest” I mean the submatrix containing the fewest cells.
(If it helps you visualize, the original inspiration for this problem was wondering how to find the smallest rectangle within a GIF that contained all the colors in its palette.)

This needs some clarification about the randomness. Clearly the random variables aren't independent, because that would conflict with the requirement that every possible value of v appear at least once. So what are the permitted distributions?

@PeterTaylor: As I understand it, you could for example first place the values from 0 to i−1 randomly on the matrix, then fill up the remaining elements with random numbers from 0 to i−1. (Or anything that produces the same distribution of matrices.)

@Wrzlprmft, probably. But could you also just make all the matrices generated by your program be permutations of the first mn characters from an infinite sequence which just repeats 0 to i?

@weston When i = mn, the second part of the question is trivial, so i ≤ mn is deliberate.

Smallest submatrix regarding which metric? Product of dimensions?

OK, I’ve simplified it. Each cell gets a random integer in the range 0 .. i-1. So over repeated trials, we would expect the occurrence of each value to converge on mn / i, just like repeated coin flips or die rolls.

APL (23 + 56 = 79)
Function 1 (23):
{¯1+⍵⍴(V⍴⍺?⍺)[V?V←×/⍵]}

This takes i as its left argument and m n as its right argument, like so:
      +mat←8 {¯1+⍵⍴(V⍴⍺?⍺)[V?V←×/⍵]} 5 5
2 4 7 1 3
7 5 6 7 3
3 1 2 4 6
0 4 2 6 1
5 5 0 0 6

Function 2 (56):
{⊃V[⍋≢∘∊¨V←V/⍨{∧/⍵∊⍨∪∊M}¨V←⊃,/{↓∘⍵¨¯1+,⍳⍴⍵}¨↑∘⍵¨,⍳⍴M←⍵]}

This takes the output from the first function as its right argument, like so:
      {⊃V[⍋≢∘∊¨V←V/⍨{∧/⍵∊⍨∪∊M}¨V←⊃,/{↓∘⍵¨¯1+,⍳⍴⍵}¨↑∘⍵¨,⍳⍴M←⍵]} mat
7 5 6
3 1 2
0 4 2

This makes me want to learn APL.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
Jumblers vs Rebuilders: Coding with Tetris Bricks - Area for Rebuilder Answers
This is the companion question to Jumblers vs Rebuilders: Coding with Tetris Bricks where Rebuilders can post their solutions.
As argued in Where should we put robbers? it allows more formatting freedom for robbers and lets them gain rep.
Only answer this question if you have cracked a yet uncracked answer to Jumblers vs Rebuilders: Coding with Tetris Bricks. Please include a link to that answer in your answer and a comment in that answer linking to your answer.
The accepted answer for this question will be the winning rebuilder.
The scoreboard will only appear in the other question.
COTO has generously made a JSFiddle for easily manipulating code-bricks. I've updated this fiddle so ES 6 is not required, and minimized it for this Stack Snippet:

Put code-bricks here and hit OK. Re-run the snippet to restart.
(You may need to replace spaces in code-bricks with some other character first.)
<textarea id='spec' rows='16' cols='80'>ABCD  a
      b   Oo
      c   oo
      d

 E    h
 F  efg   hg   GFE
 GH        f   H
           e

 I           IJK
 J   l         L
LK   kji  kl
          j
          i
                   t
 OP   p     QR    rs
MN    on     ST   q
       m
 W             z
 XY    zxw    yx     Y
 Z      y      w    WXZ</textarea>
<button id='go' type='button' onclick='go()'>OK</button>

@11	@22	@33	@44

@11

@22

@33

@44

@40	@30	@20	@10

@40

@30

@20

@10

	@11

	@22

@33	@44

@31

@42	@23	@14

@41	@32

@23

@14

@11	@22	@43

		@34

@11	@22

@33	@44

@30	@10

@40	@20

@40	@30

@20	@10

@20	@40

@10	@30

@11

@22

@33	@44

@31	@22	@13

@44

@41	@32

	@23

	@14

		@41

@12	@23	@34

	@21	@12

@43	@34

@41

@32	@23

	@14

	@30	@40

@10	@20

@10

@20	@30

	@40

@11	@22

	@33	@44

	@11

@32	@23

@44

@40	@30

	@20	@10

	@40

@20	@30

@10

@11	@22	@33

	@44

	@11

@42	@23

	@34

	@41

@32	@23	@14

@31

@22	@43

@14

Crack of xnor's 340-area answer
No cryptographic mumbo-jumbo for me.
Code:
10000001111111222
22388892357889356
69968999352143221
80867227952780811
a=23333333444444
b=71122368066667
c=47948316823798
d=79254432699451
496857235742
579159386011
606961806231
pow(a,b,c)-d
62656269644465648
67766884711672530
73487440752777431
print('Tetris'*(1
))
7873806482918808
8843930593249458
946095869869

Mapping of blocks to line numbers
    #Four spaces 9-12
    #Four spaces 9-12
    #Four spaces 9-12
    #Four spaces 9-12
    #Four spaces 17
    #Four spaces 17
    #Four spaces 17
    #Four spaces 17-20
    #Four spaces 5-8
))  #Two spaces 17
0290 1-4
0398 1-4
0866 1-4
0887 1-4
0892 1-4
0992 1-4
1108 13-16
1268 1-4
1297 1-4
1339 1-4
1555 1-4
1722 1-4
1817 1-4
1848 1-4
1930 1-4
2328 1-4
2521 1-4
2611 1-4
2747 5-8
3179 5-8
3192 5-8
3245 5-8
3284 5-8
3334 5-8
3613 5-8
3862 5-8
4086 5-8
4629 5-8
4639 5-8
4674 5-8
4695 5-8
4781 5-8
4968 9
5723 9
5742 9
5791 10
5938 10
6011 10
6069 11
6180 11
6231 11
6265 13
6269 13
6444 13
6564 13
6776 14
6884 14
7116 14
7253 14
7348 15
7440 15
7527 15
7743 15
7873 17
8064 17
8291 17
8808 17
8843 18
9305 18
9324 18
9458 18
9460 19
9586 19
9869 19
==== 5-8
a,b, 12
abcd 5-8
c)-d 12
etri 16
pow( 12
prin 16
s'*( 16
t('T 16
four spaces  9-12
four spaces  20

Solution to xnor's 212 area puzzle
print('Tetris'* (3580048039607960824098110131314133014381502228523172479258529553116373838184169435643604632480050165153525653945598563157585840631264256539704571638903974876749009825073297156762481649%38941394))

Teromino Sandbox
While not a solution ipso facto, I wanted to share this online tool I put together with other potential rebuilders. It facilitates quick and error-free manipulation of the "code tetrominos".
The shapes are specified using a simple character grid in the spec element. Most answers should be a simple copy/paste off of PPCG. The default shape spec provides numerous examples. I've also included the shape specs for my C-72 answer.
Two caveats:

the specs are whitespace-sensitive
some HTML control characters may need to be escaped

The pieces are shown in different colours. Drag them to move them, and click them to rotate them. Easy peasy.
Anyone who wants to add improvements, add new specs, fix bugs, etc., is welcome to fork the fiddle and edit this answer to provide an updated link.
Happy rebuilding. :)

Is there something strange I need to do to get this to work? It doesn't do anything for me in any of the three browsers I have installed.

@Geobits: It does use numerous ES6 features. I'm running it in the latest FF without problems. Let me test it in Chrome.

Ha! Apparently I don't often update/use Firefox. It was version 22. Latest works fine.

Chrome 38 seems to be choking on the lambda expressions. I didn't realize it couldn't handle them. :O   IE I knew couldn't handle them, but... who uses IE. :D

I've made the fiddle compatible with Chrome and put it into a Stack Snippet in the question. I hope you don't mind.

@Calvin'sHobbies: Not at all. Thanks for making the fix.

Solution to es1024's 80 area answer
main(q){char c[]
={'/T','$"e','t'
,0162,'rIi>`:i',
'?#>s',q<q/**/};
return puts(c);}

Solution to Gerli's 72 area submission
FromCharacterCode[
 {8*10+4, 10*10+1,
 100+16, 10*10+14,
 105, 115}]//Print

There's a mistake in the submission though. In the third T block, Gerli "rotated" the [ and { when rotating the block.
All done by hand. The Print was quite easy to find. Then I wrote out the remaining letters, and found them to be an anagram of FromCharacterCode. That put a lot of restrictions on the maximum code height and most blocks. Since there was no @ and only one pair of brackets, that gave away that Print was applied using suffix notation (which determined both its position and that it's preceded by the //. In the end there were only an S and two Ts left. The S was forced by the layout and the first assignment of Ts I tried worked.


text:
Circle packing in a rectangle
Your task is to write a program that finds the largest radius that N circles can have and still fit inside of a rectangle that is X by Y pixels large. (similar to this wikipedia article) Your program must find the largest possible radius and optimal position of these  N circles so that

No two circles overlap
All of the circles fit inside of the rectangle.

Your program should then print this to the console:
Highest possible radius: <some_number>
Circle 1 Focus: (x, y)
Circle 2 Focus: (x, y)
...
Circle N-1 Focus: (x, y)
Circle N Focus: (x, y)

(Obviously, you must replace some_number with the radius your program calculates, N with the number of circles you end up using, and x and y with the actual coordinates of the circle)
Lastly, it should create and save a picture with these circles drawn.
Rules:

This program must run with any size of rectangle, and any number of circles and still get a valid answer. You can use command line arguments, user input, hard coded variables, or anything else you want.
If any circle overlaps or does not fit entirely inside of the box, your submission is invalid.
Each circle must have the same radius.
Just to make this reasonable and do-able, all numbers must be precise to the 2nd decimal place.

This is code golf, so shortest program (as of 10/28/2014) wins.

FYI doing this optimally is not exactly easy. "Solutions (not necessarily optimal) have been computed for every N≤10,000."

Would you allow a solution that iterates over every possibility? By which I mean every possible center and radius for each circle up to  machine precision, not every qualitative configuration.

Not to mention that if the program must run with any number of circles, the sample output would require it to be able to produce an English name for any number.

Fair point Martin, let me make the output more specific. Also, I said 300x300 with 5 circles because it's kind of confusing to say "Some number of circles with some size of rectangle" I'll make it a little more clear.

xnor, sure that will do fine.

Did you mean "to the second decimal place"? Because the hundredth seems neither very reasonable nor very doable. ;) Although, if you only want so little precision we might as well do everything in integers. We can't really plot it more accurately anyway if 1 unit corresponds to 1 pixel.

This challenge is literally impossible, at least in 2014. The optimal solution to this problem is not known to mathematics, so it will be completely impossible to check whether answers are valid or not. I think it might be possible to save this with a little thought, e.g. by saying that your program becomes invalid if anyone finds a better solution for any values of x, y and N. Or just making it [tag:code-challenge], with the score based on the largest radius your program can find. But as it's written now there's no way you can possibly get any valid answers.

So what do you propose I do? Leave it up as one of the impossible challenges that never get solved, Like this one? Edit the heck out of this until it becomes do-able? Scratch it and start over? I can't put it in the sand box, as I'm < 50 rep.

I'm a bit confused about the precision. You say "rectangle that is X by Y pixels large". Is that simply meant to specify a dimension of XY, or is the pixel the minimum unit of granularity (so that circles must be centered on a pixel)? Or does "all numbers must be precise to the 2nd decimal place" mean that there's X100 and Y*100 possible center locations? Or is that sentence simply saying how we should give our output?

@user3524982 my suggestion was to limit it to 5 circles. That is doable. The solution is to arrange the circles in an X for squarish rectangles, and in a W for long rectangles (which ends up as a straight linde for l/w>5.) The interesting part is the transition from the X arrangement to the W arrangement, which occurs over a very small range of l/w ratios. I deleted the suggestion because it didn't get any responses so I felt it would confuse more. Then the question was closed and reopened but the issue of complexity per the current spec remains.

@user3524982 I believe you only need 5 rep and not 50 for the sandbox. 50 is for commenting on others' posts.

I think this question should be closed until it's modified. Currently, there are two conflicting winning objectives: code length and optimal positioning/radii. Since optimality has not been proven for large N or X by Y rectangle, this question is essentially unanswerable correctly.

In this site he calculated the maximum number of circles in a square using two methods , normal method and triangle method, the question is how he calculated the number of circles using triangle method.. https://www.engineeringtoolbox.com/circles-within-rectangle-d_1905.html

JavaScript 782 725 characters
first post, be gentle!
The program is now called via the wrapped function. Eg: (function(e,f,g){...})(100,200,10).
function C(e,f,g,c,a,d){if(0>g-a||g+a>e||0>c-a||c+a>f)return d;for(var b in d)if(Math.sqrt(Math.pow(d[b].x-g,2)+Math.pow(d[b].y-c,2))<2*a)return d;d.push({x:g,y:c});for(b=0;b<Math.PI;)XX=Math.cos(b)*a*2+g,YY=Math.sin(b)*a*2+c,d=C(e,f,XX,YY,a,d),b+=.01;return d}
(function(e,f,g){var c=e+f,a,d;for(a=[];a.length<g;)a=d=c,a=C(e,f,a,d,c,[]),c-=.01;console.log("Highest possible radius: "+Math.round(100*c)/100);e='<svg width="'+e+'" height="'+f+'"><rect width="'+e+'" height="'+f+'" style="fill:red" />';for(var b in a)console.log("Circle "+b+" Focus: ("+Math.round(100*a[b].x)/100+", "+Math.round(100*a[b].y)/100+")"),e+='<circle cx="'+a[b].x+'" cy="'+a[b].y+'" r="'+c+'" fill="blue" />';console.log(e+"</svg>")})(400,300,13);

Test 1
(function(e,f,g){...})(200,200,4)
Highest possible radius: 49.96
Circle 1 Focus: (49.97, 49.97)
Circle 2 Focus: (149.91, 49.97)
Circle 3 Focus: (149.99, 149.91)
Circle 4 Focus: (50.05, 149.91)
<svg width="200" height="200"><rect width="200" height="200" style="fill:blue;" /><circle cx="49.97000000021743" cy="49.97000000021743" r="49.960000000217434" fill="white" /><circle cx="149.9100000006523" cy="49.97000000021743" r="49.960000000217434" fill="white" /><circle cx="149.98958489212322" cy="149.90996831285986" r="49.960000000217434" fill="white" /><circle cx="50.04958489168835" cy="149.90996831285986" r="49.960000000217434" fill="white" /></svg>

Obviously we would expect the radius to be exactly 50, but for reasons discussed in the question's comments, I couldn't reasonably make it happen. The SVG looks like this...

Test 2
(function(e,f,g){...})(100,400,14)
Highest possible radius: 26.55
Circle 1 Focus: (26.56, 26.56)
Circle 2 Focus: (79.68, 26.56)
Circle 3 Focus: (132.8, 26.56)
Circle 4 Focus: (185.92, 26.56)
Circle 5 Focus: (239.04, 26.56)
Circle 6 Focus: (292.16, 26.56)
Circle 7 Focus: (345.28, 26.56)
Circle 8 Focus: (372.63, 72.1)
Circle 9 Focus: (319.52, 73.25)
Circle 10 Focus: (265.47, 72.64)
Circle 11 Focus: (212.35, 73.25)
Circle 12 Focus: (159.23, 72.64)
Circle 13 Focus: (106.11, 73.25)
Circle 14 Focus: (52.99, 72.64)
<svg width="400" height="100"><rect width="400" height="100" style="fill:blue;" /><circle cx="26.560000000311106" cy="26.560000000311106" r="26.550000000311105" fill="white" /><circle cx="79.68000000093332" cy="26.560000000311106" r="26.550000000311105" fill="white" /><circle cx="132.80000000155553" cy="26.560000000311106" r="26.550000000311105" fill="white" /><circle cx="185.92000000217774" cy="26.560000000311106" r="26.550000000311105" fill="white" /><circle cx="239.04000000279996" cy="26.560000000311106" r="26.550000000311105" fill="white" /><circle cx="292.16000000342217" cy="26.560000000311106" r="26.550000000311105" fill="white" /><circle cx="345.2800000040444" cy="26.560000000311106" r="26.550000000311105" fill="white" /><circle cx="372.6271770491687" cy="72.09972230654316" r="26.550000000311105" fill="white" /><circle cx="319.5195599732359" cy="73.24663493712801" r="26.550000000311105" fill="white" /><circle cx="265.47097406711805" cy="72.63752174440503" r="26.550000000311105" fill="white" /><circle cx="212.35454341475625" cy="73.25330971030218" r="26.550000000311105" fill="white" /><circle cx="159.23097406587362" cy="72.63752174440503" r="26.550000000311105" fill="white" /><circle cx="106.11454341351183" cy="73.25330971030218" r="26.550000000311105" fill="white" /><circle cx="52.99097406462921" cy="72.63752174440503" r="26.550000000311105" fill="white" /></svg>

And the SVG looks like this...

Test 3
(function(e,f,g){...})(400,400,3)
Highest possible radius: 101.68
Circle 1 Focus: (101.69, 101.69)
Circle 2 Focus: (298.23, 153.98)
Circle 3 Focus: (154.13, 298.19)
<svg width="400" height="400"><rect width="400" height="400" style="fill:blue;" /><circle cx="101.69000000059772" cy="101.69000000059772" r="101.68000000059772" fill="white" /><circle cx="298.2343937547503" cy="153.97504264473156" r="101.68000000059772" fill="white" /><circle cx="154.13153961740014" cy="298.19269546075066" r="101.68000000059772" fill="white" /></svg>

And the SVG looks like this...

They aren't all pretty.

How it works
Ungolfed code below. This program has two assumptions:

One circle will always be in the corner. This seems like a pretty safe bet.
Every circle will always be touching another circle. I don't see why they wouldn't be.

The program starts by calculating a large radius based on the box dimensions. It then tries to fit one circle in the corner of the box. If that circle fits, it extends a diameter line from that circle and tries to create a circle at the end of the line. If the new circle fits, another line will be extended from the new circle. If it doesn't fit, the line will swing 360 degrees, checking for open spaces. If the box fills up before the desired number of circles are created, the radius is reduced and it all starts over again.

Ungolfed Code (snippet)

// this functions attempts to build a circle
// at the given coords. If it works, it will
// spawn additional circles.
function C(x, y, X, Y, r, cc){

 // if this circle does not fit in the rectangle, BAIL
 if(X-r < 0 || X+r > x || Y-r < 0 || Y+r > y)
  return cc;

 // if this circle is too close to another circle, BAIL
 for(var c in cc){
  if( Math.sqrt(Math.pow(cc[c].x - X, 2) + Math.pow(cc[c].y - Y, 2)) < (r*2) )
   return cc;
 }

 // checks passed so lets call this circle valid and add it to stack
 cc.push({"x": X, "y": Y});

 // now rotate to try to find additional spots for circles
 var a = 0; // radian for rotation
 while(a < Math.PI){
  XX = Math.cos(a)*r*2 + X;
  YY = Math.sin(a)*r*2 + Y;
  cc = C(x, y, XX, YY, r, cc);
  a += .01; // CHANGE FOR MORE OR LESS PRECISION
 }

 return cc;
}

// this function slowly reduces the radius
// and checks for correct solutions
// also prints svg graphic code

(function B(x, y, n){

 var r = x + y; // pick a big radius
 var X, Y; // these are the coords of the current circle. golf by combining this with `var r..`?
 var cc = []; // array of coordinates of the circles

 // while we cant fit n circles, reduce the radius and try again
 while(cc.length < n){
  X = Y = r;
  cc = C(x, y, X, Y, r, []);
  r-=.01; // CHANGE FOR MORE OR LESS PRECISION
 }

 console.log('Highest possible radius: ' + Math.round(r*100)/100);
 var s = '<svg width="' + x + '" height="' + y + '"><rect width="' + x + '" height="' + y + '" style="fill:red" />';
 for(var c in cc){
  console.log('Circle '+c+' Focus: (' + Math.round(cc[c].x*100)/100 + ', ' + Math.round(cc[c].y*100)/100 + ')');
  s += '<circle cx="' + cc[c].x + '" cy="' + cc[c].y + '" r="' + r + '" fill="blue" />';
 }
 s += '</svg>';
 console.log(s);
 document.write(s);
})(150, 150, 5);