ARROW-3473: [Format] Clarify that 64-bit lengths and null counts are permitted, but not recommended
ARROW-3473: [Format] Clarify that 64-bit lengths and null counts are permitted, but not recommended

The Arrow metadata was changed from the initial format specification to permit 64-bit array lengths, support for which is provided by the C++ library. This clarifies that 64-bit lengths are permissible, but for best compatibility (e.g. with Java), it is recommended to use 32-bits or less in practice.

see also #2733

Author: Wes McKinney <wesm+git@apache.org>

Closes #2734 from wesm/ARROW-3473 and squashes the following commits:

11b4b08a9 <Wes McKinney> Add note about representing large data sets using chunks
067ec9932 <Wes McKinney> Clarify that 64-bit lengths and null counts are permitted, but not recommended for best inter-language compatibility

cpcloud/arrow,majetideepak/arrow,pcmoritz/arrow,itaiin/arrow,xhochy/arrow,wesm/arrow,apache/arrow,kou/arrow,wesm/arrow,apache/arrow,laurentgo/arrow,renesugar/arrow,apache/arrow,cpcloud/arrow,majetideepak/arrow,cpcloud/arrow,icexelloss/arrow,kou/arrow,pcmoritz/arrow,tebeka/arrow,pcmoritz/arrow,majetideepak/arrow,itaiin/arrow,laurentgo/arrow,renesugar/arrow,cpcloud/arrow,xhochy/arrow,laurentgo/arrow,itaiin/arrow,wesm/arrow,wesm/arrow,pcmoritz/arrow,cpcloud/arrow,cpcloud/arrow,renesugar/arrow,pcmoritz/arrow,xhochy/arrow,majetideepak/arrow,laurentgo/arrow,icexelloss/arrow,apache/arrow,cpcloud/arrow,icexelloss/arrow,kou/arrow,majetideepak/arrow,cpcloud/arrow,apache/arrow,kou/arrow,laurentgo/arrow,itaiin/arrow,laurentgo/arrow,icexelloss/arrow,wesm/arrow,kou/arrow,apache/arrow,cpcloud/arrow,majetideepak/arrow,majetideepak/arrow,cpcloud/arrow,laurentgo/arrow,xhochy/arrow,xhochy/arrow,wesm/arrow,wesm/arrow,apache/arrow,wesm/arrow,cpcloud/arrow,kou/arrow,xhochy/arrow,majetideepak/arrow,pcmoritz/arrow,apache/arrow,tebeka/arrow,laurentgo/arrow,pcmoritz/arrow,icexelloss/arrow,apache/arrow,majetideepak/arrow,laurentgo/arrow,kou/arrow,majetideepak/arrow,icexelloss/arrow,kou/arrow,xhochy/arrow,tebeka/arrow,laurentgo/arrow,pcmoritz/arrow,wesm/arrow,apache/arrow,laurentgo/arrow,majetideepak/arrow,laurentgo/arrow,renesugar/arrow,renesugar/arrow,xhochy/arrow,renesugar/arrow,icexelloss/arrow,kou/arrow,pcmoritz/arrow,renesugar/arrow,renesugar/arrow,kou/arrow,cpcloud/arrow,icexelloss/arrow,renesugar/arrow,wesm/arrow,renesugar/arrow,itaiin/arrow,majetideepak/arrow,itaiin/arrow,kou/arrow,renesugar/arrow,kou/arrow,xhochy/arrow,pcmoritz/arrow,kou/arrow,tebeka/arrow,xhochy/arrow,pcmoritz/arrow,apache/arrow,icexelloss/arrow,majetideepak/arrow,apache/arrow,renesugar/arrow,itaiin/arrow,xhochy/arrow,renesugar/arrow,majetideepak/arrow,itaiin/arrow,apache/arrow,renesugar/arrow,itaiin/arrow,renesugar/arrow,cpcloud/arrow,icexelloss/arrow,pcmoritz/arrow,icexelloss/arrow,apache/arrow,pcmoritz/arrow,itaiin/arrow,xhochy/arrow,icexelloss/arrow,tebeka/arrow,pcmoritz/arrow,itaiin/arrow,tebeka/arrow,tebeka/arrow,laurentgo/arrow,icexelloss/arrow,wesm/arrow,pcmoritz/arrow,xhochy/arrow,xhochy/arrow,tebeka/arrow,itaiin/arrow,kou/arrow,laurentgo/arrow,cpcloud/arrow,wesm/arrow,wesm/arrow,itaiin/arrow,xhochy/arrow,itaiin/arrow,laurentgo/arrow,itaiin/arrow,wesm/arrow,tebeka/arrow,majetideepak/arrow,tebeka/arrow,icexelloss/arrow