<jupyter_start><jupyter_text>Prepare data for the Soils Revealed projecthttps://github.com/Vizzuality/soils-revealed-data`<NAME> (vizzuality.)` DescriptionThis notebook transforms vector boundaries into MapBox tiles format (MBTILES) using tippecanoe and uploads the resulting tiles to MapBox.```MIT LicenseCopyright (c) 2020 VizzualityPermission is hereby granted, free of charge, to any person obtaining a copyof this software and associated documentation files (the "Software"), to dealin the Software without restriction, including without limitation the rightsto use, copy, modify, merge, publish, distribute, sublicense, and/or sellcopies of the Software, and to permit persons to whom the Software isfurnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in allcopies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS ORIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THEAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHERLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THESOFTWARE.```  Setup  Linux dependencies<jupyter_code>%%bash
# Install AWS CLI (for MapBox uploads)
apt install --no-install-recommends -y -q awscli
%%bash
# Install tippecanoe (for MapBox mbtiles)
apt install --no-install-recommends -q -y build-essential libsqlite3-dev zlib1g-dev
make
make install
add-apt-repository -y ppa:ubuntu-toolchain-r/test
apt update -q -y
apt install --no-install-recommends -q -y g++-5
export CXX=g++-5

git clone https://github.com/mapbox/tippecanoe.git
cd tippecanoe
make -j
make install
!tippecanoe -h<jupyter_output>tippecanoe: invalid option -- 'h'
Usage: tippecanoe [options] [file.json ...]
  Output tileset
         --output=output.mbtiles [--output-to-directory=...] [--force]
         [--allow-existing]
  Tileset description and attribution
         [--name=...] [--attribution=...] [--description=...]
  Input files and layer names
         [--layer=...] [--named-layer=...]
  Parallel processing of input
         [--read-parallel]
  Projection of input
         [--projection=...]
  Zoom levels
         [--maximum-zoom=...] [--minimum-zoom=...]
         [--extend-zooms-if-still-dropping] [--one-tile=...]
  Tile resolution
         [--full-detail=...] [--low-detail=...] [--minimum-detail=...]
  Filtering feature attributes
         [--exclude=...] [--include=...] [--exclude-all]
  Modifying feature attributes
         [--attribute-type=...] [--attribute-description=...]
         [--accumulate-attribute=...] [--empty-csv-columns-are-null]
         [--convert-stringified-ids-to-numbers]
         [--[...]<jupyter_text>Python packages<jupyter_code>%%bash
# Install mapbox python package
pip install mapbox
!pip list<jupyter_output>Package                  Version        
------------------------ ---------------
absl-py                  0.9.0          
alabaster                0.7.12         
albumentations           0.1.12         
altair                   4.1.0          
asgiref                  3.2.7          
astor                    0.8.1          
astropy                  4.0.1.post1    
astunparse               1.6.3          
atari-py                 0.2.6          
atomicwrites             1.3.0          
attrs                    19.3.0         
audioread                2.1.8          
autograd                 1.3            
awscli                   1.14.44        
Babel                    2.8.0          
backcall                 0.1.0          
beautifulsoup4           4.6.3          
bleach                   3.1.4          
blis                     0.4.1          
bokeh                    1.4.0          
boto                     2.49.0         
boto3                    1.12.39        
botocore        [...]<jupyter_text>Authorisation  Google cloud storageEither use user authorisation or a service account, save credentials to your drive or upload.<jupyter_code># For auth WITHOUT service account
#from google.colab import auth
#auth.authenticate_user()

# https://cloud.google.com/resource-manager/docs/creating-managing-projects
#project_id = "soc-platform"
#!gcloud config set project {project_id}
# Mount drive
from google.colab import drive
drive.mount('/content/drive')
# Copy GC credentials to home (place in your GDrive, and connect Drive)
!cp "/content/drive/My Drive/soc-platform-6a9bf204638c.json" "/root/.soc-platform-6a9bf204638c.json"
# Auth WITH service account
!gcloud auth activate-service-account \
  <EMAIL>  \
          --key-file=/root/.soc-platform-6a9bf204638c.json --project="soc-platform"

# Test GC auth
!gsutil ls "gs://vizz-data-transfer"<jupyter_output>gs://vizz-data-transfer/SOC_maps/<jupyter_text>MapBoxCreate a JSON file and add it to your drive or upload:```{"MB_USER:"user-name", "MB_TOKEN":"token"}```<jupyter_code># Copy GC credentials to home (place in your GDrive, and connect Drive)
!cp "/content/drive/My Drive/copernicus-forests-mapbox.json" "/root/.copernicus-forests-mapbox.json"
# Set up Mapbox (S3) credentials as environmental variables
import json
import os

# Set user and token as environment variables
c = json.loads(open("/root/.copernicus-forests-mapbox.json").read())
os.environ['MB_USER'] = c['MB_USER']
os.environ['MB_TOKEN'] = c['MB_TOKEN']

# Make call to mapbox api and save return to file
!curl -X POST https://api.mapbox.com/uploads/v1/${MB_USER}/credentials?access_token=${MB_TOKEN} > credentials.json
r = json.loads(open("credentials.json").read())
#print(r)

# Set credentials as environ variables
os.environ['MB_BUCKET'] = r['bucket']
os.environ['MB_KEY'] = r['key']
os.environ['AWS_ACCESS_KEY_ID'] = r['accessKeyId']
os.environ['AWS_SECRET_ACCESS_KEY'] = r['secretAccessKey']
os.environ['AWS_SESSION_TOKEN'] = r['sessionToken']<jupyter_output>% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   857  100   857    0     0   2158      0 --:--:-- --:--:-- --:--:--  2158<jupyter_text>Utils  copy_gcs<jupyter_code>import os
import subprocess

def copy_gcs(source_list, dest_list, opts=""):
  """
  Use gsutil to copy each corresponding item in source_list
  to dest_list
  """
  for s, d  in zip(source_list, dest_list):
    cmd = f"gsutil -m cp -r {opts} {s} {d}"
    print(f"Processing: {cmd}")
    r = subprocess.call(cmd, shell=True)
    if r == 0:
        print("Task created")
    else:
        print("Task failed")
  print("Finished copy")<jupyter_output><empty_output><jupyter_text>upload_to_mapbox<jupyter_code># Upload task for mapbox
import os
from mapbox import Uploader

def upload_to_mapbox(file_path, tileset_name):
  """
  Given a local file path and a MapBox tileset name
  push to MapBox AWS S3 staging and create MapBox upload task
  """
  username = os.getenv("MB_USER")
  my_token = os.getenv("MB_TOKEN")
  u = Uploader(access_token=my_token)  # handles authentication
  tileset = f"{username}.{tileset_name}"  # name your tileset
  job = u.upload(open(file_path, 'rb'), tileset)  # upload happens here
  # job = u.create(url, tileset, name=tileset_name)  # starts the tiling job
  status = job.status_code
  print(status)<jupyter_output><empty_output><jupyter_text>create_mbtiles<jupyter_code>import os
import subprocess

def create_mbtiles(source_path, dest_path, layer_name, opts="-zg --drop-densest-as-needed --extend-zooms-if-still-dropping --force --read-parallel"):
  """
  Use tippecanoe to to create a MBTILE at dest_path from source_path.
  layer_name is used for the name of the layer in the MBTILE.
  Regex file path (/*.geojson) is supported for source_path.
  """
  cmd = f"tippecanoe -o {dest_path} -l {layer_name} {opts} {source_path}"
  print(f"Processing: {cmd}")
  r = subprocess.call(cmd, shell=True)
  if r == 0:
      print("Task created")
  else:
      print("Task failed")
  print("Finished processing")<jupyter_output><empty_output><jupyter_text>Process data  Create MBTILES<jupyter_code>layer_name = "SWE_biovar_species"
source_path = "'/content/drive/My Drive/copernicus-forests/SWE_zonal_biovar_ISEA-3-HEXAGON_grid.geojson'"
dest_path = "'/content/drive/My Drive/copernicus-forests/SWE-bv-spp.mbtiles'"
create_mbtiles(source_path, dest_path, layer_name, opts="-zg --drop-densest-as-needed --extend-zooms-if-still-dropping --force --read-parallel")<jupyter_output>Processing: tippecanoe -o '/content/drive/My Drive/copernicus-forests/SWE-bv-spp.mbtiles' -l SWE_biovar_species -zg --drop-densest-as-needed --extend-zooms-if-still-dropping --force --read-parallel '/content/drive/My Drive/copernicus-forests/SWE_zonal_biovar_ISEA-3-HEXAGON_grid.geojson'
Task created
Finished processing<jupyter_text>Upload to MapBox<jupyter_code># Add to Mapbox
import glob 
import os
path = '/content/drive/My Drive/copernicus-forests/'
files = [f for f in glob.glob(path + "**/*.mbtiles", recursive=True)]
print(files)
for f in files:
  print(f)
  upload_to_mapbox(f, os.path.splitext(os.path.basename(f))[0])<jupyter_output>['/content/drive/My Drive/copernicus-forests/SWE-bv-spp.mbtiles']
/content/drive/My Drive/copernicus-forests/SWE-bv-spp.mbtiles
201
StarcoderdataJupyterStructuredCleanDedup