Serverless XYZ Basemaps

Serving imagery, cheap and lean

A Brief Primer on XYZ Tiles

Tiled web maps - Wikipedia

Visual Example: Zoom 0

.
└── 0
    └── 0
        └── 0.jpg

Visual Example: Zoom 1

.
├── 0
|   └── 0
|       └── 0.jpg
└── 1
    ├── 0
    |   ├── 0.jpg
    |   └── 1.jpg
    └── 1
        ├── 0.jpg
        └── 1.jpg

Visual Example: Zoom 2

.
├── 0
|   └── 0
|       └── 0.jpg
├── 1
│   ├── 0
|   |   ├── 0.jpg
|   |   └── 1.jpg
│   └── 1
|       ├── 0.jpg
|       └── 1.jpg
└── 2
    ├── 0
    |   ├── 0.jpg
    |   ├── 1.jpg
    |   ├── 2.jpg
    |   └── 3.jpg
    ├── 1
    |   ├── 0.jpg
    |   ├── 1.jpg
    |   ├── 2.jpg
    |   └── 3.jpg
    ├── 2
    |   ├── 0.jpg
    |   ├── 1.jpg
    |   ├── 2.jpg
    |   └── 3.jpg
    └── 3
        ├── 0.jpg
        ├── 1.jpg
        ├── 2.jpg
        └── 3.jpg

This Web Mercator grid is built into basically every web mapping framework. Go to a spot on the map, and it knows which tiles it should ask for.

So you’ve got your TIFF files…

How do you share them?

• ArcGIS Image Server?
• AGOL / Portal Hosted Tile Cache?
• GeoServer?

flowchart LR

  subgraph T[Requesting Imagery Tiles]
    F[[Imagery File]]
    C{Tile Cache}
    S([GIS Server])
    U([End User])
    D[(Storage)]
  end

  O[Other Things]

  S --> O
  S --> O
  S --> O

  F --> D & C
  S & C <--> D
  U <--> S

  style T fill:None,stroke:#c0caf5,stroke-width:4px,color:#c0caf5

In most cases, this will involve moving the imagery tile onto whatever machine is running your services, then generating the cache there.

Further, the machine and / or database running this will need to handle all the traffic and processing for the tile requests.

flowchart TD

  subgraph T[Requesting Imagery Tiles]
    direction LR
    F[[Imagery File]]
    C{Tile Cache}
    D[(Storage)]
    A[CloudFront CDN]
    U([End User])
  end

  S[GIS Server]
  O[Other Things]

  T --- S
  S ---> O
  S ---> O
  S ---> O
  S ---> O
  S ---> O
  F --> C --> D
  D -.-> A
  A <---> U

  style T fill:None,stroke:#c0caf5,stroke-width:4px,color:#c0caf5
  linkStyle 0 stroke-width:0

In short: leave your server machine out of it.

Why Bother?

• Your server is a powerful tool!

• Moving imagery tiles to a serverless context conserves your server’s resources

• Your server can focus on things that actually need its processing power

Step 1: QGIS

Generate the Tile Cache

1. Open QGIS
2. Processing Toolbox
3. Raster Tools
4. Generate XYZ Tiles (Directory)

Parameters

Most defaults are fine, but check:

• Extent: Same as input raster, or other AOI
• Min/Max Zoom: Adjust as needed; For imagery, around 19 is probably fine
• Format and quality
• Output directory: Pick a simple name for the directory, like 2023_imagery

PNG or JPG?

PNG is nice for transparency, but consider whether you really need your imagery tiles to have it. Swapping for JPG and adding even a little compression will make for massive space saving and faster data transfer.

Step 2: AWS S3

Simple Storage Solution

There are similar ways of doing this in Azure, and probably Google, etc., but I’ll leave you to figure those out if that’s what you want to do.

Create a Bucket

Getting Started with S3 - AWS Tutorial

Default configuration is fine

Storage Classes

• Most tiles will never be seen
• Set up a Lifecycle Rule to move your files to other storage classes
• Also: Intelligent-Tiering for automatically changing class based on use patterns

Save Some Money

Seldom and never-used tiles will drop into infrequent access and archive tiers. These are still readily accessible to end users, but storage costs are much smaller.

High detail tiles are typically only needed for dense residential areas. In rural areas, especially fields, zooming in to LOD 19 or 20 is actually less helpful in most cases, so your users aren’t going to be doing it.

For prior years of imagery, the number of tiles being accessed is dramatically smaller.

Loading the Tiles

1. Create a bucket
2. Drag the tile directory and drop it in!
3. Wait a long time

Tip

When loading your files, you can specify which storage class to put them into right from the get-go.

• The total size may not be much, but the sheer number of files will bottleneck this process.

• If waiting for this process to complete is going to bother you, do it overnight, or use multiple tabs.

• Alternatively, check out something like rclone for loading your files from the command line, it’s a bit faster.

Learning Things the Hard Way

Be sure to use rclone copy, not rclone sync!

Configure It

Make the entire bucket public — (AWS Tutorial)

• HTTP only
• Goofy S3 URL
• All requests hit the bucket directly (S3 GET requests are extremely cheap, but that still isn’t free)

Actually, don’t do this!

Don’t Configure It?

Use CloudFront to serve the bucket via a CDN — (AWS Tutorial)

• HTTPS
• Bucket itself remains private
• Custom URL
• Requests hit CDN first, fewer GET requests on the bucket

Step 3: CloudFront

Create a Distribution

AWS Tutorial

• Link your S3 bucket as the origin
• Restrict access to the origin
• Set a reeeeeally high default TTL (time to live)
  • Default cache settings will expire your files every 24 hours
  • Your tiles aren’t going to change, so there’s basically no reason for the CDN to check for updates

Alternate Domain Name

• Add the domain name of your choosing
• Use AWS Certificate Manager to issue an SSL cert
• Copy the CNAME name / value to your DNS provider’s settings

Note

This is optional. You can always use the Cloudfront URL that is generated for your distribution.

Step 4: Use Your Tiles!

QGIS

1. Open the Data Source Manager
2. Select XYZ
3. Click “New”
4. Enter your distribution URL distribution-url/xyz-directory/{z}/{x}/{y}.jpg¹

1. Make sure to match the directory name you defined back in Step 1, as well as the file extension.

ArcGIS Online

1. Open the Map Viewer
2. Add a layer by URL
3. Same URL as last slide
4. Move newly-added layer to your basemap
5. Save the webmap, share to your basemap group

Leaflet, etc.

from ipyleaflet import Map
from ipyleaflet.leaflet import TileLayer

tiles = TileLayer(url = "https://tiles.kendallcountyil.gov/1871/{z}/{x}/{y}.jpg")

m = Map(
  center = (41.6609, -88.5350),
  zoom = 14,
  layers = [tiles]
)

m

Questions?

Be in Touch!

josh@jcarlson.page

https://jcarlson.page

@jcarlson@mapstodon.space