Technical Proposal

From OpenAerialMap

Jump to: navigation, search

This page is intended as a draft technical proposal for the further development of the OpenAerialMap project, which will store Free and Open georeferenced photographic imagery for the entire world.

Contents

Overview

The proposed OAM architecture consists of three main pieces: a hosted catalog server, one or more imagery stores, and a larger number of leaf nodes, which provide access to image tiles through the usual web service tile APIs. The description of peer-to-peer networking features proposed for OAM has been moved to the P2P Network Proposal page.

Use Cases and Workflow

Use OAM imagery in a web application

  • A developer wishes to use OAM imagery as one or more layers in their web mapping application.
  • The developer visits openaerialmap.org and finds documentation on browsing and accessing layers, with examples.
  • The developer browses the layer catalog, and cuts and pastes WMS(-C), TMS, or XYZ web service URLs into their application configuration. The web service URLs contain a hostname that uses dynamic round-robin DNS to direct tile requests to a distributed cluster of tile servers.

Fetch OAM imagery for local use

  • A GIS analyst wishes to download OAM imagery in bulk for use in a desktop GIS application.
  • The analyst visits openaerialmap.org and browses a map overview to find layers in their area of interest.
  • The analyst can filter and browse layers by depiction time span.
  • The analyst is presented with a set of OGC WCS and WMS URLs that they can load in their application.
  • The analyst can also browse and download source images within their AoI in their original projection.
  • Optionally, the analyst can request an archive containing a set of available images clipped to their AoI.

Contribute a source WMS to OAM

  • A user finds a WMS server with imagery available under a Free license, and decides to add it to the OAM database.
  • The user logs into the OAM catalog server and creates a new layer with the WMS server URL.
  • The catalog server fetches the WMS capabilities file and augments the layer metadata.
  • A curator is notified of the new layer, and reviews the layer entry to ensure validity and appropriateness. If the layer record is good, the curator approves the new layer.
  • The storage nodes fetch updated layer information from the catalog server, and proxy and cache tile requests from a set of leaf nodes.

Contribute an imagery dataset to OAM

  • A user has an archive of aerial imagery that they wish to contribute to OAM.
  • The user creates an account on the OAM catalog server, logs in, and creates a new layer to hold their dataset.
  • The catalog server creates an FTP directory on a storage node, and provides the URL to user so that they can upload their imagery.
  • When the upload is finished, the user confirms the upload with the catalog server.
  • The catalog server contacts the storage node to fetch the imagery metadata, and updates the catalog.
  • A curator is notified of the new layer, and reviews the layer entry to ensure validity and appropriateness. If the layer record is good, the curator approves the new layer.
  • The storage nodes fetch updated layer information from the catalog server, and deliver tile and full image requests for the new layer.

Data Model

The proposed data model for the OAM service includes the concepts of users, layers, layer groups, source images, tiles, storage nodes, and cache nodes. One or more metadata servers will store and serve the objects embodied in the model.

Users

Each user will have a name, an email address, and a password. Email and password are used to log into the site. Email addresses will have to be verified. Users might also have administrator and curator flags.

Layers

A layer consists of a layer ID, an author (i.e. the user that creates the record), a title, a description, a depiction time stamp, a URI, a source URI (if the source image is stored in the network), a URI indicating the data provider (if known), a color depth, a scale/resolution, an accuracy metric, and a lon/lat bounding box.

Licensing is also a serious consideration for some potential OAM users. Each layer should be marked with the license of its source, including (at a minimum) descriptive text for the license, plus flags for public domain, attribution, non-commercial, and sharealike licensing.

If the layer is a WMS (WMS-C, TMS, KML SuperOverlay, XYZ), the URI should be the service base URL. The metadata server should fetch the WMS capabilities and set the layer metadata appropriately.

If the layer is an OAM-stored layer, the WMS(-C) URL can be composed from the storage node base URL and the layer ID, but the layer metadata should be filled in completely by the layer uploader, possibly automatically from the source file metadata.

A new layer may be flagged for review if the creating user is not an administrator or a curator. Tiles from layers flagged for review should not be cached in the network until the layer is approved by a curator.

Layer groups

A layer group is a set of layers that can be composited and served as a single layer. By default, at each scale, the composite layer group should show the raster layer with most recent depiction date and the largest scale less than or equal to the scale being rendered, although tools could be built to permit other arrangements.

The "Global Layer" group will be a virtual spherical mercator layer at the standard zoom levels that consists of whatever tile has the most recent depiction date and the largest available scale equal to or larger than that of the given grid cell. How exactly to composite/mosaic tiles when imagery from two different sources overlap within a grid cell bears some serious thought.

The way that this should probably work is that, once a tile is fetched for inclusion in the global layer, if the tile contains transparent pixels, the storage node should fetch the tiles for all layers at or above the relevant zoom level for the grid cell and all adjacent grid cells, and then composite them. The composite tile should be stamped with a depiction date of the current UTC time, and then uploaded back to the storage network.

While specific historical layers can be accessed by requesting those specific tile layers from any storage node, the storage nodes will need to implement the WMS 1.1.1 TIME parameter (possibly combined with WMS-C) to support fetching historical views of the global layer. Clients will need to use the WMS TIME parameter to request these tiles. See Annexes B and C of the WMS 1.1.1 specification for details on formatting.

Source imagery

A source image consists of an image ID, a uploading user, a reference to the layer it belongs to, an auxiliary title or description, a web service URI, and a source URI that points to the actual imagery. For the purposes of simplicity, source imagery is assumed to have all of the other metadata properties associated with the layer to which it belongs.

In principle, the web service and source URIs should be composable from the hostname of the storage node that serves the image, the layer ID to which it belongs, and the image ID.

Tiles

Image tiles should be 256x256 pixels in 8- or 24-bit compressed PNG format, with alpha transparency where appropriate, so that tile layers can be overlaid.

Tile layers will always be rendered in Web spherical mercator at the standard Web powers-of-2 zoom levels, or, in other words, the WMS-C proposed "Mercator" profile.

Explicit references to image tiles will not be stored on the metadata server, but instead will be served from cache nodes where present.

For sanity checking, the project should declare a maximum file size for tiles.

Storage nodes

Storage nodes will host the source imagery used to generate layers, and deliver tiles to cache nodes. Initially, the number of storage nodes will be small enough that their DNS entries can be managed by hand, and no machine-readable metadata will be to be maintained about them.

Cache nodes

Cache nodes will deliver tiles and proxy tile requests to the storage nodes.

The server should track all nodes in the cache layer, and provide a periodically updated listing of active nodes, including IP address, geographic location, key fingerprint (if any), uptime, storage utilization, storage capacity, and bandwidth weighting. By tracking all storage nodes, the metadata server should be able to provide metrics on the overall storage utilization and capacity of the network.

Technical Components

Catalog server

The catalog server stores the OAM data model, and provides user interfaces for account creation, login, logout, password confirmation and reset, creating and updating OAM objects, and for browsing them in tabular and geographic forms. The catalog server should also fetch and store top-level overviews of each layer for preview in the layer browser. The tabular listing should be filterable, sortable, and full-text searchable.

The server has a set of community-appointed administrators who will perform user management and catalog gardening. Administrators can appoint curators, who are empowered to approve new layer submissions.

The catalog server should have have at least one off-site hotswap backup, multiple round-robin read-only peers that can each be pressed into master service, or some other kind of high availability arrangement.

The catalog server should produce a complete, machine-readable dump of its layer metadata that can be cached by OAM nodes and used for speed or when the catalog server is unavailable. The catalog server should also ping cache nodes that make metadata requests and use the response times for tracking and load balancing. A reset timeout should be implemented when a node goes down to keep its status from flapping.

The catalog server should maintain a dynamic DNS server for load balancing. DNS lookups for tile1, tile2, tile3, etc. should do a geo-IP lookup on the requesting IP, and return a short list of IP addresses for storage nodes that is weighted by bandwidth capacity and by geographic proximity to the requester. The catalog server should also return dynamic responses for requests to xx.openaerialmap.org, where xx is an ISO-3166 country code. The DNS should also return similar results for tile[1-4].xx.openaerialmap.org. (When this is implemented, the OAM community should consider seeking sponsorship to obtain a commercial GeoIP database for best results.)

Storage nodes

For the purposes of simplicity, the OAM community should identify at least two hosting providers with high bandwidth capacity and available data storage; although, one will suffice to start.

Each hosting provider should allow the deployment of a storage node, which will permit the creation and use of FTP accounts authenticated by the catalog server. When a user creates a layer in the OAM metadata server, the storage node should automatically create a corresponding directory on the FTP server, and provide the user with an ftp:// URL and instructions on uploading imagery for that layer.

When a user uploads a source image to an OAM layer's FTP directory, and confirms the upload with the catalog server, an automated process should create an image record in the metadata service, and associate the image with the corresponding layer.

Once a layer is marked as "active" by the OAM curators, the layer may be delivered via a MapScript-driven WMS or via TileCache WMS-C/TMS/etc. using standard REST-like URLs, which will expose uploaded layers to the leaf nodes. The layer "name" will simply be the layer's generated ID. Layer groups can be delivered in the same way.

Source images should ideally be archived at the storage node in a lossless, internally-tiled format, such as LZW-compressed tiled GeoTIFF. Source images can be left in their original projection.

Layers whose source is a WMS should be archived at storage nodes, as well, in tile form.

Rather than accessing the live metadata service for every tile or image request, storage nodes should periodically fetch and cache the complete layer model from the metadata service, and use an in-memory copy of the model to render web service requests.

The storage node will need custom code in mod_python and MapScript to serve imagery by layer over WMS. The storage node should run TileCache for tile requests with a large in-memory cache for speed. Disk space should be reserved for source imagery and archived offsite WMS tiles.

Storage nodes should never discard source images or archived WMS requests, but should discard the latter in favor of the former if space becomes a premium.

Cache nodes

Client requests for tiles should routed through cache nodes via "standard" tiling web services. If the cache node has the tile, it should return it. If the cache doesn't have the tile, it should proxy the request to a storage node for handling. Cache nodes can run TileCache with a mix of in-memory and on-disk cache.

Cache nodes should periodically fetch and cache the layer list from the metadata server. First, this 'heartbeats' the cache node against the metadata server for tracking purposes. Second, the cache node can use the layer list to configure TileCache, and to fall back to proxying tile requests directly to their sources in the event that the metadata or storage nodes go down.

Cache nodes should be as easy to set up as possible, and work out of the box with minimal configuration.

Performance and Reliability

Prior efforts at implementing the OAM concept have foundered partly on the single-point-of-failure problem, particularly when the network administration staff at the hosting provider isn't getting paid extra to do the necessary maintenance. Redundant storage of tiles in the storage network will guarantee reliability as nodes come on and offline. Once implemented, the crypto key infrastructure will guarantee data integrity.

The centralized metadata server introduces another intrinsic single-point-of-failure, but hopefully as the network grows, the community will be able to employ existing best practices in high-availability Web engineering to host this resource redundantly, as well.

Overall storage capacity is the big problem. As a rule, any fully tiled layer will take up 4/3 of the size of the original source layer at max resolution, plus the size of the source images (if stored in the network). Furthermore, if we adopt a liberal replication factor in the storage network, e.g. k=3, every source image and all the tiles generated for it will take up 3x the space of both.

This means, as a rule of thumb, that the network must store ((4/3) + 1) * 3 = 7 MB of imagery plus tiles for every 1 MB of source imagery uploaded. If we load up all of the approximately 4 TB of LandSat-7 data at a 30m resolution, and generate a complete tile set, we will need 16-28 TB of storage in the network to hold it all. If stored on EC2, this would cost up to US$3,000 per month -- and that's just for one layer at a low resolution.

However, it's hard to see how a permanent, durable network for storing all of the world's free aerial imagery indefinitely could do otherwise. The consolation is that whenever the network runs out of space, it can just stop caching new tiles until more room is added.

One possibility for optimizing locality in the cache network would be to implement a DNS server on the metadata side that divides the Earth into (say for example) 16x16 = 256 storage zones. This "GeoDNS" server could then just return a capacity-weighted, round-robin subset of the k-local nodes for each zone, with a low enough TTL that nodes and clients get the benefit of caching, but get reasonably fast updates when nodes go offline. Furthermore, a smart client (e.g. OpenLayers, TileCache, GeoWebCache) could be optimized to request tiles directly from their storage zones, rather than using an arbitrary proxy node or merely using one that's local to the client.

Security Considerations

The following is a scratch list of possible points of attack on the network:

  • DoS attacks on the metadata server(s)
  • Malicious uploading of bogus tiles or data in the network (solved by using tile signatures)
  • Compromise of administrator or system accounts on the metadata servers
  • Compromise of the metadata server private key
  • Bogus storage node join floods

It's worth noting that all but the last consideration have applied generally to Web-based applications for years. Best practices for preventing or mitigating these attacks should be considered in the long-run. Storage node join floods should be addressed separately.

Proposed Roadmap

The long-term success of OAM will depend in the short-term on getting basic functionality working quickly.

We suggest the use of Python as the main implementation language because of its widespread use in the OSGeo community. We likewise recommend the use of GeoDjango and PostGIS on the web server side, and TileCache on the web service side.

If written in Python, the storage node libraries should be carefully written to permit their use in Jython, to make it easier to integrate an OAM node into (say) GeoServer.

The use of GDAL and/or MapScript for reading raster metadata, converting raster formats, and extracting and reprojecting tiles on the fly is recommended. Hopefully, by building on tools already in widespread use in the OSGeo community, we will maximize the number of developers willing and able to contribute to the project.

Immediate next steps to bootstrap the next phase of OAM might include:

Phase 1

  • Implement the user, layer, and source image models and UIs for the catalog server in (Geo)Django.
  • Implement a TileCache Layer module that proxies all requests back to the storage nodes and caches tiles locally.
  • Deploy an instance of the metadata server.
  • Deploy an instance of the storage node, with FTP uploads and a (possibly proxying) WMS server.
  • Deploy one or more cache nodes that proxy requests to the storage node.

Phase 2

  • Implement the cache nodes model in the metadata server.
  • Add layer metadata caching to the cache node module.
  • Add cache node 'heartbeat' checking to the metadata server.
  • Implement round-robin DNS among cache nodes on the metadata server.
  • Deploy additional cache nodes.

Phase 3

  • Implement the layer groups model and UI in the metadata server.
  • Implement the "global virtual layer" in the storage node.
  • Ensure high-availability for the metadata service.
  • Deploy additional storage nodes.
  • Stress-test the cache node network.

Phase 4

  • Refine the "global layer" support, including support for the TIME parameter.
  • Consider implementing features from the P2P Network Proposal.
  • Consider building desktop upload and download tools.
  • Address lower-risk security considerations (as described above).
  • What's the future of OAM?

Additional Reading

Related Links

Retrieved from "http://wiki.openaerialmap.org/Technical_Proposal"
Personal tools