Batch Statistical API

The Batch statistical API is in beta release. It might misbehave in case of large requests. We may change the interface, although no major changes are expected. If you have suggestions for improvements or any feedback, please share your thoughts on our forum.
The Batch statistical API is only available for enterprise users. If you don't have an enterprise account, and would like to try it out, contact us for a custom offer.

The Batch statistical API enables you to request statistics similarly as with the Statistical API but for multiple polygons at once and/or for longer aggregations. A typical use case would be calculating statistics for all parcels in a country.

Similarly to the Batch processing API, this is an asynchronous REST service. This means that data will not be immediately returned in the response of the request but delivered to your object storage, which needs to be specified in the request (e.g. S3 bucket, see AWS bucket access below).

You can find more details about the API in the API Reference or in the examples of the workflow.

Workflow

The Batch statistical API workflow is very similar to the Batch Processing API workflow but some parts of the workflow are not yet supported. Currently supported are:

  • user's actions START and STOP.
  • request's statuses CREATED, PROCESSING, DONE, and FAILED.

We are working on supporting the missing user's action ANALYSE and statuses ANALYSING, ANALYSIS_DONE, and PARTIAL.

Automatic deletion of stale data

Stale (inactive) requests will be deleted after a certain period of inactivity, depending on their status:

  • requests with status CREATED are deleted after 7 days of inactivity
  • requests with status FAILED are deleted after 15 days of inactivity
  • all other requests are deleted after 30 days of inactivity.

Note that only such requests themselves will be deleted, while the requests' result (created statistics) will remain under your control in your S3 bucket.

Input polygons as GeoPackage file

The Statistical batch API accepts a GeoPackage file containing features (polygons) as an input. The GeoPackage must be stored in your object storage (e.g. AWS S3 bucket) and Sentinel Hub must be able to read from the storage (find more details about this in the bucket access section below). In a batch statistical request, the input GeoPackage is specified by setting the path to the .gpkg file in the input.features.s3 parameter.

All features (polygons) in an input GeoPackage must be in the same CRS supported by Sentinel Hub. An example of a GeoPackage file can be downloaded here.

Evalscript and Batch statistical API

The same specifics as described for evalscript and Statistical API apply also for Batch statistical API.

Evalscripts smaller than 32KB in size can be provided directly in a batch statistical request under evalscript parameter. If your evalsript exceeds this limit, you can store it to your S3 bucket and provide a reference to it in a batch statistical request under evalscriptReference parameter.

Processing results

Outputs of a batch statistical request are json files stored in your object storage. Each .json file will contain requested statistics for one feature (polygon) in the provided GeoPackage. You can connect statistics in a json file with corresponding feature (polygon) in the GeoPackge based on:

  • id of a feature from GeoPackage is used as name of json file (e.g. 1.json, 2.json) and available in the json file as id property OR
  • a custom column identifier of type string can be added to GeoPackage and its value will be available in json file as identifier property.

The outputs will be stored in the bucket and the folder specified by output.s3.path parameter of the batch statistical request. The outputs will be available in a sub-folder named after the ID of your request (e.g. s3://<my-bucket>/<my-folder>/db7de265-dfd4-4dc0-bc82-74866078a5ce).

Batch Statistical deployment

Batch deploymentBatch URL end-point
AWS EU (Frankfurt)https://services.sentinel-hub.com/api/v1/statistics/batch

AWS bucket access

As noted above, the Batch statistical api uses aws S3 to:

  • read GeoPackage file with input features (polygons) from a S3 bucket,
  • read evalscript from a S3 bucket (this is optional because an evalscript can also be provided directly in a request),
  • write results of processing to a S3 bucket.

One bucket or different buckets can be used for all three purposes.

Bucket regions

The buckets to which the results of batch statistical processing are written must be in the same region as the Batch statistical API deployment. The only available region at the moment iseu-central-1 (Frankfurt).

Bucket policy

IAM user or IAM role (depending on which of access methods described below is used) must have permissions to read and/or write to the corresponding S3 bucket.

Bucket from which evalscript and/or GeoPackage file are read must have GetObject S3 permission listed in its policy. Bucket where results are stored must additionally have PutObject and DeleteObject permissions in its policy.

Example of a valid bucket policy for IAM user and a bucket that's used for both reading input data and writing results:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Batch statistical permissions",
"Effect": "Allow",
"Principal": {
"AWS": "<iam-user-arn->"
},
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::<bucket-name>/*"
]
}
]
}

Access your bucket using accessKey and secretAccessKey

In order to let Sentinel Hub access the bucket, one of the options is to provide accessKey and secretAccessKey pairs in your batch statistical request:

Example:

s3 = {
"url": "s3://<your-bucket>/<path>",
"accessKey": "<your-bucket-access-key>",
"secretAccessKey": "<your-bucket-access-key-secret>"
}

Access key and secret must be linked to an IAM user that has permissions to read and/or write to the corresponding S3 bucket.

The above JSON for accessing the S3 bucket can be used in:

  • input.features.s3 to specify the bucket where GeoPackage file is available,
  • (optional) evalscriptReference.s3 to specify the bucket where evalscript .js file is available,
  • output.s3 to specify the bucket where the results will be stored.

To learn how to configure an access key and access key secret on AWS S3, check this link, specifically, under the Programmatic access section.

Access your bucket using assume IAM role flow

The other option is to provide ARN of your IAM role that has access to the bucket:

s3 = {
"iamRoleARN": "<your-IAM-role-ARN>",
}

When creating an IAM role, you have an option to add additional layer of security by specifying externalId parameter. In case you opt for this, make sure you use your SH domain account id for its value, which you can find on User settings page in Dashboard.

If your IAM role is shared among several principals, and you want to distinguish between their activities by setting roleSessionName in trust policy of each principal, set its value for SH principal to sentinelhub.

Example of a trust policy for IAM role specifying permissions for SH user:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::614251495211:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "<your-SH-domain-account-id>"
},
"StringLike": {
"sts:RoleSessionName": "sentinelhub"
}
}
}
]
}

To give IAM role access to the bucket, you can either add permission policy to the IAM role or set bucket policy in a similar way as above, using your IAM role as principal.

To learn how to create a role for IAM user, check this link.

Examples

Example of a Batch Statistical Workflow