Working with metadata in evalscript

This user guide will show you how to work with metadata in evalscripts. We will focus on using objects scenes, inputMetadata, and outputMetadata. Use cases, covered with the examples below, include accessing metadata and using it in processing, passing the metadata to an output file userdata.json, and adding your own metadata to the file.

Note that metadata normally provided in raster format is available as bands in Sentinel Hub. Such metadata can be accessed and processed in evalscript in the same manner as any other input band. This is not covered in this guide, but you can find basic examples and such metadata listed in the Data section for each data collection e.g. sunAzimuthAngles.

Each example below begins with a description that highlights the important points of the example. All examples output also processed satellite images (average values of NDVI or band B02) but we do not display them here, since the focus is on metadata. To run the examples, you only need to have Python installed on your machine and an active Sentinel Hub account. You will always need to run the code in the chapter "Authentication" while the rest of the examples can be run independently.

The jupyter notebook with all examples can be downloaded here.

Authentication

First, we need to fetch an access token, which we will use to authenticate all Sentinel Hub requests. To do so, replace <client_id> and <client_secret> in the code snippet below with your client id and client secret, respectively and run the code. To learn how to get your client id and client secret, read this documentation.

from oauthlib.oauth2 import BackendApplicationClient
from requests_oauthlib import OAuth2Session
import os, io, tarfile, json, requests

client_id = '<client_id>'
client_secret = '<client_secret>'
client = BackendApplicationClient(client_id=client_id)
oauth = OAuth2Session(client=client)
oauth.fetch_token(token_url='https://services.sentinel-hub.com/auth/realms/main/protocol/openid-connect/token',
                          client_secret=client_secret, include_client_id=True)

{'access_token': 'eyJraWQiOiJzaCIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiI0MmYwODZjNy1kMzI3LTRlOTMtYWMxNS00ODAwOGFiZjI0YjIiLCJhdWQiOiIyNDUyNGRjNi03ZDkwLTQzZjEtYTYxZS1hNjIzZWNiZWIzNDUiLCJqdGkiOiJkZDQ2ZDVmMS04ZDUwLTQxMWItYjg0Yi00MjNiOTliMTE2ZDkiLCJleHAiOjE2MjEzNDQ5MTIsIm5hbWUiOiJhbmphLnZyZWNrb0BzaW5lcmdpc2UuY29tIiwiZW1haWwiOiJhbmphLnZyZWNrb0BzaW5lcmdpc2UuY29tIiwic2lkIjoiZWU2NTM1NzktZGEwOS00MmNiLTgxYjItMjBhMmUwMDhhM2VkIiwib3JnIjoiYTUyY2U2ZGEtMjI5MC00N2MyLTg0YjEtNWZkNTg5ZGFhYzI1IiwiZGlkIjoxLCJhaWQiOiJkOTExMGJjMS1iOGE1LTQ5NTYtOGVhYS1jZTAyMzBhNDkxMjIiLCJkIjp7IjEiOnsicmEiOnsicmFnIjoyLCJyYSI6eyJidGMiOjF9fSwidCI6MjAwMDB9fX0.Xl3kwL1K3sIhRCY4-6WE7MGrC6tb0d_igfKaytUfhmpzyySn2j5S_IX8B5gtO0HnKTSCcU7BY1lO1Q0dowK2FMd8bH-rgRsul5LS5TCBKTxjFn1LMS_jQDAWEu5Iuo6EmhKx7u4MK5DpuUkIPLE9faF9H-5qxJJA8vxAOXaM1Hz35n-aUNn0gKrYHWCqMlQAwwTizaHk2ueg0ONfWhuzJnIpuaj9iSYWzbFWHKVZdINJTN9O9XijaSFl6o3S2FEoxYLCfrZaYHUL8AKd0lt8DZkr1nXVUkC3FH8fLNCfPLOJIGwQboj8zjn4QDt_ahQCKkMmvvnJr36phWuQ8ZWqKQ',
 'expires_in': 3599,
 'expires_at': 1621344910.501437}

The access token is stored in the oauth object, which will be used to send all subsequent requests.

Check which metadata is available

The metadata is stored in two objects, which we call inputMetadata and scenes. Their properties are documented here and here, respectively. However, the properties of the scenes object can be different depending on the selected:

• mosaicking (e.g. ORBIT or TILE),
• data collection (Sentinel-2 L2A, Sentinel-1, Sentinel-5p, ...),
• function in the evalscript (evaluatePixel, preProcessScenes, updateOutputMetadata).

A convenient way to check which metadata is for your request available in scenes is to dump (i.e. write) all properties of the object to userdata.json file. This can be achieved with the Processing API as shown in this basic example. The two examples below show few more tricks that can be used to explore scenes object.

Properties of scenes object and mosaicking ORBIT

This example shows:

• How to access metadata when mosaicking is ORBIT using scenes.orbits.
• How to pass metadata from scenes to userdata.json file using outputMetadata.userData in updateOutputMetadata function.

url = 'https://services.sentinel-hub.com'

evalscript = """
//VERSION=3
function setup() {
  return {
    input: ["B02", "dataMask"],
    mosaicking: Mosaicking.ORBIT,
    output: {
      id: "default",
      bands: 1
    }
  }
}

function evaluatePixel(samples, scenes, inputMetadata, customData, outputMetadata) {
  //Average value of band B02 based on the requested scenes
  var sumOfValidSamplesB02 = 0
  var numberOfValidSamples = 0
  for (i = 0; i < samples.length; i++) {
    var sample = samples[i]
    if (sample.dataMask == 1){
        sumOfValidSamplesB02 += sample.B02
        numberOfValidSamples += 1
    }
  }
  return [sumOfValidSamplesB02 / numberOfValidSamples]
}

function updateOutputMetadata(scenes, inputMetadata, outputMetadata) {
  outputMetadata.userData = {
    "inputMetadata": inputMetadata
  }
  outputMetadata.userData["orbits"] = scenes.orbits
}
"""

request = {
  "input": {
    "bounds": {
      "bbox": [13.8, 45.8, 13.9, 45.9]
    },
    "data": [{
      "type": "sentinel-2-l1c",
      "dataFilter": {
        "timeRange": {
          "from": "2020-12-01T00:00:00Z",
          "to": "2020-12-06T23:59:59Z"
        }
      }
    }]
  },
  "output": {
    "responses": [{
        "identifier": "default",
        "format": {
          "type": "image/tiff"
        }
      },
      {
        "identifier": "userdata",
        "format": {
          "type": "application/json"
        }
      }
    ]
  },
  "evalscript": evalscript
}

headers = {
  'Content-Type': 'application/json',
  'Accept': 'application/x-tar'
}

response = oauth.post(f"{url}/api/v1/process", headers=headers, json = request)
tar = tarfile.open(fileobj=io.BytesIO(response.content))
userdata = json.load(tar.extractfile(tar.getmember('userdata.json')))
userdata

{'inputMetadata': {'serviceVersion': '4.263.0', 'normalizationFactor': 0.0001},
 'orbits': [{'tiles': [{'date': '2020-12-06T10:08:08Z',
     'shId': 15161628,
     'cloudCoverage': 100,
     'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/UL/2020/12/6/0'},
    {'date': '2020-12-06T10:08:05Z',
     'shId': 15161463,
     'cloudCoverage': 98.26,
     'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/VL/2020/12/6/0'}],
   'dateTo': '2020-12-06T23:59:59Z',
   '__idx': 0,
   'dateFrom': '2020-12-06T00:00:00Z'},
  {'tiles': [{'date': '2020-12-04T10:18:05Z',
     'shId': 15142759,
     'cloudCoverage': 99.93,
     'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/UL/2020/12/4/0'},
    {'date': '2020-12-04T10:17:56Z',
     'shId': 15142728,
     'cloudCoverage': 98.5,
     'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/VL/2020/12/4/0'}],
   'dateTo': '2020-12-04T23:59:59Z',
   '__idx': 1,
   'dateFrom': '2020-12-04T00:00:00Z'},
  {'tiles': [{'date': '2020-12-01T10:08:10Z',
     'shId': 15117250,
     'cloudCoverage': 22.85,
     'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/UL/2020/12/1/0'},
    {'date': '2020-12-01T10:08:06Z',
     'shId': 15117286,
     'cloudCoverage': 46.81,
     'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/VL/2020/12/1/0'}],
   'dateTo': '2020-12-01T23:59:59Z',
   '__idx': 2,
   'dateFrom': '2020-12-01T00:00:00Z'}]}

Properties of scenes object and mosaicking TILE

This example shows how to:

• Access scenes metadata when mosaicking is TILE using scenes.tiles and write it to userdata.json file.
• How to calculate a maximum value of band B02 and write it to userdata.json file. Note that we use a global variable maxValueB02 so that we can assign a value to it in evaluatePixel function but write its value to metadata in updateOutputMetadata function. The advantage of this approach is that maxValueB02 is written to metadata only once and not for each output pixel.

url = 'https://services.sentinel-hub.com'

evalscript = """
//VERSION=3
function setup() {
  return {
    input: ["B02", "dataMask"],
    mosaicking: Mosaicking.TILE,
    output: {
      id: "default",
      bands: 1
    }
  }
}

var maxValueB02 = 0

function evaluatePixel(samples, scenes, inputMetadata, customData, outputMetadata) {
  //Average value of band B02 based on the requested tiles
  var sumOfValidSamplesB02 = 0
  var numberOfValidSamples = 0
  for (i = 0; i < samples.length; i++) {
    var sample = samples[i]
    if (sample.dataMask == 1){
        sumOfValidSamplesB02 += sample.B02
        numberOfValidSamples += 1
        if (sample.B02 > maxValueB02){
            maxValueB02 = sample.B02
        }
    }
  }
  return [sumOfValidSamplesB02 / numberOfValidSamples]
}

function updateOutputMetadata(scenes, inputMetadata, outputMetadata) {
  outputMetadata.userData = { "tiles":  scenes.tiles }
  outputMetadata.userData.maxValueB02 = maxValueB02
}
"""

request = {
  "input": {
    "bounds": {
      "bbox": [13.8, 45.8, 13.9, 45.9]
    },
    "data": [{
      "type": "sentinel-2-l1c",
      "dataFilter": {
        "timeRange": {
          "from": "2020-12-01T00:00:00Z",
          "to": "2020-12-06T23:59:59Z"
        }
      }
    }]
  },
  "output": {
    "responses": [{
        "identifier": "default",
        "format": {
          "type": "image/tiff"
        }
      },
      {
        "identifier": "userdata",
        "format": {
          "type": "application/json"
        }
      }
    ]
  },
  "evalscript": evalscript
}

headers = {
  'Content-Type': 'application/json',
  'Accept': 'application/x-tar'
}

response = oauth.post(f"{url}/api/v1/process", headers=headers, json = request)
tar = tarfile.open(fileobj=io.BytesIO(response.content))
userdata = json.load(tar.extractfile(tar.getmember('userdata.json')))
userdata

{'tiles': [{'date': '2020-12-06T10:08:08Z',
   'shId': 15161628,
   'cloudCoverage': 100,
   '__idx': 0,
   'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/UL/2020/12/6/0'},
  {'date': '2020-12-06T10:08:05Z',
   'shId': 15161463,
   'cloudCoverage': 98.26,
   '__idx': 1,
   'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/VL/2020/12/6/0'},
  {'date': '2020-12-04T10:18:05Z',
   'shId': 15142759,
   'cloudCoverage': 99.93,
   '__idx': 2,
   'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/UL/2020/12/4/0'},
  {'date': '2020-12-04T10:17:56Z',
   'shId': 15142728,
   'cloudCoverage': 98.5,
   '__idx': 3,
   'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/VL/2020/12/4/0'},
  {'date': '2020-12-01T10:08:10Z',
   'shId': 15117250,
   'cloudCoverage': 22.85,
   '__idx': 4,
   'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/UL/2020/12/1/0'},
  {'date': '2020-12-01T10:08:06Z',
   'shId': 15117286,
   'cloudCoverage': 46.81,
   '__idx': 5,
   'dataPath': 's3://sentinel-s2-l1c/tiles/33/T/VL/2020/12/1/0'}],
 'maxValueB02': 0.8795000000000001}

Output metadata into userdata.json file

In this example, we write several pieces of information to the userdata.json file:

• A version of the software with which the data was processed. We take this information from inputMetadata.
• Dates when the data used for processing was acquired. We take this information from scene.tiles.
• Values set by user and used for processing, such as thresholds (e.g. ndviThreshold) and array of values (e.g. notAllowedDates).
• Dates of all tiles available before we filtered out those acquired on dates given in notAllowedDates array. These dates are listed in tilesPPSDates property of userData. Note how we used a global variable tilesPPS: we assigned it a value in preProcessScenes and output it in updateOutputMetadata function.
• Dates of all tiles available after the filtering. These dates are listed in tilesDates property of userData.
• Description of the processing implemented in the evalscript and links to external resources.

url = 'https://services.sentinel-hub.com'

evalscript = """
//VERSION=3
function setup() {
  return {
    input: ["B08", "B04", "dataMask"],
    mosaicking: Mosaicking.TILE,
    output: {
      id: "default",
      bands: 1
    }
  }
}

// User's inputs
var notAllowedDates = ["2020-12-06", "2020-12-09"]
var ndviThreshold = 0.2

var tilesPPS = []
function preProcessScenes(collections) {
  tilesPPS = collections.scenes.tiles
  collections.scenes.tiles = collections.scenes.tiles.filter(function(tile) {
    var tileDate = tile.date.split("T")[0];
    return !notAllowedDates.includes(tileDate);
  })
  return collections
}

function evaluatePixel(samples, scenes, inputMetadata, customData, outputMetadata) {

  var valid_ndvi_sum = 0
  var numberOfValidSamples = 0
  for (i = 0; i < samples.length; i++) {
    var sample = samples[i]
    if (sample.dataMask == 1){
        var ndvi = (sample.B08 - sample.B04)/(sample.B08 + sample.B04)
        if (ndvi <= ndviThreshold){
          valid_ndvi_sum += ndvi
          numberOfValidSamples += 1
        }
    }
  }

  return [valid_ndvi_sum / numberOfValidSamples]
}

function updateOutputMetadata(scenes, inputMetadata, outputMetadata) {
  outputMetadata.userData = {
    "inputMetadata.serviceVersion": inputMetadata.serviceVersion
  }

  outputMetadata.userData.description = "The evalscript calculates average ndvi " +
  "in a requested time period. Data collected on notAllowedDates is excluded. " +
  "ndvi values greater than ndviThreshold are excluded. " +
  "More about ndvi: https://www.indexdatabase.de/db/i-single.php?id=58."

  // Extract dates for all available tiles (before filtering)
  var tilePPSDates = []
  for (i = 0; i < tilesPPS.length; i++){
    tilePPSDates.push(tilesPPS[i].date)
  }
  outputMetadata.userData.tilesPPSDates = tilePPSDates

  // Extract dates for tiles after filtering out tiles with "notAllowedDates"
  var tileDates = []
  for (i = 0; i < scenes.tiles.length; i++){
    tileDates.push(scenes.tiles[i].date)
  }
  outputMetadata.userData.tilesDates = tileDates

  outputMetadata.userData.notAllowedDates = notAllowedDates
  outputMetadata.userData.ndviThreshold = ndviThreshold
}
"""

request = {
  "input": {
    "bounds": {
      "bbox": [13.8, 45.8, 13.9, 45.9]
    },
    "data": [{
      "type": "sentinel-2-l1c",
      "dataFilter": {
        "timeRange": {
          "from": "2020-12-01T00:00:00Z",
          "to": "2020-12-15T23:59:59Z"
        }
      }
    }]
  },
  "output": {
    "responses": [{
        "identifier": "default",
        "format": {
          "type": "image/tiff"
        }
      },
      {
        "identifier": "userdata",
        "format": {
          "type": "application/json"
        }
      }
    ]
  },
  "evalscript": evalscript
}

headers = {
  'Content-Type': 'application/json',
  'Accept': 'application/x-tar'
}

response = oauth.post(f"{url}/api/v1/process", headers=headers, json = request)
tar = tarfile.open(fileobj=io.BytesIO(response.content))
userdata = json.load(tar.extractfile(tar.getmember('userdata.json')))
userdata

{'notAllowedDates': ['2020-12-06', '2020-12-09'],
 'tilesDates': ['2020-12-11T10:08:07Z',
  '2020-12-11T10:08:03Z',
  '2020-12-04T10:18:05Z',
  '2020-12-04T10:17:56Z',
  '2020-12-01T10:08:10Z',
  '2020-12-01T10:08:06Z'],
 'inputMetadata.serviceVersion': '4.263.0',
 'description': 'The evalscript calculates average ndvi in a requested time period. Data collected on notAllowedDates is excluded. ndvi values greater than ndviThreshold are excluded. More about ndvi: https://www.indexdatabase.de/db/i-single.php?id=58. ',
 'tilesPPSDates': ['2020-12-11T10:08:07Z',
  '2020-12-11T10:08:03Z',
  '2020-12-09T10:18:04Z',
  '2020-12-09T10:17:56Z',
  '2020-12-06T10:08:08Z',
  '2020-12-06T10:08:05Z',
  '2020-12-04T10:18:05Z',
  '2020-12-04T10:17:56Z',
  '2020-12-01T10:08:10Z',
  '2020-12-01T10:08:06Z'],
 'ndviThreshold': 0.2}