Using extensions to support pagination in OpenAPI

Recently, I gave a talk at the API Specifications Conference in Vancouver, BC. The talks were not recorded, but attendees and people following along online found it interesting how we are using OpenAPI extensions at Transposit and how some of these extensions might be useful for the future progression of the OpenAPI Specification.

This blog post is the second in a series of a few posts where I’ll cover:

• How Transposit uses the OpenAPI Specification
• AWS APIs, Boto, and why we had to support them differently with OpenAPI
• How we created OpenAPI extensions and what we learned from this process

This series is for anyone who wants to understand APIs better, extend OpenAPI, or get into the guts of Amazon Web Service (AWS) APIs.

In the last post in this series, we covered how Transposit has created a dynamic API client with OpenAPI documents, our research into how the Boto library supports Amazon Web Services (AWS) APIs, and how we support AWS APIs’ parameter serialization by extending the style property in OpenAPI.

This second post will focus on how Boto support pagination in their dynamic API client and how we generalized that for OpenAPI.

Pagination#

One of the helpful parts about Boto is that it includes metadata that describes how to paginate through APIs. As a developer you can search the documentation and read how to use the pagaintion in a specific API. But we want the generated API client to be able to do that with any API, in a similar way that this can be done in Boto. For example:

# Create a session and a client
session = botocore.session.get_session()
client = session.create_client('s3', region_name='us-west-2')

# Create a reusable Paginator
paginator = client.get_paginator('list_objects')

# Create a PageIterator from the Paginator
page_iterator = paginator.paginate(Bucket='my-bucket')

for page in page_iterator:
    print(page['Contents'])

(Source: https://botocore.amazonaws.com/v1/documentation/api/latest/topics/paginators.html)

In the example above we get a paginator for the list_objects operation and we call the paginator instead of calling the operation. The page_iterator will perform the pagination and fetch the results from multiple pages while tracking the pagination state and making the additional HTTP requests when needed without any additional code.

What will actually happen when you run this code is something like the following HTTP requests and responses (most of the headers and data were removed in these examples for brevity).

First the boto client will send the first HTTP request - it will only include the parameters, in this case it's the bucket name:

GET /my-bucket HTTP/1.1
Host: s3.us-west-2.amazonaws.com
X-Amz-Target: .ListObjects

The response from S3 will be something like this (it will actually be XML but we'll display it here as the equivalent JSON):

{
  "ListBucketResult": {
    "Contents": [
      {
        "LastModified": "2019-09-06T20:29:12.000Z",
        "Key": "file-1.txt"
      },
      ...
      {
        "LastModified": "2019-11-07T22:26:29.000Z",
        "Key": "file-1000.txt"
      }
    ],
    "IsTruncated": true,
    "MaxKeys": 1000,
    "Name": "my-bucket"
  }
}

These are the results for the first page (assuming that the bucket contains more than 1000 files).
To get the next page boto will extract the next page token based on the pagination schema that we will describe below, in this case the next page token is the last file name - file-1000.txt, and it will construct and send the following request:

GET /my-bucket?marker=file-1000.txt HTTP/1.1
Host: s3.us-west-2.amazonaws.com
X-Amz-Target: .ListObjects

Note the new query parameter - marker=file-1000.txt.

This will continue in the same way until there are no more results (when the IsTruncated field is false).

Boto pagination schema#

In boto, the schema for pagination is specified in the file - paginators-1.json.
To continue our example above with S3 - the pagination schema for the list objects operation is:

{
  "pagination": {
    "ListObjects": {
      "limit_key": "MaxKeys",
      "input_token": "Marker",
      "more_results": "IsTruncated",
      "output_token": "NextMarker || Contents[-1].Key",
      "result_key": "Contents"
    }
  }
}

The ListObjects key should match an operation key in the service-2.json file that we saw in the first post, in this case the ListObjects operation schema is:

"operations":{
  "ListObjects":{
    "name":"ListObjects",
    "http":{
      "method":"GET",
      "requestUri":"/{Bucket}"
    },
    "input":{"shape":"ListObjectsRequest"},
    "output":{"shape":"ListObjectsOutput"},
  }
  ...
}

In the section below we will describe the fields in paginators-1.json and how they map to other schema information in service-2.json, these json snippets are from the botocore repository in case you want to review this yourself.

In the pagination schema there are two types of configurations:

• a field that tells us how to identify a specific parameter in the request.
• a field that tells us how to extract information from the response.

limit_key and input_token are the first type - they describe a parameter in the request. The other fields more_results, output_token and result_key are the second type - they tell us how to extract information from the response.

Let's start with limit_key and input_token and put the relevant schema snippets side by side:

paginators-1.json

{
  "pagination": {
    "ListObjects": {
      "limit_key": "MaxKeys",
      "input_token": "Marker",
      "more_results": ...,
      "output_token": ...,
      "result_key": ...
    }
  }
}

service-2.json

"shapes":{
  "ListObjectsRequest":{
    "type":"structure",
    "required":["Bucket"],
    "members":{
      "Bucket":{
        "shape":"BucketName",
        "location":"uri",
        "locationName":"Bucket"
      },
      "MaxKeys":{
        "shape":"MaxKeys",
        "location":"querystring",
        "locationName":"max-keys"
      },
      "Marker":{
        "shape":"Marker",
        "location":"querystring",
        "locationName":"marker"
      },
      ...
    }
  },
  ...
}

You can see that the pagination configuration "limit_key": "MaxKeys" points to the MaxKeys parameter in the request - we can use this parameter to set how many results we want to get in the response.

The configuration “input_token”: “Marker” points to the Marker parameter in the request - this parameter is used to continue the pagination after the first request. This is the same marker parameter that we saw in the S3 example above.

The rest of the pagination configuration fields more_results, output_token and result_key are related to the response, let's see the pagination and the response schema side by side:

paginators-1.json

{
  "pagination": {
    "ListObjects": {
      "limit_key": ...,
      "input_token": ...,
      "more_results": "IsTruncated",
      "output_token": "NextMarker || Contents[-1].Key",
      "result_key": "Contents"
    }
  }
}

service-2.json

"shapes":{
  "ListObjectsOutput":{
    "type":"structure",
    "members":{
      "IsTruncated":{
        "shape":"IsTruncated"
      },
      "NextMarker":{
        "shape":"NextMarker"
      },
      "Contents":{
        "shape":"ObjectList"
      }
    }
  },
  ...
}

The configuration "more_results": "IsTruncated" tells us that the response body will include a field with the name IsTruncated - when we get a response we can extract this value and decide if there are more results after the current page and if we should send another request in order to get the next page.

The configuration "output_token": "NextMarker || Contents[-1].Key" looks a little more complicated, but let's break this into smaller parts:

NextMarker and Contents are fields in the API response body.

Remember our first example - we're talking about cursor/token pagination - the response contains a token that we feed into the next request - in this case output_token tells us where to find in the response that cursor/token.

In this case the expression NextMarker || Contents[-1].Key is a jmespath expression and what it means is "if the field NextMarker exists in the response use that value as the pagination cursor, otherwise take the key of the last item in Contents".

AWS uses jmespath expressions in the Python SDK and CLI to transform the results in the client side, for example:

Python SDK:

filtered_iterator = page_iterator.search("Contents[?Size > `100`][]")

CLI:

aws ec2 describe-volumes --query \
   'Volumes[*].{ID:VolumeId,AZ:AvailabilityZone,Size:Size}'

In the pagination configuration a jmespath expression is used in order to specify some logic that we need to perform on the data in the response body in order to extract the actual information that we need - we'll talk about this more later.

And the last configuration "result_key": "Contents" - Contents is also a jmespath expression that tells the api client how to extract the actual results from the response body.

What are the "actual results"?

Again, let's look at our first example, the response was something like this:

{
  "ListBucketResult": {
    "Contents": [
      {
        "LastModified": "2019-09-06T20:29:12.000Z",
        "Key": "file-1.txt"
      },
      ...
      {
        "LastModified": "2019-11-07T22:26:29.000Z",
        "Key": "file-1000.txt"
      }
    ],
    "IsTruncated": true,
    "MaxKeys": 1000,
    "Name": "test5345345"
  }
}

As a user that use the AWS CLI or even the AWS Python SDK what you really want to see is the data under the Contents - basically the list of files in the bucket. In most cases you don't really care about the other fields in the response - IsTruncated, MaxKeys, etc. So the result_key configurations tells us what we actually need to take from the response and we can ignore the rest.

Generalize Pagination for OpenAPI#

We took what we learned from AWS and built it into OpenAPI extension. AWS supports only cursor pagination because all their APIs use cursor pagination but OpenAPI tries to capture the behaviour of different APIs. So we need to generalize this schema.

We've identified 4 types of pagination that cover the vast majority of APIs:

• Cursor/token
• Offset
• Page
• Next page URL

Let's review the extension configuration for each one.

Cursor Pagination#

This pagination is the same as the boto cursor pagination that we just talked about but we need to convert all the boto schemas that we saw to OpenAPI schemas.

Let's do another side by side comparison of how we can convert the above snippets of the boto ListObjects operation schema to OpenAPI schema, in this case we only need the schema from service-2.json:

Boto schema

{
  "version":"2.0",
  "metadata":{ ... },
  "operations":{
    "ListObjects":{
      "name":"ListObjects",
      "http":{
        "method":"GET",
        "requestUri":"/{Bucket}"
      },
      "input":{"shape":"ListObjectsRequest"},
      "output":{"shape":"ListObjectsOutput"}
    }
  },
  "shapes":{
    "ListObjectsRequest":{
      "type":"structure",
      "required":["Bucket"],
      "members":{
        "Bucket":{
          "shape":"BucketName",
          "location":"uri",
          "locationName":"Bucket"
        },
        "MaxKeys":{
          "shape":"MaxKeys",
          "location":"querystring",
          "locationName":"max-keys"
        },
        "Marker":{
          "shape":"Marker",
          "location":"querystring",
          "locationName":"marker"
        },
        ...
      }
    },
    "ListObjectsOutput":{
      "type":"structure",
      "members":{
        "IsTruncated":{
          "shape":"IsTruncated"
        },
        "NextMarker":{
          "shape":"NextMarker"
        },
        "Contents":{
          "shape":"ObjectList"
        },
        ...
      }
    }
  }
}

OpenAPI schema

openapi: "3.0.0"
info:
  version: 1.0.0
  title: AWS S3
servers:
  - url: https://s3.us-west-2.amazonaws.com
paths:
  /{bucket}:
    get:
      operationId: ListObjects
      parameters:
        - name: bucket
          in: path
          required: true
          schema:
            type: string
        - name: max-keys
          in: query
          required: false
          schema:
            type: integer
            format: int32
        - name: marker
          in: query
          required: false
          schema:
            type: string
      responses:
        '200':
          description: ...
          content:
            application/xml:
              schema:
                type: object
                properties:
                  IsTruncated:
                    type: boolean
                  NextMarker:
                    type: string
                  Contents:
                    type: array
                    items: 
                      type: object
                xml:
                  name: 'ListBucketResult'

(If you read the first post closely - I actually say that this direct conversion from boto schema to OpenAPI schema is not possible. The difference here is that we just convert one operation and not the entire AWS S3 API)

Now, we can add an extension to the OpenAPI operation schema with the pagination schema:

x-pagination:
  cursor:
    cursorParam: "marker"
    limitParam: "max-keys"
    cursorPath: "NextMarker"
    resultsPath: "Contents"

Because we have 4 different types of pagination and each one can have different configuration we need a type to differentiate between them, we choose to make the type as the object name and under each type we'll have the specific configuration.

In this case the type of the pagination is cursor, and the configuration for this pagination is described in the additional four fields - these fields are the same fields that we saw in the boto pagination schema:

• cursorParam: "marker" and limitParam: "max-keys"describe the page token and page size parameters in the request.
• cursorPath: "NextMarker" and resultsPath: "Contents" describe the two values that we need to extract from the response - where to find the next cursor and where to find the actual results.

Offset#

This is probably the most common type of pagination that is currently used by APIs.
Offset pagination is simple - the client asks the API to get items 0-99, then asks for items 100-199, etc. This pagination has some problems, but comparing pros and cons of pagination methods is not the purpose of this post so we will not talk about this here.

This is how the actual requests and responses will look like:

GET /list?offset=0&pageSize=100

The response will be something like:

{
  "items": [
    {
      "name": "item-0"
    },
    ...
    {
      "name": "item-99"
    }
  ]
}

Then we increment the offset by the value of pageSize and send the next request:

GET /list?offset=100&pageSize=100

And we continue until we get less than 100 items in the response.

Note that the pageSize value cannot be changed between requests, otherwise you will get wrong results.

This is how the OpenAPI schema for this operation will look like:

/list:
  get:
    operationId: list
    parameters:
      - in: query
        name: offset
        schema:
          type: integer
      - in: query
        name: pageSize
        schema:
          type: integer
          minimum: 1
          maximum: 100
          default: 50
    responses:
      '200':
        description: ...
        content:
          application/json:
            schema:
              type: object
              properties:
                items:
                  type: array
                  items: 
                    type: object

And in this case the pagination extension will be:

x-pagination:
  offset:
    offsetParam: "offset"
    limitParam: "pageSize"
    resultsPath: "items"

The type for this pagination is 'offset'.

offsetParam and limitParam describe the two parameters:

• offsetParam - tells us which parameter controls the index of the first item.
• limitParam - tells us which parameter controls the number of items in the response.

resultsPath tells us where to find the actual results in the response.

Page#

Page pagination is almost the same as offset pagination, but instead of asking for specific item indexes we ask for pages.

To get the first page we ask for page 0, then page 1, etc.

GET /list?page=0&pageSize=100

The response will be something like:

{
  "items": [
    {
      "name": "item-0"
    },
    ...
    {
      "name": "item-99"
    }
  ]
}

Then we increment the page by one and send the next request:

GET /list?page=100&pageSize=100

And we continue until we get less than 100 items in the response.

Note that also here the pageSize value cannot be changed between requests.

This is how the OpenAPI schema for this operation will look like:

/list:
  get:
    operationId: list
    parameters:
      - in: query
        name: page
        schema:
          type: integer
      - in: query
        name: pageSize
        schema:
          type: integer
          minimum: 1
          maximum: 100
          default: 50
    responses:
      '200':
        description: ...
        content:
          application/json:
            schema:
              type: object
              properties:
                items:
                  type: array
                  items: 
                    type: object

And in this case the pagination extension will be:

x-pagination:
  pageOffset:
    pageOffsetParam: "page"
    limitParam: "pageSize"
    resultsPath: "items"

The type for this pagination is pageOffset.

The parameters are similar to the offset pagination:

• pageOffsetParam - tells us which parameter controls the page number.
• limitParam - tells us which parameter controls the number of items in the response.
• resultsPath tells us where to find the actual results in the response.

Next Page URL#

Next page URL pagination is similar to cursor pagination, in the sense that we get some value from the response and we need to use it, but it's different because we use the extracted URL as is and we don't need to construct a new request based on the schema and parameters (the API should take care of that for us).

This pagination is not commonly used, and sometimes even if the API response contains a 'next page url' value we can still use one of the other pagination types.

In the first request we don't need to specify the page and we will get the first page:

GET /list HTTP/1.1
Host: api.example.io

The response will be something like:

{
  "items": [
    {
      "name": "item-0"
    },
    ...
    {
      "name": "item-99"
    }
  ],
  "nextPageUrl": "https://api.example.io?nextPage=ABCD123..."
}

From this point we just extract the nextPageUrl from the response and send a request to that URL, we do that until the nextPageUrl is not specified in the response.

This is how the OpenAPI schema for this operation will look like:

/list:
  get:
    operationId: list
    parameters:
      - in: query
        name: pageSize
        schema:
          type: integer
          minimum: 1
          maximum: 100
          default: 50
    responses:
      '200':
        description: ...
        content:
          application/json:
            schema:
              type: object
              properties:
                items:
                  type: array
                  items: 
                    type: object
                nextPageUrl:
                  type: string

And in this case the pagination extension will be:

x-pagination:
  nextUrl:
    limitParam: "pageSize"
    nextUrlPath: "nextPageUrl"
    resultsPath: "items"

The pagination type is nextUrl.

The only parameter in the request is limitParam that is similar to the other paginations and tells us which parameter controls the number of items in the response.

And from the response we need to extract two values:

• nextUrlPath - tells us where to find in the response the URL that we will use to get the next page.
• resultsPath tells us where to find the actual results in the response.

How Transposit uses pagination#

With this additional schema information, you can see how we can make our dynamic API client handle pagination and automatically request more pages from the api as needed.

This allows the developer to decide how many items they need without knowing how the pagination works in each API, for example:

SELECT * FROM api.list_items
LIMIT 100

SELECT * FROM api.list_items AS L
  JOIN api.get_item_details AS D
  ON L.itemId = D.id
LIMIT 10 

What about nested values?#

But we're not done, in the examples above we just used a value in a single level.

For example, this single level JSON:

{
  "items": [
    …
  ]
}

If we want to describe the items field as the resultsPath we can just use:

resultsPath: "items"

But what if the results are inside a nested object like in this response:

{
  "results": {
    "sales": {
      "items": [
        …
      ]
    },
    "metadata": {
      "nextPageToken": "ec366c7e-..."
    }
  }
}

In this example, how do we describe the resultsPath and cursorPath?

In most APIs, the values that we need to extract from the response are nested in a simple JSON structure, so we started with a support for basic path:

resultsPath: "results.sales.items"
cursorPath: "results.metadata.nextPageToken"

But we need to make it more general. For example, what if the cursor token is in a header?

Runtime expressions or jmespath?#

Earlier in this post we mentioned that AWS uses jmespath expressions to solve a similar problem. So one option is to use jmespath.
But jmespath expressions are limited and can work only with the JSON that we get in the response body and cannot access headers or other information in the raw response.

Another option is to reuse the existing runtime expressions from OpenAPI specification.

Runtime expressions#

Runtime expressions are already defined in the OpenAPI Specification and used in links:

...
links:
  GetUserById:
    operationId: getUser
    parameters:
      username: $response.body#/id

Runtime expressions allow us to access all the information from the request or the response - URL, HTTP method, headers, status code, and the request or response body:

If we go back to our last example with nested response:

{
  "results": {
    "sales": {
      "items": [
        …
      ]
    },
    "metadata": {
      "nextPageToken": "ec366c7e-..."
    }
  }
}

This how we will define the results path and cursor path for the pagination with runtime expressions:

resultsPath: "$response.body#/results/sales/items"
cursorPath: "$response.body#/results/metadata/nextPageToken"

As mentioned above runtime expressions also allows us to access the cursor path if it's in a header, for example in this HTTP response:

200 OK
x-next-page-token: ec366c7e-...

{
  "results": {
    "sales": {
      "items": [
        …
      ]
    }
  }
}

In this case the the runtime expressions for the results path and cursor path in the pagination configuration will be:

resultsPath: "$response.body#/results/sales/items"
cursorPath: "$response.header.x-next-page-token"

What about OpenAPI links?#

When we were working on this we thought: why we don't just use OpenAPI links for pagination?

This topic can cover a separate and long discussion, but the short answer is that in some cases we need to describe some runtime logic for the pagination. For example, with page pagination we know that we need to increment the page number after each request - in most cases the API will not include the 'next page index' value in the response, so we need to put that logic somewhere.
We might want to write the link schema as in this example:

/list:
  get:
    operationId: list
    parameters:
      - in: query
        name: pageSize
        schema:
          type: integer
      - in: query
        name: page
        schema:
          type: integer
    responses:
      '200':
        links:
          nextPage:
            x-pagination: true
            x-resultsPath: "items"
            operationId: list
            parameters:
              page: '$request.query.page + 1'
              pageSize: '$request.query.pageSize'

Note the expression $request.query.page + 1 in the link page parameter - this is currently not possible with OpenAPI Specification, so we stayed with the extension and we need to hardcode that logic in the api client based on the pagination type.

Conclusion#

The Transposit platform would not be useful without handling pagination transparently for our developers. At the same time we try to be compliant with the OpenAPI Specification as much as possible in order to make it easy for us and for our developers to add and maintain data connectors.

There are many discussions about pagination support in OpenAPI Specification, but none of these discussions has evolved into an actual specification. For the Transposit platform we need some solution for pagination now and we prefer to take a practical approach and experiment with different specifications for pagination.

Transposit has a unique position in the OpenAPI community because not only do we create and maintain OpenAPI documents for many third party APIs, we also parse and execute requests based on these OpenAPI documents - we can truly say that we see and work with many APIs that have different and sometimes unique behaviour.

The pagination extension that we described in this post is just the first step and it will be an iterative process (in fact we're already in the process of making a few small changes to this extension and we'll publish that in a separate post). We know that there are other APIs that use different types of paginations that are not mapped to one of the four pagination types that we described above or can be mapped to one of these pagination types but still have some small differences.

If you think any of the extensions we shared in this post are useful to you or if you have a comment, let us know! We are currently considering writing up a OpenAPI Specification proposal for them or publicly sharing more about them. Tweet @transposit or send an email to support@transposit.com to let us know.

In part three of this series, we’ll talk about other features that can be implemented with OpenAPI extensions and we're currently exploring.

Lastly, thanks to Taylor Barnett for helping me with this blog post series.