Home

Creating Your First Connector

Suppose you have a public API with JSON data accessible at a URL like this: https://api.jsonbin.io/b/5c5cded2e9e7c118390e07d5.

Using SwaggerHub or similar editor, you can implement an OpenAPI (f.k.a. Swagger) file representing this single API endpoint.

To turn an OpenAPI file into a basic data connector in Transposit:

1. Create a new connector#

Go to the Transposit developer console and create a new data connector with a sensible name, e.g. bare_bones. (Note: The name must contain only lowercase alphabetic characters, numbers, and underscores.)

2. Clone the repository#

Clone the application's code repository, e.g.:

  git clone https://console.transposit.com/git/yourname/bare_bones

If you are using SSO (e.g. Google, Slack, GitHub) to sign in to Transposit, you will need to use your Git access token located here as your password when using Git with Transposit.

3. Modify the source code#

Update the application's source code tree to contain two files, manifest.json and bare_bones.yaml. Note that this may require deleting existing boilerplate code.

manifest.json

{
  "v2": {
    "swagger": {
      "source": "bare_bones.yaml"
    }
  }
}

You will also need to specify what authentication the API uses (e.g. OAuth, query or path parameters, basic, etc.) in your manifest.json in the swagger section. See the Connector Authentication section for more info and examples.

bare_bones.yaml

openapi: 3.0.0
servers:
  - url: https://api.jsonbin.io
info:
  version: "1.0.0"
  title: Exemplary Bare Bones API
paths:
  /b/5c5cded2e9e7c118390e07d5:
    get:
      operationId: getFunData
      responses:
        '200':
          description: Easy

If your API has operations that support pagination, consider adding pagination information to your OpenAPI file, as described in the Pagination section.

Transposit will validate the manifest and the yaml file when you commit and push them; it will not accept invalid files. However, if you want to ensure the manifest is valid JSON or that the YAML is valid OpenAPI before pushing, you can use an an external tool.

Useful tools for validating the manifest.json file:

Useful tools for validating the bare_bones.yaml file:

4. Commit and push#

Commit your changes and push back to Transposit. Transposit will run some validation on your OpenAPI file.

git push origin master

You can also tag your API using SemVer formatting. Tags are recommended so that applications using your data connector can control which version they are using. This will also prevent inadvertent breakage if the data connector's operations change.

git tag v1.0.1

If you use this, make sure you push your tag as well:

git push origin v1.0.1

5. Add the data connector to an existing application#

You can now add your data connector yourname/bare_bones as a dependency in another Transposit application. Then you can invoke the getFunData operation.

The application can either specify latest or a specific version number for your connector. Navigate to Develop > Data connections and modify the version on the properties page for each connection.

6. Add to connectors list and authenticate#

Once you have created a custom data connector, you need to add it in Transposit's Connectors list in order to authenticate the service. Go to Settings > Connectors > Add connector > Custom connector. The format needed is: maintainer/service_name. You will now be able to configure and connect your new connector.

Connector Authentication#

Specify the type of authentication your data connector uses in your manifest.json. Use the appropriate authentication described in the sections below.

Auth via HTTP basic authentication#

For data connectors that implement authentication with HTTP basic authentication, Transposit will ask for the username/password, Base64 encode the concatenation of the two, and dump that into an Authorization: Basic header.

In your manifest.json, you will need to set the type value to "BASIC". For example:

{
  "v2": {
    "swagger": {
      "auth": {
        "description": "Find your username and password at ...",
        "type": "BASIC"
      },
      "source": "source.yml"
    },
    "description": "My data connector"
  }
}

Auth via header parameters#

For data connections that implement authentication with header parameters, the header parameter name (as documented by the external API site) and value (typically an API key) must be provided when adding the connection to an application.

In your manifest.json, you will need to set the type value to "HEADER". For example, the below code will send a header with the key "Authorization" (i.e. -H "Authorization: <token>"):

{
  "v2": {
    "swagger": {
      "auth": {
        "description": "Find your token here! http://myservice.io",
        "type": "HEADER",
        "params": {
          "header": {
            "string": "Authorization"
          }
        }
      },
      "source": "source.yml"
    },
    "description": "My data connector"
  }
}

The description field will show for all non-OAuth types in the auth modal within Transposit. It is recommended to include information about where to get the token and how to format it.

Auth via query parameters#

For data connections that implement authentication with query parameters, only the query parameter value (typically an API key) must be provided when adding the connection to an application.

For example, if the URL in your OpenAPI file is something like: /getapps?api_key=<token>

In your manifest.json, you will need to set the type value to "PARAMETER". For example:

{
  "v2": {
    "swagger": {
      "auth": {
        "description": "Find your token here! http://myservice.io",
        "type": "PARAMETER",
        "params": {
          "name": {
            "string": "api_key"
          }
        }
      },
      "source": "source.yml"
    },
    "description": "My data connector"
  }
}

Make sure to change the string value to what the API expects.

Auth via path parameters#

Use this authentication method when the service dictates that the API key should be passed through as a path parameter. For example, if the URL in your OpenAPI file is something like: /v1/myapi/{api_key}/create

In your manifest.json, you will need to set the type value to "PATH". For example:

{
  "v2": {
    "swagger": {
      "auth": {
        "description": "Find your token here! http://myservice.io",
        "type": "PATH",
        "params": {
          "path": {
            "string": "api_key"
          }
        }
      },
      "source": "source.yml"
    },
    "description": "My data connector"
  }
}

Make sure to change the string value to what the API expects.

Auth via OAuth 2.0#

If you are creating an OAuth 2.0 connector, please reach out to Transposit so we can help you add it as an available connector on the Connectors page.

For data connections that implement authentication with OAuth 2.0, Transposit uses a combination of the OAuthConfig and a params property as described below to indicate how to send the access token in the second leg of OAuth.

In your manifest.json, you will need to set the type value to "OAUTH". For example, this code will send a header of the form Authorization: Bearer <token>:

{
  "v2": {
    "swagger": {
      "auth": {
        "type": "OAUTH",
        "authSettingsRequired": true,
        "params": {
          "header": {
            "string": "Authorization"
          },
          "type": {
            "string": "Bearer"
          }
        },
        "oauthConfig": {
          "name": "",  // The name of the data connector
          "authUri": "",  // URL to request authorization, probably has /authorize in its path
          "accessTokenUri": "",  // URL to request the token, probably has /token in its path
          "responseType": "code",  // This will send "response_type: code" in the access request
          "oauthDocumentation": "",  // Link to the documentation, not required
          "scope": "",  // The access scopes you're requesting for your app
          "parameterLocation": "",  // Where to send the client id, secret, code, and grant_type as part of the acess token request. Either "QUERY" or "BODY", defaults to QUERY.
          "needsBasicAuthHeader": "",  // If the connector needs its client id and secret sent in the form Authorization: Basic base64(client_id:client_secret)
          "accessType": "offline",  // empirically only for Google apis
          "prompt": "select_account consent",  //empirically only for Google apis
          "accessTokenMethod": "GET", // GET or POST to make to the oauth endpoint
          "accessTokenPath": "access_token" // What value in the returned map holds the access token
        }
      },
      "source": "source.yml",
      "additionalConfigs": [
        {
          "fieldKey": "oauthConfig.scope",
          "required": false
        }
      ]
    },
    "description": "My data connector"
  }
}

Auth via OAuth Client Credentials#

If your service uses an OAuth Client Credentials flow, your `manifest.json` will look very similar to the OAuth 2.0 one described above, with a different `type` and a subset of the `oauthConfig` settings.

{
  "v2": {
    "swagger": {
      "auth": {
        "type": "OAUTH_CLIENT_CREDENTIALS",
        "authSettingsRequired": true,
        "params": {
          "header": {
            "string": "Authorization"
          },
          "type": {
            "string": "Bearer"
          }
        },
        "oauthConfig": {
          "name": "",  // The name of the data connector
          "accessTokenUri": "",  // URL to request the token, probably has /token in its path
          "parameterLocation": "QUERY",  
          "needsBasicAuthHeader": "true", 
          "accessTokenMethod": "POST", // GET or POST to make to the oauth endpoint
          "accessTokenPath": "access_token" // What value in the returned map holds the access token
        }
      },
      "source": "source.yml"
    },
    "description": "My data connector"
  }
}

Auth via password#

For data connections that implement authentication with a username and password, when users add a credential, Transposit does a POST to the login form and extracts out a response, usually from a set-cookie header. This way Transposit can use an auth token for future API requests.

In your manifest.json, you will need to set the type value to "PASSWORD". For example:

{
  "v2": {
    "swagger": {
      "auth": {
        "type": "PASSWORD",
        "params": {
          "form": {
            "url": {
              "string": "https://api.example.com/v1/authentication/login"
            },
            "form": {
              "value": [
                {
                  "key": "username",
                  "type": "STRING",
                  "val": "{{username}}"
                },
                {
                  "key": "password",
                  "type": "STRING",
                  "val": "{{password}}"
                },
                {
                  "key": "persistLogin",
                  "type": "BOOLEAN",
                  "val": "true"
                }
              ]
            }
          },
          "extract": {
            "string": "$.headers.Set-Cookie.['EqAuth.v1']"
          },
          "header": {
            "string": "Cookie"
          }
        }
      },
      "source": "source.yml"
    },
    "description": "My data connector"
  }
}

The various pieces include:

  • url: This is the URL that the login form posts to. Usually, you can find this as the form action.
  • form: These are the form values Transposit posts to the login URL. They typically include username, password, and a rememberMe boolean. The username and password form that the user sees returns variables for username and password, so you can use mustache style templating (e.g. {{example}}) to use those values in the post.
  • extract: Transposit uses a JSON path extractor to extract the cookie or token from the response. You can use any standard JSON path syntax to extract (e.g. notice the escape syntax in the example above to escape a dot in one of the keys).
  • header: The header you want to send with following authorized requests. Usually, this is "Cookie".

Auth via runtime#

Runtime auth can be used for connectors that require an authentication configuration that does not match the other authentication styles that Transposit supports. Unlike the other authentication styles that have a predefined way of consuming credentials (e.g. Transposit knows to take a Header auth type credential and pass it into the HTTP request as a header parameter), the runtime authentication type leaves consumption of the credential up to you.

In your manifest.json, you will need to set the type value to "RUNTIME". For example:

{
  "v2": {
    "swagger": {
      "auth": {
        "type": "RUNTIME",
        "params": {
          "runtime": {
            "runtimeAuth": {
              "paramNames": ["api_key", "client_secret"]
            }
          }
        }
      },
      "source": "source.yml"
    },
    "description": "My data connector"
  }
}

In this example, the runtime auth includes two parameters that the user will be prompted to provide values for in the UI, similar to other form-like auth types.

Transposit will not automatically do anything with the runtime authentication credential as part of invoking the data connector's operations; instead, the user can retrieve the credential values through a JavaScript operation by invoking the special API method:

function example() {
    return api.auths("connectorAlias");
}

Which will return a JavaScript object with the credential values:

{
    "api_key": "abc123",
    "client_secret": "def456"
}

The developer can now retrieve these credential values in the Transposit operation code, form them into whatever auth configuration style the data connector expects, and then pass them into the data connector's operations just like any other parameters.

Adding a configurable Base URL#

Some connectors use a Base URL that is specific to each individual instance. To allow for this in your connector, you can add an additionalConfigs property in your manifest file to allow each team to configure this property:

{
  "v2": {
    "swagger": {
      ...
      "additionalConfigs": [
        {
          "fieldKey": "baseUrl",
          "required": true,
          "description": "https://<your instance>.com/api"
        }
      ]
    }
  }
}

Adding Pagination#

Many APIs allow clients to fetch pages of data. Transposit uses an OpenAPI extension to support the most common flavors of pagination. Adding this extension will allow Transposit to automatically paginate your API call for you.

The x-pagination extension#

You can add the x-pagination extension to any operation in your OpenAPI file. In it, you can define a resultsPath and/or a pagination strategy to use for the operation. Some fields expect numbers, but others expect values that refer to parameters or locations in the request or response body. For these, we suggest using OpenAPI's runtime-expression syntax to describe the location.

/list:
  get:
    operationId: list
    x-pagination:
      # Describe how to page through results here, further details below

The Result Path#

Many APIs nest the list of results you care about within a JSON structure. You can use the resultsPath to make your connector return those results directly, on its own or in conjunction with any other pagination strategy.

x-pagination:
  resultsPath: "$response.body#/path/to/items"
  • resultsPath describes the location of the results in the response body. The Transposit operation will return these results directly.

Offset Pagination#

Use this strategy if the API lets you skip to the Nth element and return a number of results starting from there.

x-pagination:
  resultsPath: "$response.body#/path/to/items"
  offset:
    offsetParam: "$request.query.offset"
    limitParam: "$request.query.limit"
    maxLimit: 100
  • offsetParam describes where in the request to put the starting index of the response.
  • limitParam describes where in the request to put the number of results to include per page.
  • maxLimit describes the largest page size Transposit should ever request.

Page-Offset Pagination#

Use this strategy if the API lets you fetch the Nth page of results and return a number of results starting from there.

x-pagination:
  resultsPath: "$response.body#/path/to/items"
  pageOffset:
    pageOffsetParam: "$request.query.page_number"
    limitParam: "$request.query.page_size"
    maxLimit: 100
    startPage: 0
  • offsetParam describes where in the request to put the starting index of the response.
  • limitParam describes where in the request to put the number of results to include per page.
  • maxLimit describes the largest page size Transposit should ever request.
  • startPage describes the number of the page to send for the first set of results. This should almost always have a value of either 0 or 1.

Cursor Pagination#

Use this strategy if the API returns a value to include as a parameter with the next request to continue where the last request left off.

x-pagination:
  resultsPath: "$response.body#/path/to/items"
  cursor:
    cursorPath: "$response.body#/pagination/token"
    cursorParam: "$request.query.token"
    limitParam: "$request.query.limit"
    maxLimit: 100
  • cursorPath describes where in the last response to find the cursor token.
  • cursorParam describes where in the next request to put the cursor token.
  • limitParam describes where in the request to put the number of results to include per page.
  • maxLimit describes the largest page size Transposit should ever request.

Next-URL Pagination#

Use this strategy if the API returns a URL that will return results starting from where the last request left off.

x-pagination:
  resultsPath: "$response.body#/path/to/items"
  nextUrl:
    nextUrlPath: "$response.body#/pagination/continuation_path"
    limitParam: "$request.query.limit"
    maxLimit: 100
  • nextUrlPath describes where in the response from the to find the next URL to call.
  • limitParam describes where in the request to put the number of results to include per page.
  • maxLimit describes the largest page size Transposit should ever request.

Header Pagination#

Pagination via header parameters is not currently supported. Please reach out to us if you have a connector that uses header pagination.