OpenAPI Test Case Extensions

The OpenAPI description format (most commonly known as Swagger) allows its users to extend their specification for an API at various points by making use of vendor extensions. This allows them to add any additional data that can better describe the API. By convention, these extension properties are always prefixed by x- and must have a valid JSON value. APIMatic utilizes this feature to provide users with extensions that lets them extend their API specification in order to configure and enhance the output from APIMatic products that suits their needs better.

This document focuses on extensions that allow users to specify test data in their OpenAPI/Swagger API specification which APIMatic can then utilize to automatically generate valid test cases in each language upon SDK generation. Other extensions available are documented here:

Cover

The test data extensions that APIMatic offers are based on the Gavel specification.

Gavel Specification

Gavel is a tool from Apiary.io that is used to validate HTTP API calls based on comparisons between expected and real requests/response JSON objects. For our purposes, we have made use of only the HTTP Request and Expected HTTP Response JSON objects present in the gavel specification.

HTTP Request

This helps you define the details about an HTTP request that will be sent.

Property	Type	Explanation
method	`string`	This refers to the HTTP methods used for sending the request. Valid values include `GET`, `POST`, `DELETE`, `PATCH`, `PUT`
uri	`string`	Uniform resource identifier used to locate and identify a resource. May contain query parameters `(/pets?pet-id=1)` and template parameters `(/pets/{pet-id})`.
headers	`Map[string, string]`	Key-value pairs containing information for request headers like the `Content-Type`, `Accept`, etc. Key should be the header name while the value would be the header value.
body	`string`	The request body can contain input parameters depending on the `Content-Type` specified in the headers. This is not applicable in case of the HTTP method `GET`.

Example:

{
  "method": "GET",
  "uri": "/ip",
  "headers": {
    "user-agent": "curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5",
    "host": "httpbin.org",
    "accept": "*/*"
  },
  "body": ""
}

Expected HTTP Response

This helps you define the expected response for the HTTP request you sent using details of the previous section.

Property	Type	Explanation
statusCode	`string`	The expected HTTP status code of the response e.g. `200`.
headers	`Map[string, string]`	Key-value pairs containing information for request headers like the `Content-Type`, `Accept`, etc. Key should be the header name while the value would be the expected value of the header.
body	`string`	The expected body data of the response.

Example:

{
  "statusCode": "200",
  "headers": {
    "content-type": "application/json",
    "date": "Wed, 03 Jul 2013 13:30:53 GMT",
    "server": "gunicorn/0.17.4",
    "content-length": "30",
    "connection": "keep-alive"
  },
  "body": "{\n  \"origin\": \"94.113.241.2\"\n}"
}

Specifying Test Cases Using Gavel Specification

To specify one or more test cases based on gavel specification in OpenAPI/Swagger would require the use of our x-unitTests extension in the Operation Object (v2, v3). This extension is an array of objects where each object is comprised of the following:

Property	Type	Explanation
request	`HTTP Request Object`	Required This is where you define the input test data for your operation.
expectedResponse	`Expected HTTP Response Object`	Required This is where you define the response specific test data corresponding to the request data defined under the `request` property.

APIMatic will take this array of test cases within each operation and convert them to test cases in the preferred programming language. For any test case, if the real response from the operation is different from that specified then the test case will fail. In this way the two objects help specify test data for test cases.

Example:

A POST operation containing one query parameter pet-id that retrieves the information of a pet (e.g. its name) based on its pet-id can be tested with a test case using the following specification:

"post": {
  "operationId": "findPets",
  "parameters": [
  {
    "name": "pet-id",
    "in": "query"
  }],  
  "x-unitTests": [
    {
      "request": {
        "method": "GET",
        "uri": "/pets?pet-id=1"
      },
      "expectedResponse": {
        "statusCode": "200",
        "headers": {
          "content-type": "application/json"
        },
        "body": {
          "name": "Dolly"
        }
      }
    }
  ]
}

note

If the request object contains parameters within body (invalid for GET method) then the Content-Type MUST be present in the request headers property.

Extended Gavel Specification

We have extended the Gavel specification with some configuration flags to fine-tune test case generation in APIMatic. These are optional flags that will assume default values if they are not explicitly specified.

Configuration Flags in Unit Test Object

Flags	Type	Default Value	Explanation
x-testName	`string`	Operation `name`	Specifies the name of the test case to be generated.
x-testDescription	`string`	Operation `description`	Describes what the test case does.
x-testShouldPass	`boolean`	`true`	Should this test pass? If `false`, the test would be required to fail to pass.
x-testEnabled	`boolean`	`true`	Is this test enabled? Disabled tests are not generated and are not validated. Tests can be disabled in case they are outdated after an operation description is updated.

Example:

  "x-unitTests":[
    {
      "request":{
        "method":"GET",
          "uri":"/pets?pet-id=1"
      },
      "expectedResponse":{
        "statusCode":"200",
        "headers": {
              "content-type": "application/json"
        },
        "body":" {\"name\":\"Dolly\"}"
      },
      "x-testName":"getPetInfo",
      "x-testEnabled":"true",
      "x-shouldPass":"true",
      "x-testDescription":"Get pet information from its id"
    }
  ]

Configuration Flags in HTTP Expected Response Object

These flags can be specified along with the Expected Response Object:

Flags	Type	Default Value	Explanation
x-allowExtraHeaders	`boolean`	`true`	Specifies whether other headers than those specified in `headers` object within the `expectedResponse` are allowed or not.
x-bodyMatchMode	Enum [`string`]	`NONE`	Valid values are `NONE`, `RAW`, `KEYS`, `KEYSANDVALUES`, `NATIVE`. Specifies how `body` in the `expectedResponse` is compared to the actual response body. More information on the Match modes can be found at Body Match Modes.
x-arrayOrderedMatching	`boolean`	`false`	If `true`, testing of arrays will include order checking of elements as well.
x-arrayCheckCount	`boolean`	`false`	If `true`, the arrays will be tested to see if they are equal in length. If both `x-arrayOrderedMatching` and `x-arrayCheckCount` are `true`, arrays will be strictly checked for equality i.e. their order as well as size must match.

Example:

  "x-unitTests": [
    {
      "request": {
        "method": "GET",
        "uri": "/pets?pet-id=1"
      },
      "expectedResponse": {
        "statusCode": "200",
        "headers": {
          "content-type": "application/json"
        },
        "body": " {\"name\":\"Dolly\"}",
        "x-bodyMatchMode": "KEYS",
        "x-allowExtraHeaders": "true"
      }
    }
  ]

Inline Test Data Specification

To specify test values for security parameters or operation parameters you can also use the x-testValue extension property.

Security Scheme Object

You can use the x-testValue within the Security Scheme Object to specify a test value for the particular security parameter e.g. you can specify a test API key as follows:

  "apikey": {
    "type": "apiKey",
    "name": "apikey",
    "in": "query",
    "x-testValue":"4d883edc9e6eba86bf1cc2dd4024d612"
  }

note

The alternative to this would be to specify this value within gavel specification (within the Request uri if the apikey is in query or within the Request headers if the apikey is in headers)

Basic authentication generally requires a username and a password. To specify test values for these parameters you can declare x-testValue as an array of type BasicAuthTestValue Object.

BasicAuthTestValue Object consists of two properties:

Property	Type	Details
name	String	Name of the parameter e.g. username
value	String	A test value for that parameter

Example

  "basicAuth": {
    "type": "basic",
    "x-testValue":[{
      "name":"username",
      "value":"user123"
    },
    {
      "name":"password",
      "value":"pass123"
    }]
  }

Operation Parameters

We can also specify inline test values for operation parameters using x-testValue as follows:

  "get": {
    "operationId": "findPets",
    "parameters": [
      {
        "name": "pet-id",
        "in": "query",
        "description": "",
        "required": true,
        "x-testValue": "1"
      }
    ]
  }

note

In case a parameter test value is specified in both the gavel specification and in inline specification of parameters the value in the gavel specification will be given preference.

Gavel Specification​

HTTP Request​

Expected HTTP Response​

Specifying Test Cases Using Gavel Specification​

Extended Gavel Specification​

Configuration Flags in Unit Test Object​

Configuration Flags in HTTP Expected Response Object​

Inline Test Data Specification​

Security Scheme Object​

Example​

Operation Parameters​

Gavel Specification

HTTP Request

Expected HTTP Response

Specifying Test Cases Using Gavel Specification

Extended Gavel Specification

Configuration Flags in Unit Test Object

Configuration Flags in HTTP Expected Response Object

Inline Test Data Specification

Security Scheme Object

Example

Operation Parameters