SHARE THIS ARTICLE

Swagger Test Case Extensions

Swagger allows its users to extend their Swagger specification for an API at various points by making use of Swagger vendor extensions. This allows them to add any additional data that can better describe the API. By convention, these extension properties are always prefixed by x- and must have a valid JSON value. Using these extensions in the specification, the users can specify relevant test data which APIMATIC can then utilize to automatically generate valid test cases in each language upon SDK generation.

Cover

Gavel Specification

Gavel is a tool from Apiary.io that is used to validate HTTP API calls based on comparisons between expected and real requests/response JSON objects. For our purposes, we have made use of only the HTTP Request and Expected HTTP Response JSON objects present in the gavel specification.

HTTP Request

PropertyExplanation
methodThis refers to the HTTP methods used for sending Request like GET, POST, DELETE, PATCH, PUT
uriUniform resource identifier used to locate and identify a resource. May contain query parameters (/pets?pet-id=1) and template parameters (/pets/{pet-id})
headersKey-value pairs containing information like content-type, acceptable encoding etc.
bodyThe Request body can contain input parameters depending on the content-type specified in the headers. This is not applicable in case of method GET

Example:

{
  "method": "GET",
  "uri": "/ip",
  "headers": {
    "user-agent": "curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5",
    "host": "httpbin.org",
    "accept": "*/*"
  },
  "body": ""
}

Expected HTTP Response

PropertyExplanation
statusCodeThis refers to the HTTP status codes which are standard response codes that help identify if request was successful or not e.g. 200 means a successful request
bodySchemaExpected body can be defined using JSON schema specified under this property
headersKey-value pairs containing information like content-type, acceptable encoding etc.
bodyThe actual response data is located inside this response body

Example:

{
  "statusCode": "200",
  "headers": {
    "content-type": "application/json",
    "date": "Wed, 03 Jul 2013 13:30:53 GMT",
    "server": "gunicorn/0.17.4",
    "content-length": "30",
    "connection": "keep-alive"
  },
  "body": "{\n  \"origin\": \"94.113.241.2\"\n}",
  "bodySchema": {
    "type": "object",
    "$schema": "http://json-schema.org/draft-03/schema",
    "required": true,
    "properties": {
      "object": {
        "type": "object",
        "required": false,
        "properties": {
          "origin": {
            "type": "string",
            "required": true
          }
        }
      }
    }
  }
}

The Request object can be used to extract input parameters for an operation and the Expected Response object can help indicate the type of response expected from this operation. If the real response from the operation is different from that specified then the test case will fail. In this way the two objects helps specify test data for test cases.

Specifying Test Case Using Gavel Specification

So now coming to how a user can actually specify test cases in swagger using vendor extensions and gavel specification. The users can define an array of test cases within the Operation Object that can help test that particular operation. This array of test cases is represented by the name x-unitTests and contains objects each representing a single test case. APIMATIC will take this array of test cases within each operation and convert them to test cases in the preferred programming language.

Example:

"get": {
  "operationId": "findPets",
  "parameters": [
    {
    ------------
  }],
  "x-unitTests":[
    {
      "request":{
        -------
      },
      "expectedResponse":{
        -------
      }
    }
  ]
}

The single test case object contains the Request object and the Expected Response object. These are used to model the input parameters received in the request and the resulting response from the operation. The details of how to specify these objects are given in the previous section.

Example:

A GET operation containing one query parameter pet-id that retrieves the information of a pet (e.g. its name) based on its pet-id can be tested with a test case using the following specification:

  "x-unitTests": [
    {
      "request": {
        "method": "GET",
        "uri": "/pets?pet-id=1"
      },
      "expectedResponse": {
        "statusCode": "200",
        "headers": {
          "content-type": "application/json"
        },
        "body": {
          "name": "Dolly"
        }
      }
    }
  ]

In this way multiple objects can be specified within the array to generate multiple test cases.

Note   If the Request object contains parameters within Body (invalid for GET method) then the content-type MUST be present in the Request headers.

Extended Gavel Specification

We can extend the gavel specification to configure some of the flags for test case generation in APIMATIC. These are optional flags and absence of any will lead to them acquiring default values.

FlagsTypeDefault ValueExplanation
x-testNameStringOperation nameSpecifies the name of the test case to be generated
x-testDescriptionStringOperation descriptionDescribes what the test case does
x-testShouldPassBooleanTrueShould this test pass? If false, the test would be required to fail to pass.
x-testEnabledBooleanTrueIs this test enabled? Disabled tests are not generated and are not validated. Tests can be disabled in case they are outdated after an operation description is updated

Example:

  "x-unitTests":[
    {
      "request":{
        "method":"GET",
          "uri":"/pets?pet-id=1"
      },
      "expectedResponse":{
        "statusCode":"200",
        "headers": {
              "content-type": "application/json"
        },
        "body":" {\"name\":\"Dolly\"}"
      },
      "x-testName":"getPetInfo",
      "x-testEnabled":"true",
      "x-shouldPass":"true",
      "x-testDescription":"Get pet information from its id"
    }
  ]

Flags in expectedResponse object

FlagsTypeDefault ValueExplanation
x-allowExtraHeadersBooleanTrueSpecifies whether other headers than those specified in Headers object within the expectedResponse are allowed or not
x-bodyMatchModeEnum [String]NONEValid values are [“NONE”,“RAW”,“KEYS”, “KEYSANDVALUES”,“NATIVE”]. Specifies how Body in the expectedResponse is compared to the actual response body. More information on the Match modes can be found at Body Match Modes
x-arrayOrderedMatchingBooleanFalseIf true, arrays testing will include order checking of elements as well.
x-arrayCheckCountBooleanFalseIf true, the arrays will be tested to see if they are equal in length. If both x-arrayOrderedMatching and x-arrayCheckCount are true, arrays will be strictly checked for equality i.e. their order as well as size must match
x-matchResponseSchemaBooleanTrueSet whether the test case should check the response body with the schema if it is a model. (Not implemented yet)
  "x-unitTests": [
    {
      "request": {
        "method": "GET",
        "uri": "/pets?pet-id=1"
      },
      "expectedResponse": {
        "statusCode": "200",
        "headers": {
          "content-type": "application/json"
        },
        "body": " {\"name\":\"Dolly\"}",
        "x-bodyMatchMode": "KEYS",
        "x-allowExtraHeaders": "true"
      }
    }
  ]

Inline Test Data Specification

To specify test values for security parameters or operation parameters you can also use the x-testValue extension property.

Security Scheme Object

You can use the x-testValue within the Security Scheme Object to specify a test value for the particular security parameter e.g. you can specify a test API key as follows:

  "apikey": {
    "type": "apiKey",
    "name": "apikey",
    "in": "query",
    "x-testValue":"4d883edc9e6eba86bf1cc2dd4024d612"
  }

Note   The alternative to this would be to specify this value within gavel specification (within the Request uri if the apikey is in query or within the Request headers if the apikey is in headers)

Basic authentication generally requires a username and a password. To specify test values for these parameters you can declare x-testValue as an array of type BasicAuthTestValue Object.

BasicAuthTestValue Object consists of two properties:

PropertyTypeDetails
nameStringName of the parameter e.g. username
valueStringA test value for that parameter

Example

  "basicAuth": {
    "type": "basic",
    "x-testValue":[{
      "name":"username",
      "value":"user123"
    },
    {
      "name":"password",
      "value":"pass123"
    }]
  }

Operation Parameters

We can also specify inline test values for operation parameters using x-testValue as follows:

  "get": {
    "operationId": "findPets",
    "parameters": [
      {
        "name": "pet-id",
        "in": "query",
        "description": "",
        "required": true,
        "x-testValue": "1"
      }
    ]
  }

Note   In case a parameter test value is specified in both the gavel specification and in inline specification of parameters the value in the gavel specification will be given preference.

Conclusion

In this way the swagger vendor extensions can be effectively used to specify additional data like test data to your API description which can be really useful in automatic generation of test cases for testing the operations specified in the swagger specification.


Have questions? Submit a request.