HyperREST

08 Jun 2014
KATT - Klarna API Testing Tool

This is a written distillation of a talk given at Stockholm Node.js Redux with these slides and these files: some KATT runs against the Spotify Metadata API.

By pure coincidence (?!), Klarna also decided to talk about the same tool, KATT, at Erlang User Conference (just two weeks apart).


Definition of testing - KATT POV

  1. Testing
  2. behaviour of
  3. HTTP *
  4. JSON *
  5. APIs
  6. with an Erlang/NodeJS * tool

The * stands for not only, because

  • KATT could be made HTTP agnostic
  • KATT is already syntax agnostic, just that only plaintext and JSON support is built in
  • KATT is originally written in Erlang, and ported to NodeJS

The port to NodeJS happened because I was writing katt-player (only exists in NodeJS) which meant porting 80% of KATT’s logic, so why not do it all the way?

In higher abstraction lingo, that means

  1. Testing
  2. chained semantics of
  3. a transfer
  4. and messaging
  5. protocol

and it does NOT mean testing

  • usability
  • standard
  • performance
  • typing *

Check out SmartBear’s talks at Nordic APIs 2013


Common practice

Today, HTTP APIs are tested in NodeJS along these lines:

# Mocha + Chai + SuperAgent + SuperTest
should = require('chai').should()
api = require('supertest') 'http://example.com'
describe 'my api', () ->
  describe 'GET /collection', () ->
    it 'responds as expected', (done) ->
      api.get('/collection')
        .set('authorization', 'Bearer AbCdEf123456')
        .set('accept', 'application/json')
        .expect(200)
        .expect('content-type', /json/)
        .expect('cache-control', 'no-cache')
        .end (err, res) ->
          return done err  if err?
          res.body.should.have.property('items').and.be.instanceof Array
          done()

You know where I am getting at, but just in case, my main points for this slide:

  • how much of this is HTTP/JSON/behaviour, and how much is boilerplate?
  • when “responds as expected” ends up being the norm - what do you actually test?
  • naming, one of the two hardest things in computer science…
  • can you chain requests into a workflow? yes, more boilerplate - callback hell, async, …
  • can you reuse previous requests? - maybe, with even more boilerplate
  • can you share them with your customers? improbable, as it will be filled with business critical edge-cases
  • are they standalone?
  • can a non-js-developer understand them, have an overview of them?

Alternative practice

I thought about giving this talk primarily because of a post by Steve Klabnik, now at Balanced Payments on “TDD your API”.

Since 2012 (almost the same time when KATT was born), their concept and tooling went through different phases, from reStructured Text, YAML, and ended up with Cucumber, along these lines:

Feature: Credit cards
  Scenario: Add a card to a customer
    Given I have tokenized a card
    And I have created a customer
    When I make a PATCH request to /cards/:card_id with the body:
      """
        [{
          "op": "replace",
          "path": "/cards/0/links/customer",
          "value": ":customer_id"
        }]
      """
    Then I should get a 200 OK status code
    And I make a GET request to /cards/:card_id
    Then the response is valid according to the "cards" schema
    And the fields on this card match:
      """
        {
          "links": { "customer": ":customer_id" }
        }
      """

When I first read that, I got stuck. Maybe because of things like this, or maybe because I don’t buy into Cucumber, but surely because I couldn’t understand how you could spend time (2 years!) deciding and changing syntax.

Neither of the 3 DSLs has anything to do with HTTP. They all wrap HTTP tests, but you don’t write HTTP, nor JSON.

Similarly, they don’t wrap user behaviour either (see the front page of Cucumber), but system behaviour. I stands for BalancedPayment’s backend.

Where is the HTTP ?


KATT practice

A KATT scenario is inspired by a copy-cat of the legacy Apiary blueprint. A bit of text and markup, but basically pure HTTP transactions (request-response) chained together to describe (not an API, like Apiary does) but a user scenario. A if-this-then-that pure HTTP scenario.

For Klarna Checkout this was perfect. Lots of HTTP transactions put together were describing a great deal of paths in a huge state machine with lots of “if x, but if y, and z, …”. Complex, but not complicated!

You can see an example of a KATT scenario here and here.

Make sure to read the README, so that I don’t have to repeat what it does and how it does it

KATT’s functionality can be summarized to:

  • make sure that a chain of HTTP transactions (request-reponse) passes different assertions
  • assertions are by default partial (i.e. don’t check all HTTP headers)
  • assertions can be made complete via markup (i.e. don’t allow more JSON properties in this object than the ones defined in this test)
  • you can get values from, and set values to a key-value store specific for a scenario, so that you handle cases when data is dynamic, and you don’t care what value it has, but you need it in a later HTTP request/response
  • built in support for plaintext and JSON, but extensible

Simple. Sharp. What you need!


KATT additions

Keeping API tests to yourself doesn’t make much sense, and given a few years, sharing them publicly with customers will be a defacto standard.

Klarna didn’t do that publicly, but it did so internally - with the Klarna Checkout frontend team. Thanks to KATT player, it was possible for HTTP APIs tests to instantiate a mock backend for the frontend team to do their job, while the real API work is implemented.

KATT util is also an interesting tool: it lints and formats KATT scenarios (useful when non-developers are actually writing the tests and don’t actually run them; just lint and format in Jenkins and bingo!). This tool also comes with a har2katt converter. Never heard of HAR ? You have support for it in Firefox and Chrome, but could be tons better!


Future

It will be interesting to hear from Jonathan at EUC on KATT. Don’t know if they added new functionality (and didn’t push internal changes to github - tsk tsk tsk!) or if they found unforeseen limitations.

NOTE: the NodeJS tools seem a bit abandonware, as I haven’t had a reply to my github issues. I am running some forks for my personal use at https://github.com/for-GET

From my POV, it was a pleasure to code and work with KATT and the tooling around it.

Despite that, I think KATT can be even much simpler, for instance:

  • the format for instance - could be Literate HTTP. Allows for 100% pure HTTP transactions stiched together. If you want some human descriptions of the request/response, just add a special HTTP header.
  • validators - include just the plaintext = validator for method, request_target, header, payload, status_code. Define an interface for extensions
  • define proper steps - currently KATT does 3 things at one - gets values, sets values and validates

First step - literate HTTP - check! Next steps, to come.


Key points

  • test HTTP by writing HTTP
  • test JSON by writing JSON
  • test API behaviour by writing API behaviour
  • STOP boilerplating and cucumbering!
  • make your API tests public
  • let customers run a mock of your API locally via katt-player

Questions?

  • How do I test types with KATT? - You don’t. Validate request payloads against a schema (and throw an HTTP error otherwise), but also validate your response payloads against a schema (and log an error and trigger an alarm otherwise) in production. For JSON use JSON schema.

Andrei