Carter Bancroft

The Mysterious Let's Encrypt API

Carter Bancroft -

I'm writing this because, frankly, I cannot believe how little information there is out there on interacting with Let's Encrypt programmatically. Hopefully what follows can save you some pain and suffering.

tldr;

Here's a working Node.js/JavaScript example I built demo'ing how to make a request to the Let's Encrypt API under ACME Draft 18. If you wanna know how the sausage is made, read on.

Let's Encrypt and ACME

First, a distinction: Let's Encrypt is a company/product. It provides an API for generating and managing SSL certs. The Let's Encrypt API is based on the Automatic Certificate Management Environment (ACME) specification.

Due to the secure nature of SSL, the ACME protocol is strict about what requests need to be like. Hence the need for this post.

Goals

This article won't explain everything but it will provide a baseline for understanding this slightly intimidating topic. Specifically:

  1. We'll learn how to make a signed request to the Let's Encrypt API via the newAccount endpoint.
  2. We'll get more comfortable with the ACME spec. I've provided links in almost every section to relevant areas and I recommend taking a look.

A Word on Third Party Let's Encrypt Libraries

In Node land, for example, Greenlock is the main contender. My experience however is that these libraries can be mysterious and complicated. Debugging them is tricky and the documentation is often confusing. Understanding ACME even a little could save you a lot of pain. Further, if you have the right use-case, rolling your own cert generating Let's Encrypt client might be worthwhile.

A Caveat

ACME isn't a complete standard yet. Drafts are still being written and Let's Encrypt keeps their API in sync with ACME's current state.  This means information here soon be outdated. In fact this post was inspired by the only other real article I could find on how to do this which was written in 2017 and, as helpful as it was, is no longer accurate.

Hopefully with each draft there will be fewer and fewer changes but until it's finalized we're stuck with uncertainty. This post is based on draft 18.

Let's get started.

The API URL

Let's Encrypt's root production API endpoint is:

https://acme-v02.api.letsencrypt.org/

They also provides a sandbox API for testing which we'll be using here, because we're good API neighbors:

https://acme-staging-v02.api.letsencrypt.org/

Getting API Endpoints

Let's Encrypt provides a directory to look up API endpoints. Unlike most APIs where you just find endpoints by googling the docs and hardcoding URLs somewhere in your app, ACME provides a phone book.

From your terminal, try this:

$ curl https://acme-staging-v02.api.letsencrypt.org/directory

You should see something like:

{
  "4nbvPXkqKak": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
}

There's your phone book. It has all the endpoints available to us for working with the Let's Encrypt API. newAccount and newNonce are the only ones we'll use today.

Try curling the production URL or even v01 URLs and note the differences.

Instead of hardcoding these endpoints, we'll always look up them in /directory before calling them. If endpoints change between drafts (and they do) this will provide some robustness. Here's what I did:

const request = require('request-promise-native')
const directoryUrl = 'https://acme-staging-v02.api.letsencrypt.org/directory'

// Here are all the actions provided in the directory. We can use this for
// looking up the endpoints we need.
const ACTIONS = {
  keyChange: 'keyChange',
  newAccount: 'newAccount',
  newNonce: 'newNonce',
  newOrder: 'newOrder',
  revokeCert: 'revokeCert',
}

const getDirectory = async () => {
  const options = {
    uri: directoryUrl,
  }

  const res = await request.get(options)
  let parsed = JSON.parse(res)
  return parsed
}

Then, to get an endpoint we can do something like:

const directory = getDirectory()
const endpoint = directory[ACTIONS.newAccount]
If we REALLY want to be good API neighbors we should cache the directory so we don't need to make the call every single time we make a request.

Making Requests

Unlike you're probably used to, requests to any ACME based API need to be signed. This is the most complicated part of the whole process so we'll take this step by step.

Step 1: Base64 Encoding

Pretty much everything we send over the wire has to be Base64 encoded and made "URL safe." In this context URL safe means, + becomes -, / becomes _, and = will be removed.

Here's the functions:

// Clean up a base64 encoded string to be URL safe
const makeUrlSafe = encoded => {
  const urlSafe = encoded
    .replace(/\+/g, '-')
    .replace(/\//g, '_')
    .replace(/=/g, '')

  return urlSafe
}

// Encode a JSON object into a base64, url safe string
const encodeJson = obj => {
  const stringified = JSON.stringify(obj)
  const encoded = Buffer.from(stringified).toString('base64')
  const urlSafe = makeUrlSafe(encoded)

  return urlSafe
}

Step 2: The Nonce

Per ACME spec, most endpoints require a nonce. The nonce is a server side generated one-time use token. If a request includes a nonce that was part of some previous request, Let's Encrypt rejects it. This means if a previous request was intercepted it can't be faked.

It turns out Let's Encrypt includes a new nonce in every single response, no matter the endpoint so if we're making a chain of requests we can just use the nonce from the previous response for the next one.

For our case the API also provides a specific endpoint for getting nonces and since this is our first request we'll need to use it. In the Let's Encrypt directory it was under newNonce.

Let's make a HEAD request to that endpoint and parse the response for the replay-nonce property. Check this out:

const getNonce = async url => {
  const options = {
    uri: url,
  }

  const res = await request.head(options)
  return res['replay-nonce']
}

The url param comes from the directory object we got earlier. Like so:

const nonceUrl = directory[ACTIONS.newNonce]

Step 3: Generating Keys

To sign URLs we need a keypair that we can use to signed request objects on our side and allow Let's Encrypt to verify the request on their side.

Let's Encrypt and the current ACME draft support RSA and EdDSA for signing. EdDSA is the future but I'm going with RSA since that's what most examples I've seen use. It's good enough.

To generate an RSA keypair you could either do it from your terminal or programmatically. In most cases you'll only want to generate this once, store it in a secure place and re-use it for all your requests since this keypair will be your account credentials.

For simplicity, I'm doing it programmatically using and NPM package, keypairs (npm i --save @root/keypairs), which includes some niceties for dealing with ACME. It's from the same folks who built Greenlock.

Example time:

const Keypairs = require('@root/keypairs')

const generateKeypairs = async () => {
  // Necessary stuff for generating RSA pairs
  const options = {kty: 'RSA', modulusLength: 2048}
  const pair = await Keypairs.generate(options)

  const privatePem = await Keypairs.export({jwk: pair.private})
  const publicPem = await Keypairs.export({jwk: pair.public})

  return {
    publicJwk: pair.public,
    publicPem: publicPem,
    privateJwk: pair.private,
    privatePem: privatePem,
  }
}

For request authentication, the ACME spec requires JWK (or a KID but that's for another post). Those links provide details but just know that @root/keypairs generates a valid JWK automagically.

Above, I'm returning an object that includes private and public JWKs and their PEM counterparts because we'll need those later.

Step 4: Building our JSON Web Signature

Request data to be sent to any ACME based API needs to be in the JSON Web Signature (JWS) format. Now that we can generate keypairs and base64 encode objects we can build up a JWS for our request.

A JWS requires the following (from the request authentication section):

  1. A "protected header"
  2. The request payload
  3. A signature

The Protected Header

The protected header is an object with the following shape:

{
  alg: '', // The algorithm used for generating our keypaid and signature
  nonce: '', // The replay nonce we got from Let's Encrypt
  url: '', // The full URL we are making a request to, https and all
  jwk: <object>, // The JWK object we got from our keypairs
}

That object then needs to be base64/URL safe encoded into a string. Here's some code:

const getProtectedHeader = (url, nonce, publicJwk) => {
  const headers = {
    alg: 'RS256',
    nonce,
    url,
    jwk: publicJwk,
  }

  const protectedHeader = base64.encodeJson(headers)
  return protectedHeader
}

The Request Payload

This is a typical JSON object specifying options required for the request. It will vary depending on what endpoint you're hitting but since our goal is to create an account this is what the payload should look like (per the ACME spec):

  const payload = {
    contact: ['mailto:test@test.com'],
    termsOfServiceAgreed: true
  }

This provides a contact for the account (fill in your own, obvi) and also automatically agrees to Let's Encrypt's Terms of Service.

Per usual, this object needs to be base64/url safe encoded into a string.

const encodedPayload = base64.encodeJson(payload)

The Signature

Now, we can generate a signature. To do this we need:

  • The private PEM string from the keypair we generated.
  • The protected header.
  • The encoded payload.

Code:

const crypto = require('crypto')

const getSignature = (privatePem, pro, payload) => {
  const sign = crypto.createSign('SHA256').update(`${pro}.${payload}`)
  const signature = sign.sign(privatePem, 'base64')
  const encodedSignature = base64.makeUrlSafe(signature)

  return encodedSignature
}

We're using Node's built-in Crypto lib to do the signing here. But, what the heck is this: ${pro}.${payload}?

Turns out, the value we want to sign is the encoded protected header followed by a period followed by the encoded payload.

Also note that the signing algorithm will already generate a base64 encoded string. The only thing left to do is to use our base64 code to make it URL safe.

At this point, you've got yourself a signature, let's build a JWS:

  const jws = {
    protected: protectedHeader,
    payload: encodedPayload,
    signature,
  }

That sucker right there is just about everything we need.

Step 5: Making a Signed Request

The actual request object requires, as it's body, that JWS we just generated. It also requires this special content type application/jose+json and the request type we're making is POST.  If you're using the Node request lib this is what your options will look like:

  const options = {
    uri: url,
    json: jws,
    headers: {
      'Content-Type': 'application/jose+json'
    },
  }

Then you can just do something like:

return await request.post(options)

In our case this will create a new account and the response will mostly be our request echoed back. At this point, using the keypairs we generated we could go on to request SSL certs. Neat eh?

Handling Errors

One nice thing about the ACME spec is it has pretty useful and well documented error responses. You should familiarize yourself with the documentation here so you can debug what the heck is going on when something blows up.

Resources