Stager API

Apcera stagers let you create and manage your organization's lifecycle management processes. Stagers run as jobs, and you can write them in any language you wish. They are run via staging pipelines, which execute stagers in a specified order.

Apcera provides you with the Stager API for writing custom stagers. The Stager API is a web-services based API that provides the mechanism for creating stagers. In addition, Apcera provides a library in the Ruby language to negotiate the complexities of the Stager API and provide a simple front end.

This article describes how to use the web services-based Stager API. If you want to use the Stager API Ruby library, refer to this article.

If you are just getting started writing custom stagers, or prefer less complexity, it is recommended that you use the Ruby library for the Stager API. Anything you can do in the web service-based Stager API, you can do with the Ruby library.

How to write a custom stager using the Stager API

To interact with the staging coordinator, the stager uses the URL specified in the STAGER_URL environment variable

The stager performs the following steps:

1. Download a package by sending a GET command to $STAGER_URL/data. The package is normally a gzipped tarfile.

2. Unzip and untar the package contents.

3. Perform tasks on the package; if the stager alters the package, it creates a gzipped tarfile of the altered package and uploads it with a POST to $STAGER_URL/data. For each task,

   • If successful, issue a POST to $STAGER_URL/done.

   • Otherwise, issue a POST to $STAGER_URL/failed.

API endpoints

Operation	Endpoint	Description
GET	$STAGER_URL/data	Download package
POST	$STAGER_URL/data	Upload package to Staging Coordinator
GET	$STAGER_URL/meta	Obtain package metadata
PUT	$STAGER_URL/meta	Modify package metadata
POST	$STAGER_URL/snapshot	Updates the package with a snapshot of the stager's filesystem
POST	$STAGER_URL/done	Marks successful completion of Stager
POST	$STAGER_URL/failed	Marks failure of Stager
POST	$STAGER_URL/relaunch	Relaunches Stager with new dependencies

Status codes

You might see the following codes in response to HTTP commands sent to $STAGER_URL:

Status code	Description
200	Successful response
450	Request is missing a UUID field
451	Given UUID is not valid
452	UUID is not current
445	Failed to marshal environment variables during package metadata download
448	Specified resource not supported during package metadata alteration
499	May be returned for multiple errors. Error message describes the error.

These codes and messages are subject to change.

Uploading data

Uploading new package data can be done by making a POST to $STAGER_URL/data. The request body then replaces the contents of the package. When uploading, you must also supply the SHA256 checksum through the query string on the request, so you must make the request to $STAGER_URL/data?sha256={sha256}. The SHA256 cannot use form fields for the request because stagers use the request body for the package data. The upload request must also have the Content-Length header set to the size of the data being uploaded. Most HTTP clients automatically do this. If not, you will see a "You must provide a Content-Length when uploading" error message returned.

Snapshots

Update the package by making a POST to $STAGER_URL/snapshot. The snapshot functionality allows capturing the changes made to the filesystem of the stager from outside of the isolation context. Apcera uses a layered filesystem outside of the stager, so snapshot helps to capture only what has been changed within the stager and skips files that haven't been modified. This is useful in scenarios where a stager makes changes to the filesystem in numerous locations rather than in one directory.

Typically, a runtime stager puts the application in an /app directory and expects everything to be located in that directory. However, if you write a stager to compile runtime packages, the stager may create files in many places or run scripts. There may be changes across the filesystem, so the snapshot is ideal in this scenario because it captures only the files that have changed.

The snapshot request can be filtered within the stager by passing the directory value with the request.

Metadata

Package metadata can be accessed via GET $STAGER_URL/meta.

Sample response:

{
  "dependencies": [
    {"type": "os", "name": "linux"},
    {"type": "package", "name": "build-essential"}
  ],
  "provides": [
    {"type": "runtime", "name": "ruby"},
    {"type": "runtime", "name": "ruby-1.9"}
  ],
  "environment": {
    "PATH": "/opt/apcera/ruby-1.9.3/bin:$PATH"
  },
  "templates": [
    {
      "path": "foo",
      "left_delimiter": "<<",
      "right_delimiter": ">>"
    },
    {
      "path": "bar"
    }
  ]
}

The template uses default delimiters {{ and }}.

The Stager can alter the metadata by issuing a PUT to $STAGER_URL/meta. The PUT accepts the following form fields:

Request Field	Type	Description
resource	string	Signifies what is being altered, one of: environment, provides, dependencies, templates.
action	string	One of: add, remove, for all resource form values.
key	string	Specifies the property name to set under the given resource.
value	string	Value associated with a key, used for add action only.
type	string	Required for dependencies and provides, can be one of the following: os, runtime, file, package
name	string	Required for dependencies and provides, can be any string.

Many languages and frameworks support flipping a development/production switch. In Ruby, a curl call to set the RAILS_ENV variable to production would be:

$ curl -X PUT -F resource=environment -F action=add -F key=RAILS_ENV
-F value=production $STAGER_URL/meta

Special package environment variables

A stager can set environment variables on the package that have a special purpose when the package is used with a job. At run time, environment variables from the job and all the packages are merged together and layered so that the package for application has precedence over its dependencies (such as the Ruby or Node runtime and so forth).

These special environment variables are typically meant to provide a default if the equivalent value isn't set on the job itself.

• START_COMMAND — The default command to launch the application. Typically a stager determines the command to launch the application based on its language/framework and dependencies.

• START_PATH — The directory in which to start the application. Typically, stagers put the application in a /app directory.

Relaunching a Stager

The system cannot dynamically alter the filesystem of a Stager once it is running. To finish staging, a Stager should relaunch if it needs files from previously altered dependencies. Relaunch includes altered dependencies and environment variables.

To relaunch, make a POST to $STAGER_URL/relaunch.

• Apcera only does basic SHA checksum and length validation checks on packages.
• When uploading a new package, Apcera expects the tar file to include the full path for all files. So if a file named index.html is meant to be in /app, the tarfile should have it as /app/index.html. Apcera extracts all packages relative to the Isolation Context's root directory.
• Stagers stream output from stdout and stderr back to users.
• Currently, Stagers will run for a maximum time limit of ten minutes. The Staging Coordinator terminates Stagers when this limit is reached, and thus fails the staging process.

Package dependencies within a stager

All of the dependencies of the package being staged are made available to the stager within its filesystem under a /stagerfs directory. This allows the stager to make use of the package's dependencies for any steps required. Often the libraries needed by the application must be compiled or processed by the same runtime version. For instance, you need the same version of Node to install npm packages that will be used when the application is run.

Stagers can also chroot to the /stagerfs directory to run commands within the package's equivalent filesystem.

For any package dependencies that are placed in the /stagerfs directory that define environment variables such as PATH, the paths given are not altered, so they do not reference /stagerfs.

RSpec stager

RSpec is a Ruby testing tool. This bash script implements an Rspec stager that does the following:

1. curl fetches content from $STAGER_URL/data, writes the content to $TMPDIR/raw.tar.gz, and discards all status and error messages by sending them to /dev/null. If the fetch fails, curl exits, aborting the script.

2. It changes to the /app directory and extracts content from the retrieved tarball.

3. If there is no spec/ dir, the stager makes a POST to $STAGER_URL/failed, marking the stager as failed.

4. If spec/ exists, the stager runs bundle install to install the application's dependencies and run rspec to test the application.

5. If the tests pass, the stager indicates completion by making a POST to $STAGER_URL/done. If the tests fail, the stager indicates this by making a POST to $STAGER_URL/failed.

#!/bin/bash

set -e

export TMPDIR=/tmp

# download the package to stage
echo "Downloading to /app"
curl $STAGER_URL/data > $TMPDIR/raw.tar.gz 2>/dev/null
cd /app
tar xzf $TMPDIR/raw.tar.gz --strip-components=1 app

# make sure we have rspec tests
if [ ! -d spec ]; then
  echo "No spec folder found. Not a valid project."
  curl -X POST $STAGER_URL/failed
  exit 1
fi

# run bundler to install test dependencies
echo "-----> Running bundle install for test dependencies"
bundle install --without development --path vendor/bundle \
 --binstubs vendor/bundle/bin --deployment

# run tests
echo "-----> Running tests"
set +e
bundle exec rspec
rspec_exit=$?
set -e

# check the exit code
if [[ $rspec_exit != 0 ]]; then
  echo "-----> Rspec failed!"
  curl -X POST $STAGER_URL/failed >/dev/null 2>&1
  exit $rspec_exit
fi

# success, mark stager done
echo "-----> Rspec tests passed."
curl -X POST $STAGER_URL/done >/dev/null 2>&1

To deploy the stager to Apcera, use the following command:

$ apc stager create ruby-rspec -p rspec-stager --start-command=./rspec-stager --additive