Welcome to Habitus
Habitus is a standalone build flow tool for Docker. It is a command line tool that builds Docker images based on their Dockerfile
and a build.yml
.
Install
Just run the install script on macOS or Linux!
curl -sSL https://raw.githubusercontent.com/cloud66/habitus/master/habitus_install.sh | bash
Or download the latest version from our release page and choice the right platform. Habitus can run on Linux, Windows and macOS. Copy the Habitus application into /usr/local/bin/habitus
and check if it has the executable flags, if not run chmod a+x /usr/local/bin/habitus
Build files
Habitus uses a yml file as a descriptor for builds. Here is an example:
build:
version: 2016-03-14 # version of the build schema.
steps:
builder:
name: builder
dockerfile: Dockerfile.builder
secrets:
id_rsa:
type: file
value: _env(HOME)/.ssh/my_private_key
artifacts:
- /go/src/github.com/cloud66/iron-mountain/iron-mountain
- /go/src/github.com/cloud66/iron-mountain/config.json
- /go/src/github.com/cloud66/iron-mountain/localhost.crt
- /go/src/github.com/cloud66/iron-mountain/localhost.key
cleanup:
commands:
- rm -rf /root/.ssh/
deployment:
name: ironmountain
dockerfile: Dockerfile.deployment
depends_on:
- builder
uploader:
name: uploader
context: ./uploader
dockerfile: Dockerfile
depends_on:
- ironmountain
command: s3cmd --access_key=_env(ACCESS_KEY) --secret_key=_env(SECRET_KEY) put /app/iron-mountain s3://uploads.aws.com
Build files can be made up of multiple steps. Each step is independent of the other ones and downstream steps can use upstream ones as source (in FROM
command).
In the example above, there are three steps: builder
, deployment
and uploader
. All steps work out of the same working directory. dockerfile
states which Dockerfile is used to build this step.
Here is a list of all step elements:
name:
Name of the generated imagedockerfile:
Dockerfile used to build the stepcontext:
(not mandatory) Context directory where the Dockerfile is stored (cf. Docker "build" Reference Documentation )artifacts:
List of all files to be copied out of the image once it's built. See belowsecrets:
List of all secrets available to the build process. See belowcleanup:
List of all cleanup steps to run on the image once the step is finished. See belowdepends_on:
Lists all the steps this step depends on (and should be built prior to this step's build)command:
A command that will run in the running container after it's built
Artifacts
Artifacts are files to be copied outside of a build image in a step. This can be used when a step is build of a compiled language like Go or Java where the image requires build dependencies. The next step can then use the build step's artifacts in a runtime dependency only image.
Each artefact has two parts: source and destination. Source is the path from within the image and destination where the file will be copied to on the "build server". If destination is missing, the current folder will be used. Full path and file permissions of the source will be preserved during copy. So a file that comes from /app/build/result/abc
of the image will go to ./app/build/result/abc
of the build server if no destination is set.
Here is an example:
- /go/src/service/go-service
or
- /go/src/service/go-service:/tmp/go-service
Artifacts are copied from the container and can be used with ADD
or COPY
commands in downstream steps. Habitus copies artefact file permissions as well.
Here is an example that uses an artefact generated in step builder
FROM ubuntu
ADD ./iron-mountain /app/iron-mountain
Secrets
Managing secrets during a Docker build is tricky. You have might need to clone a private git repository as part of your build process and will need to have your private SSH key in the image before that. But that's not a good idea. Here is why:
First of all, you will need to copy your private SSH key to your build context, moving it out of your home directory and leaving it exposed to accidental commits to your Git repo.
Secondly you will end up with the key in the image (unless you use the cleanup
step with Habitus, see below). Which will be in the image forever.
Third issue is that you will end up sharing SSH keys even if you decided to take the risk and add SSH keys to the image but squash them later.
Habits can help you in this case by using secrets
Habitus has an internal web server. This webserver only serves to requests comming from inside of the building containers. Building containers can ask for "secrets" during build with a simple wget
or curl
. The secret is delivered to the build process which can be used and removed in the same layer leaving no trace in the image.
Here is an example:
# Optional steps to take care of git and Github protocol and server fingerprint issues
RUN git config --global url."git@github.com:".insteadOf "https://github.com/"
RUN ssh-keyscan -H github.com >> ~/.ssh/known_hosts
# using Secrets
ARG host
RUN wget -O ~/.ssh/id_rsa http://$host:8080/v1/secrets/file/id_rsa && chmod 0600 ~/.ssh/id_rsa && ssh -T git@github.com && rm ~/.ssh/id_rsa
This is a snippet from a sample Dockerfile that uses the secrets
feature of Habitus.
First, we make sure all git requests to Github go through the git protocol rather than HTTP (this is optional)
Second, we are going to add Github's server fingerprint to our known_hosts
file to avoid the "yes/no" prompt
Now we can use Habitus and pull our private SSH key as a secret. It will be stored in ~/.ssh/id_rsa
and the next commands will use it (ssh -T git@github.com
is a test that it worked) and the last step (very important) is removing it from the image with no traces and no need to use cleanup
.
You notice that here we are using the Build Arguments in the Dockerfile with ARG
. This allows us to pass in the Habitus IP address into the container without saving it into the image. You can pass in Build Arguments into Habitus with --build
parameter:
$ habitus --build host=192.168.99.1 --secrets=true --binding=192.168.99.1
Secret configuration
IMPORTANT: Secrets are supported in Habitus 0.4 onward and only with build.yml
schema versions on or more recent than 2016-03-14
. Make sure your are using the --secrets=true option to enable the secret service.
You can define as many secrets for your build. Each secret has 3 attributes:
- Name: Name is what the build will refer to this secret as. In this example the name is
id_rsa
- Type: Currently we support
file
andenv
astype
. Which means Habitus supports secrets stored in a file or expose an environment variable from the host. - Value: Value depends on the
type
. Forfile
,value
is the path to the file holding the secret on the build host (ie. your laptop or build server). Forenv
,value
is the name of the environment variable taken from your build host (ie. your laptop or build server)
secrets:
id_rsa:
type: file
value: _env(HOME)/.ssh/my_private_key
Check here for an example using file secrets.
secrets:
exposed_secret_env:
type: env
value: SECRET_KEY
Check here for an example using environment variable secrets. NOTE: The value of the exposed_secret_env in this example is mapped to the host machine as HABITUS_SECRET_KEY instead of SECRET_KEY due to security reasons.
Habitus IP address
Finding the correct IP address to server Habitus API (and secrets) on can be tricky. You don't want to bind it to 0.0.0.0
since it will make Habitus and your secrets available to your entire local network (and possibly the internet if you're running a tunnel) during the build time.
On a Linux machine where Docker can run natively you can bind Habitus to 127.0.0.1
. However on a Mac (OSX) Docker runs inside of a VM (VirtualBox in most cases thorugh Boot2Docker). This means you need to find the VM address of your Mac and use that to bind Habitus to. By default, Boot2Docker (and Docker Machine) use 192.168.99.1
which is what Habitus uses by default.
You can find your VM address by the following command:
$ ifconfig -a
What you are looking for is the inet
address of a vboxnet
like the following:
vboxnet3: flags=8943 mtu 1500
ether 0a:00:27:00:00:03
inet 192.168.99.1 netmask 0xffffff00 broadcast 192.168.99.255
On a Mac, if your VM IP address is different from 192.168.99.1
you can configure Habitus using the --binding
parameter:
$ habitus --binding 10.0.99.1 --secrets=true
You can also change the Habitus API port from the default 8080
using the --port
parameter.
Check here for an example using secrets
Cleanup
Cleanup is a step that runs after the build is finished for a step. At the moment, cleanup is limited to commands:
cleanup:
commands:
- apt-get purge -y man perl-modules vim-common vim-tiny libpython3.4-stdlib:amd64 python3.4-minimal xkb-data libx11-data eject python3 locales golang-go
- apt-get clean autoclean
- apt-get autoremove -y
- rm -rf /var/lib/{apt,dpkg,cache,log}/
This runs the commands in the provided order on the image and then as a last step squashes the image to remove anything that's been removed. This is particularly useful when it comes to private information like ssh private keys that need to be on the image during the build (to pull git repos for example) but can't be published as part of the built image.
Image sequencing
Habitus allows dovetailing (sequencing) of images from different steps. This means a step can use the image built by the previous step as the source in its Dockerfile FROM
command. This is done automatically if FROM
command refers to an image name used by a previous step.
Habitus automatically parses the FROM
image name and replaces it with the correct name when it is used in multi-tenanted setup. This enables multiple builds of the same build file to run in parallel with different session UIDs (see below).
Please note if you are using step A's result in step B's FROM
statement, you need to make sure A is listed under depends_on
attribute of B. Otherwise both A and B will be built in parallel.
Build stage targeting
This feature allows you to enhance docker multi-stage builds with Habitus.
Habitus allows reusing the same Dockerfile from different steps while targeting each step to a specific build stage. This means a step can use the image built by the previous step as the source in its Dockerfile FROM
command, similarly to we have discussed in `
encing` above, but with different pros and cons.
One of pros is faster builds due to faster assets transfer between steps, because now assets transfer happens directly between build contexts, rather than routed via the node habitus is running. An another one is that Docker users used to multi-stage builds can easily adapt to Habitus for its advanced features, without rewriting their multi-stage-ready Dockerfiles. For more inforation, see this issue.
To use this feature, specify < href="https://github.com/mumoshu/habitus/blob/42f4ec5997f771863afd5120f36772d1a92f9c9d/examples/multistage/build.yml">a target
per step in your build.yml
.
The specified target
should exist within your Dockerfile
referenced in the step. In other words, there must be a FROM $your_base_image AS $target_ie_build_stage_name
in the Dockerfile
.
Step dependencies
Steps can depend on each other. This can be specified by the depends_on
attribute.
Steps can depend on one or more of the other steps. This will determine the build order for steps. Independent steps are built in parallel and according to the build order defined by dependencies.
Environment variables
Environment variables can be used in the build file with the _env(VAR)
format:
artifacts:
- /go/src/go-service/_env(SERVICE_NAME)
This will be replaced before the build file is fed into the build engine. By default Habitus inherits all environment variables of its parent process. This can be overridden by passing environment variables into Habitus explicitly through the env command parameter:
$ habitus -env SERVICE_NAME=abc -env RAILS_ENV=production
In the example above, you can pass in AWS S3 key and secret like this:
$ habitus -env ACCESS_KEY=$ACCESS_KEY -env SECRET_KEY=$SECRET_KEY
Running commands
Habitus allows you to run an arbitary command inside of a built container. This can be useful in many cases like uploading the build artifacts to webserver, resetting your exception handling service after each build or starting your release process.
command
attribute is optional. If present, the image is built and a container is started based on it to run the command.
command
runs after the build, cleanup and copying of the artifacts are done.
An example to upload a build artefact to S3 can be like this
FROM cloud66/uploader
ADD ./iron-mountain /app/iron-mountain
cloud66/uploader
is a simple Docker image that has S3CMD installed on it.
The Dockerfile here is a simple one that starts from cloud66/uploader
and adds one of the build artifacts to the image so it can be uploaded to S3.
Command line parameters
Habitus accepts the following command line parameters:
f
: Path to the build file. If not specified, it will default tobuild.yml
in the work directory.d
: Path to work directory. This is the path where Dockerfiles should exist for each step and the build happens. Defaults to the current directory.binding
: IP address to bind Habitus API to. Habitus API provides services like secrets to the building containers. Default is192.168.99.1
build
: Dockerfile Build Arguments passed into the build processcerts
: Path of the key and cert files used to connect to the Docker daemon. Defaults to$DOCKER_CERT_PATH
env
: Environment variables used in the build process. If not specified Habitus inherits all environment variables of the parent process.force-rm
: Forcefully remove intermediate images.force-rmi
: Forces removal of unwanted images after the buildhost
: Address for Docker daemon to run the build. Defaults to the value of$DOCKER_HOST
. For Docker on OSX, useunix:///var/run/docker.sock
keep-all
: Overrides the keep flag for all steps so you can inspect and debug the created images of each step without changing the build file.level
: Logging level. Acceptable values:debug
,info
,notice
,warning
,error
and critical. Defaults todebug
no-cache
: Don't use docker build caching.no-cleanup
: Don't run cleanup commands. This can be used for debugging and removes the need to run as sudonoprune-rmi
: Doesn't prune unwanted images after the buildport
: Port to server Habitus API on. Default is8080
rm
: Remove intermediate built images.sec-providers
: Comma separated list of secret provider types available during the build. Default isfile,env
secrets
: Enable or disable support for secrets during build. Default isfalse
use-stat=
: Enable or disable coping artifacts permissions usingstat
(need to be installed on the artifact image). Default istrue
suppress
: Suppress docker build output.tls
: Use TLS to connect to the Docker daemon (default is true)uid
: A unique ID used for a build session to allow multi-tenancy of Habitus
Development Environment for Habitus
Habitus requires running in privileged more (sudo) so it can run the squash method (keeping file permissions across images). It also requires the following environment variables: DOCKER_HOST
and DOCKER_CERT_PATH
. These are usually available when Docker is running on a machine, but might not be available in sudo mode. To fix this, you can pass them into the app with commandline params:
$ sudo habitus --host $DOCKER_HOST --certs $DOCKER_CERT_PATH
Dependencies
You would also need gnu tar to be available on the machine:
OSX
Multi-tenancy for Habitus
Habitus supports multi-tenancy of builds by using a uid
parameter.
All builds and images will be tagged with the uid
for this unless a step name explicitly has a tag. In that case the tag is concatenated with the -uid
.