Concourse build system for iOS and Android (part 1)

Jakub Łyskawka

Ever felt like waiting for your iOS app to be archived or release APK to be built is just a waste of time? Wouldn't it be great if you could just push your commit to the repo and don’t worry about anything else? Imagine what could you do with all that saved time…

Well, for example you could spend it writing long, complex and repetitive configuration files in an unreadable, indentation sensitive language. If it sounds tempting, read on!

Continuous integration and mobile apps

Using some kind of continuous integration solution in any development workflow undoubtedly has many benefits:

  • It allows you to run tests automatically
  • It can easily produce various build versions for different environments
  • It can automate the deployment process to reduce the possibility of human error
  • It makes it easier to keep track of version numbers and build artifacts
  • It saves developer’s time (when you finally manage to make it work reliably)

There are many cloud based CI solutions available nowadays. These are great options to automate workflows for web applications and backend systems, however mobile development brings its own challenges, which are often not so easily solvable with cloud services.

IDEs for mobile platforms were definitely not designed with automation and command line usage in mind. This is especially true for Xcode - it relies heavily on graphical configuration editors and runs only on macOS, which happens not to be the number one OS choice for server environments. Android Studio and Xcode are quite convenient for a single developer, but when there are many people involved in a project or a few people working on many projects, then their shortcomings become more and more visible.

If you want to perform some UI tests on your application, then you’ll need a specialized service that allows you to use device simulators to run tests. Some of them even offer farms of physical devices, but it costs some money. Alternatively, with an in-house CI solution you can set up your own worker to perform tests or even hook up your own devices and run tests on them.

Another factor that can lead to excessive head scratching is the complexity of code signing and provisioning procedures. This becomes an issue if you try to automate the deployment process. The release version of the app needs to be signed, the signing assets should be kept in a secure place, probably separate from the source code. If you’re developing for iOS you’re going to have even more fun with provisioning profiles and entitlements. Oh, and you need to securely communicate with a store to upload your final build, which requires another set of credentials.

Concourse CI

For the reasons mentioned above and just to experiment a bit, we decided to set up an in-house CI/CD environment capable of building and deploying Android, iOS and Flutter apps. When it comes to open source CI servers the most obvious choice is Jenkins. However, having some previous experience with the old mustachioed waiter we were looking for other options. That’s how we found Concourse.

Concourse is a quite new CI system with many interesting features:

  • It is distributed - it consists of many independent nodes
  • It offers a nice web interface to visualize build status and a convenient command line interface for management
  • It is docker based - by default all the builds are executed in docker containers
  • It uses YAML configuration files for describing pipelines which can be kept under version control
  • It has a powerful concept of resources which can be used to easily implement custom integrations and new features

Let's take a closer look at these features to understand what they mean in practice.

Basic setup

Concourse is written in Go and is available as a standalone binary for Linux, Mac or Windows. A node is simply a machine running a Concourse process. While the whole system can consist of many independent nodes, there needs to be at least one web node. The web node is responsible for providing the web interface for the system and scheduling tasks for worker nodes. It also requires a PostgreSQL database for storing configuration and build metadata, however all other build artifacts will be kept elsewhere with the help of external resources.

In order to do something useful, you will also need at least one worker node. Worker node is another machine running Concourse and a docker daemon. When a web node discovers a worker node, it can ask it to execute some tasks. Since everything happens inside docker containers, there is no need to manually set up the build environment. The worker node will automatically download necessary docker images, fetch code from a git repo, execute build tasks inside a container and upload any artifacts to proper resources. Unfortunately this is not entirely true for iOS apps, but we’ll take care of this inconvenience later.

To configure nodes you will use a fly command line tool. It allows you to check the status of worker nodes, update configurations and perform any other management tasks. You can find more details about the basic Concourse setup in the official docs.

Pipelines, resources, jobs, tasks

Concourse uses some abstract terms to describe what needs to be automated and how to do it. It’s a good idea to become familiar with these terms before trying to configure your projects. Below there is a quick overview of the essential concepts.

Pipelines

A pipeline is a description of a single continuous integration workflow. Most likely you will want to have a pipeline for every project that you develop and maintain. It will be visualized on a web UI, which allows you to see the status of the project and view a build history. The pipeline contains a list of resources and jobs, and a declarative description of how to use them together to perform desired tasks. On the code level the pipeline is just a YAML file with some required sections and an established format. You can read more about pipelines here.

Resources

Resources are one the core concepts of Concourse. They represent various external artifacts, which can be used as inputs or outputs for executed tasks. Resources are also versioned and can be automatically observed for changes. When Concourse discovers that a new version of a resource is available, it can trigger a job execution using the new resource version as an input.

Here are some examples of resources:

  • A git repository - with commits treated as subsequent versions
  • S3 or similar storage bucket - a place where a finished build product can be saved
  • Auto-incrementing version number
  • Slack channel - can be used as an output to inform about build status

I think you’ve got the idea. What’s nice about the resources is that they are just docker images underneath, so it’s really easy to create your own resource if the already available ones don’t fit your particular needs. Sharing your new resource is a great way to contribute to the Concourse community.

Jobs

Jobs define the actual actions to be executed inside a pipeline. Each job has a name, and will be visible as a node in a graph on the web UI. The most important part of the job is a list of steps to be executed. These can be for example:

  • Fetching a new version of the resource
  • Executing a task with the input provided by the resource
  • Pushing a new version of a resource after performing some task on it

The resources that your job operates on will be visible on the web UI as graph edges entering and exiting the block representing the job.

Tasks

The task is the smallest unit of work that you can configure, usually it will be some actual command to be executed. A basic task description consists of:

  • A platform on which it can be executed (you will need a worker node with a matching platform)
  • A docker image which will be used to execute the task
  • A command to be executed inside the container
  • List of inputs and outputs - matching resources available to the particular job

When you click on a job node in the web UI, you can see the list of tasks it contains. You can click on any task to see details about it, for example the console log.

Summary

I hope this introduction gives you a basic overview of how the Concourse CI works. In the next part we will see how to apply this knowledge to create working pipelines for building Android and iOS apps and how to solve some problems you may encounter. Stay tuned!