A Brief History of Open Privacy Automated Build Systems

I’ve had a few experiences with build systems over my career. After joining a small startup, I was horrified to find that their deployment ‘process’ for 6 micro-services, entailed devs logging into each of the two servers they initially had for production, and manually checking out the latest code, rebuilding it, and restarting it. I spent some of my early time there building an automated build system with Jenkins.

Later, I worked at a large tech company which had a pretty decent and robust system that had been developed in-house. Sadly, even in that company, operational excellence levels varied quite wildly from team to team. Some teams had full Continuous Integration (CI), which is a fully automated build pipeline deploying code that passes the tests to production automatically, which speaks highly of the team trusting their tests to catch any and all issues. Meanwhile, many other teams not only had manual approval steps in their build pipelines but many of those approvals were attached to very long documents of required manual testing steps.

It generally meant code pilled up waiting for a “release window” on some schedule meant to minimize the amount of time developers spent doing manual testing. It also ocasionally meant bypassing testing when pressure was applied during a rush.

A Quality Philosophy Interlude

Build automation could be seen as simply having a system that reliably generates build artifacts from your code such as apps you can deploy to a server or distribute to users via outlets like Google Play Store. However one of the big values from automation for us and with respect to Operational Excellence is testing. This is especially relevant as we develop many libraries that never compiled into standalone applications but are meant for use downstream. For these libraries we still have automation, explicitly for testing.

Before we can automate tests, we need tests. To start, we use tools to enforce code quality, or at least formatting standards. Most languages have tools to help with this, and thus most of our projects end up with a quality.sh script.

We have found integration testing to be one of the more useful automation primitives. This tests that all your modules work together, and can catch additional unintended bugs being introduced in new code that unit tests might still miss. Specifically, our Cwtch integration test has caught numerous bugs that would have been tricky to find through unit testing alone. For example, the test uses Go’s runtime.NumGoroutine() to sample the count of Go routines at a few times through the run which allows us to track if we’ve introduced a thread leak in any new code.

Our First Build System

When I started evaluating build automation options for Open Privacy I took an initial first step with Jenkins, because of my previous experiment with it. However, we were not happy with the complexity of the setup or the complexity of our desired integrations.

We decided to give Drone CI a try after finding out that it had reasonably good integration with our code repository system and its new (at the time) technology that built each pipeline from a series of Docker containers with instructions written in a Docker Compose-like YAML.

Pretty quickly we were able to get a pipeline up for our core library Cwtch, exercising the quality, unit and integration testing. As we soon started experimenting with user interfaces we tested out pipelines for them as well. There was an early prototype of what a native Android UI would be like, using Go Mobile to wire in our Go library. This eventually gave way to our QT/QML UI.

We also wrote and published a Docker container for use with Drone for commenting on Gitea pull requests with build results as at the time the integration of Drone and Gitea was not that well developed. It’s on Docker Hub at openpriv/drone-gogs (Called Gogs because we wrote it before Gitea’s forked from Gogs).

Alas, everything grows and changes with time. As such, while we were heads down working on the Cwtch UI beta in QT/QML, Drone “grew” a version with the release of their 1.x series.

We had started using Drone during their early releases and were still on 0.8 which didn’t have a clear upgrade to 1.0. We deferred that work in favour of other priorities. However, with new UI options on the horizon it finally made a lot more sense to also upgrade our whole build system.

Moving to Drone CI 1

The documents for the 1.x line of Drone CI are a bit sparser than previous versions, but have everything you need. Drone’s capabilities have expanded and it can now run builds on native systems and plugs into a few other runner systems besides Docker. In part because all our existing infrastructure was on Docker and in part because it’s simply a good fit for the task, we went ahead and stuck to the Docker runner.

Something we did when setting up Drone 0.8 that was recommended then, but seems to have been dropped from the docs now, is to set it up as a Docker Compose style service. We also use Nginx to proxy for it as we have mutliple servers on the same host. It took a little work updating our Docker Compose script from the 0.8 syntax to the version 1 syntax as the invocation of Drone has changed. They now use ports 80 and 443 directly for web and for RPC, whereas 0.8 used to serve the website on port 8000 and RPC on port 9000. We’ve included the relevant scripts for anyone interested in this style of setup in the appendix. If anyone else is upgrading, just be very careful with the new arguments, some replace old ones with similar but different names.

Drone’s and Gitea’s authentication for user accounts was moved to Gitea’s more mature OAuth2. Close inspection of the terse steps are recommended. It is in part managed in Gitea in the site admin under “applications” needing an OAuth2 app to be setup using the Drone CI login URL.

We got the new system up and running and it was simple for us to re-enable the existing pipelines we’d been using with Drone 0.8 as they worked seamlessly (for an example, Cwtch’s drone.yml.

This week we’ve been working on setting up the new pipelines for our new Flutter/Cwtch UI helper library, which takes a lot of the QT/QML UI’s helper functions for UIs and then also restructures some of Cwtch’s app’s interface into one easier for the Flutter app and framework style to make use of. As we will probably blog more about later, the way we incorporate the Cwtch Go library into our UI has changed, and we’ve moved to both using Go’s ability to make c-shared style libraries for desktop bundling and to Go Mobile to build a Java .aar for Android.

Getting Drone to use Go to compile the required shared library was easy. However, having briefly experimented with Go Mobile builds 3 years ago has turned out to be fortuitous. A quick Google search confirmed Open Privacy still appears to be the sole supplier of Docker containers bundling Go, the Android SDK and NDK, and Go Mobile all together for use in builds (compatible with Drone CI) which we made during that early phase. It is available in the openpriv/android-go-mobile Docker container. It appears to have received some pulls in the intervening time, so it has a small user base, which is rewarding to see. Open Privacy has a commitment to open infrastructure work and this docker container is a another good example of that.

We are currently in the middle of updating the android-go-mobile container from Go 1.10 to 1.16 and a newer Android SDK and Target API version to use in our new build pipeline. We’ll be taking care to make sure the previous version remains accessible from Docker Hub so people won’t be surprised or forced into upgrading to the new version without time to test and migrate. I’ll tweet more about this when it is done.

Inefficiently Lazy

There is an old joke that programmers are all lazy and just write code to do everything for them. Likewise, it’s the old response joke that programmers will then often end up spending a longer time trying to automate simple tasks than they would have by just doing them.

Regardless, our build automation pipeline has certainly caught many bugs before they happened, helping us deliver quality and reliable code - and that quality is worth investing in.

Appendix

/etc/drone/docker-compose.yml

version: '2'

services:
  drone-server:
    image: drone/drone:1
    ports:
      - 8000:80
      - 8443:433
    volumes:
      - /var/lib/drone:/var/lib/drone/
    restart: always
    environment:
      - DRONE_OPEN=false
      - DRONE_ORGS=xxxxxxxxxx,xxxxxxxxxxx,xxxxxxxxxxxxxxxxx....
      - DRONE_ADMIN=xxxxxxxx,xxxxxxxxxx,xxxxxxxxxxxxxxxxxxxx...
      - DRONE_SERVER_HOST=build.openprivacy.ca
      - DRONE_SERVER_PROTO=https
      - DRONE_GITEA=true
      - DRONE_GITEA_CLIENT_ID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      - DRONE_GITEA_SERVER=https://git.openprivacy.ca
      - DRONE_GITEA_WITH_AUTH=true
      - DRONE_AGENTS_ENABLED=true
      - DRONE_LOGS_TRACE=true
      - DRONE_LOGS_PRETTY=true
      - DRONE_LOGS_COLOR=true
    env_file: /etc/drone/secret.env

  drone-agent-1:
    image: drone/drone-runner-docker:1
    mem_limit: 3G
    command: agent
    restart: always
    depends_on:
      - drone-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_RPC_HOST=build.openprivacy.ca
      - DRONE_RPC_PROTO=https
      - DRONE_RUNNER_CAPACITY=1
      - DRONE_LOGS_TRACE=true
      - DRONE_LOGS_PRETTY=true
      - DRONE_LOGS_COLOR=true
      - DOCKER_API_VERSION=1.38
    env_file: /etc/drone/secret.env

  drone-agent-2:
    image: drone/drone-runner-docker:1
    mem_limit: 3G
    command: agent
    restart: always
    depends_on:
      - drone-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_RPC_HOST=build.openprivacy.ca
      - DRONE_RPC_PROTO=https
      - DRONE_RUNNER_CAPACITY=1
      - DRONE_LOGS_TRACE=true
      - DRONE_LOGS_PRETTY=true
      - DRONE_LOGS_COLOR=true
      - DOCKER_API_VERSION=1.38
    env_file: /etc/drone/secret.env

/etc/drone/secrets.env

DRONE_GITEA_CLIENT_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
DRONE_RPC_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

/etc/systemd/system/drone.service

[Unit]
Description=Drone server
After=docker.service nginx.service

[Service]
Restart=always
ExecStart=/usr/bin/docker-compose -f /etc/drone/docker-compose.yml up
ExecStop=/usr/bin/docker-compose -f /etc/drone/docker-compose.yml stop

[Install]
WantedBy=multi-user.target

/etc/nginx/sites-enabled/build.openprivacy.ca

upstream drone {
  server 127.0.0.1:8000;
}

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
  server_name build.openprivacy.ca;

  location '/.well-known/acme-challenge' {
    default_type "text/plain";
    root      /var/www/letsencrypt;
    allow all;
  }

  location /files {
        alias /home/buildfiles/buildfiles/;
        autoindex on;
  }


    listen 443 ssl; 
    ssl_certificate /etc/letsencrypt/live/build.openprivacy.ca/fullchain.pem; 
    ssl_certificate_key /etc/letsencrypt/live/build.openprivacy.ca/privkey.pem; 
    include /etc/letsencrypt/options-ssl-nginx.conf; 
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; 


  location / {
    proxy_pass              http://drone;
    include proxy_params;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_buffering off;
        chunked_transfer_encoding off;
        proxy_read_timeout 86400;
  }

}

server {
    if ($host = build.openprivacy.ca) {
        return 301 https://$host$request_uri;
    } 


  listen 80;
  server_name build.openprivacy.ca;
    return 404; 


}