Here's a conversion I keep hearing recently:

A: Let's put all our projects in one repo.
B: Why?
A: Because Google and Facebook do monorepo.

Whenever I hear this, I'm very tempted to show this picture.

Finding the origin of this picture is left as an exercise for the readers. On a more serious note, I want to write down my thoughts on multirepo vs monorepo.

What is multirepo?

One project one repository. Each project is an independent working unit. It can be a mobile app, frontend app, backend service or standalone CLI app.

• Each project has full autonomy to manage its evolution and deployment. There should be little to no coupling between projects. If projects depend on each other, the coupling between projects is API contracts, nothing else.
• Each project manages dependencies on its own. Common library is in a repo of itself. Projects that depend on it can use any version of that library that they deem fit. It can be argued that sharing code is also introducing coupling. And it may result in long tail of maintenance of old libraries. Anyway, Managing dependencies is hard.
• Engineering teams are decoupled and can work on different projects in parallel without stepping on each other's toes.
• Deployment Pipeline can be easily setup for each project.
• Access control can be applied at project level.

This repo structure is how most open source projects are run. And it's also probably what most developers are familiar with. Besides, it plays nicely with microservices architecture.

What is monorepo?

One monolithic repo that contains everything. Literally, everything.

• All projects (regardless they are related or not) and their dependent libraries, including 3rd party code that are not written by you nor your colleagues, live in one single repo.
• There is one and only one verison of each dependency in the entire repo, which is the latest (HEAD in git terminology). Whenever a dependency needs to be updated, the update should be done for all projects depend on it and make sure that all projects still work. So the repo should always be in a consistent state. At any commit, all projects should work.
• Cross-project changes is easier. Large scale refactoring is easier and can be done in one single atomic commit.
• Extensive code sharing.
• Everyone can see all code.

How to choose?

If you are in a two-man startup, close this page now and keep working on your monolith. The choice between multirepo and monorepo is irrelevant to you. This question is only relevant when your company is operating at scale, i.e. >100 developers.

Given the distinct characteristics of multirepo and monorepo, how to choose one over another? I think there are two main factors to consider, tooling and culture.

Tooling

In monorepo, running build is not as trivial as multirepo. You probably don't want to run tests and builds for all projects since that's just unnecessarily wasting time and computing resources. So the first thing to figure out is, given a change with one or more commits, which project(s) should build and what tests should run. And in order to figure this out, it's necessary to have acyclic directed graph (DAG) of dependencies for all projects. When a change is submitted, it's checked against the DAG of dependencies to see which projects are affected. All affected projects are possible to break, so tests are run only for these affected projects and their transitive dependents. Good news is that Google has open sourced their build tool bazel and Facebook has something similar called buck. While in multirepo, this problem doesn't exist because there is no need to figure out which project to build. Whenever a change happens to a project, that project's deployment pipeline is triggered.

Source code version control is another tooling challenge imposed by monorepo. It's well known that git is bad at scaling. So is mercurial. Quoting Linus Torvalds

Git fundamnetally never really looks at less than the whole repo. Even if you limit things a bit (ie check out just a portion, or have the history go back just a bit), git ends up still always caring about the whole thing, and carrying the knowledge around.
So git scales really badly if you force it to look at everything as one huge repository. I don't think that part is really fixable, although we can probably improve on it.

Although sparse checkout and shallow clone may alleviate the scaling problem, it's not a sustainable solution to large organizations. Some anecdote suggests that the practical limit of git is 15GB of .git directory. This is probably why Microsoft invented GVFS, Facebook chose to patch Mercurial and Google builds Piper. The point is, if your organization decides to go monorepo, think carefully about what version control to use.

Large scale refactoring in monorepo doesn't come for free. It requires dedicated tooling support.

Setting up deployment pipeline is complicated in monorepo. In multirepo, it's straightforward to have one project one pipeline. But in monorepo, one possible way is to have the first stage to figure out relevant projects and then trigger child pipelines for each relevant project. And each child pipeline may trigger other pipelines according to the DAG dependency graph. From what I see, the only off-the-shelf Continuous Delivery (CD) tool in the market that supports pipeline fan-in and fan-out is GoCD. Other CD solutions in the market have very simple pipeline modeling. They are designed for multirepo, not monorepo. For example, there isn't an elegant solution for monorepo in GitLab after one year, neither is Travis CI.

In short, for multirepo to work, open source and commercial tools in the market are most likely sufficient. But for monorepo, depending on the scale, it may require high tooling investment.

Culture

Multirepo and monorepo not only have different tooling requirments, but also varying engineering culture and philosophy. Multirepo values decoupling and engineering velocity, while monorepo favours standardization and consistency. It's all trade-offs. Whichever approach a company takes is a reflection of the company's culture. Netflix favours Freedom & Responsbility so it prefers mutlirepo. And Google values consistency and code quality so it prefers monorepo. What's important here is to pick one approach that fits your organization's engineering culture, rather than fitting your organization's engineering culture to a certain repo structure.

Conclusion

Choosing multirepo or monorepo is not trivial. There is no single absolute right or wrong answer. Companies like Amazon and Netflix are living evidence that multirepo at large scale works. On the other hand, companies like Google and Facebook are living evidence that monorepo at large scale also works. Each approach has its own set of principles and practices to follow. Each approach also has its own challenges. Deciding between the two boils down to tooling and culture. Whichever approach an organization takes should be backed up by a list of solid reasons why one is preferred over the other in that organization, not Because Google and Facebook do monorepo. That's cargo cult engineering. And You Are Not Google.

Recently, I found out an interesting problem in Go. The problem can be reduced to a simple client request to a HTTP server.

Suppose we have a HTTP server, which serves only one rooted path /foo/.

package main

import (
	"io"
	"log"
	"net/http"
	"net/http/httputil"
)

func handleFoo(w http.ResponseWriter, req *http.Request) {
	// request details
	dump, _ := httputil.DumpRequest(req, true)
	log.Println(string(dump))

	if auth := req.Header.Get("Authorization"); auth != "Bearer GoodToken" {
		http.Error(w, "401 Unauthorized", http.StatusUnauthorized)
		return
	}

	io.WriteString(w, "Hello World!")
}

func main() {
	http.HandleFunc("/foo/", handleFoo)
	log.Fatal(http.ListenAndServe(":12345", nil))
}

handleFoo simplily verifies that correct token is sent in the Authorization header. Otherwise, it returns 401 Unauthorized.

The client sends request with correct token to the server.

package main

import (
	"io"
	"log"
	"net/http"
	"os"
)

func main() {
	req, _ := http.NewRequest(http.MethodGet, "http://localhost:12345/foo", nil)

	req.Header.Set("Authorization", "Bearer GoodToken")

	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Fatal(err)
	}

	defer resp.Body.Close()
	if _, err := io.Copy(os.Stdout, resp.Body); err != nil {
		log.Fatal(err)
	}
}

What do you think the response is? 401 Unauthorized or Hello World!?

The answer is it depends. It depends on the version of Go that the client code is running. If the client code is running on Go <1.8.0, the response is 401 Unauthorized. Otherwise, it's Hello World!.

But why?

It's not trivial to see at first glance. Let me list down what happens step by step.

1. Client sends

        GET /foo HTTP/1.1
        Host: localhost:12345
        Authorization: Bearer GoodToken
        ...
   

2. Server receives the request. ServeMux determines that the requested path /foo match the registered rooted path /foo/. ServeMux decides to send redirect (doc).

3. Server responds with header

        HTTP/1.1 301 Moved Permanently
        Location: /foo/
        ...
   

4. Client receives response and follows redirect by sending another request to server.

5. Server receives the 2nd request and let handleFoo handle it.

Both 1.8.0 and versions <1.8.0 follow the same steps when processing the reqeust. The difference lies in Step #4.

Prior to 1.8.0, following redirect in Go will NOT copy the original headers even if it's for the same domain. This wasn't changed util this commit. What happened in Go <1.8.0 is that the Authorization header wasn't copied unpon redirect. Therefore, handleFoo returns 401 Unauthorized.

Initially, I was quite surprised by the behaviour. After reading through this issue and HTTP spec, I realized that http/client.go didn't actually do anything wrong because the HTTP spec didn't specify following redirect should copy original headers. It's just that the well known HTTP clients, e.g. curl, httpie, rest-client, etc., have established the convention.

Be wary of http/client.go.

While working on a continuous delivery pipeline to automate deployment to Google Container Engine (GKE), I found that getting kubectl to work is very complex and convoluted, especially when it needs to be noninteractive. So I want to find out the easiest way to get kubectl working noninteractively.

Assuming that gcloud and kubectl are already installed but not necessarily setup, ONLY two commands are needed to get kubectl working noninteractively (verified with Google Cloud SDK 141.0.0 and kubectl 1.5.2)

gcloud auth activate-service-account --key-file ${PATH_TO_KEY}

gcloud container clusters get-credentials ${CLUSTER} --zone ${ZONE} --project ${PROJECT}

The first command gcloud auth activate-service-account is to authorize access to Google Cloud Platform using a service account. PATH_TO_KEY is the path to the private key of the service account. The idea is very similar to IAM in AWS. One service account is roughly equivalent to an IAM group in AWS. And the private key of the service account is like Access key ID and Secret access key. You can create a service account and generate its private key here. If you only need to deploy to GKE, Container Engine Developer is enough for the role.

The second command gcloud container clusters get-credentials fetches cluster credentials and saves it in ~/.kube/config. The environment variables in the command are self-explanatory.

You can now use kubectl to deploy to GKE. Probably this?

kubectl set image deployment/${DEPLOYMENT} ${CONTAINER_NAME}=${IMAGE}:${IMAGE_VERSION}

In summary, this solution is simple, noninteractive (great for CI/CD) and secure (fine-grained permissions defined by service account). If you have a better solution, please let me know.

Recently I worked on a project where executing remote commands is very slow to start. This is expected because an SSH connection has to be established first.

$ time ssh server exit
ssh server exit  0.04s user 0.01s system 0% cpu 7.901 total

It took nearly 8 seconds to ssh into server.

That's slow!

After a bit of googling, there is an elegant way to speed this up. Simply put this at the bottom of your ~/.ssh/config.

Host *
  Compression yes
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h:%p
  ControlPersist 4h
  ServerAliveInterval 60

What this does is to try to share the master connection via a socket with multiple ssh sessions. It falls back to creating a new one if one does not already exist. The socket file is defined by ControlPath. ControlPersist 4h specifies that keep the master connection in background for four hours after the client connection is closed. ServerAliveInterval 60 means that the client will wait for 60s to send a message to the server (if no data has been received during this period) to keep the connection alive.

Create sockets directory

$ mkdir ~/.ssh/sockets

Running time ssh server exit again won't see any difference. But look into ~/.ssh/sockets, a socket file is created.

srw-------  1 user group    0 Dec  4 22:12 user@***.***.***.***:22

Try again

$ time ssh server exit
ssh server exit  0.01s user 0.00s system 40% cpu 0.027 total

Now it takes less than 0.5% of 8 seconds.

What's interesting is that, this config not only speeds up SSH connections, but also git, if you're authenticating with SSH.

While coding for an algorithmic problem, I discovered that Ruby's stack level is much shallower than Java. This caused a recursive DFS solution written in Ruby failed due to stack level too deep (SystemStackError), while the same code written in Java passed. Whether recursion or tail recursion should be used is not the point of this post. This post is to find out what the max stack level is and what limits the stack level.

Ruby

Consider the following code

def recurse(n)
  return 1 if n == 1
  1 + recurse(n-1)
end

def binary_search
  answer, a, b = 0, 0, 1_000_000_000

  while a<=b
    mid = (a+b)/2
    begin
      recurse(mid)
      answer = mid
      a = mid + 1
    rescue SystemStackError
      b = mid - 1
    end
  end

  answer
end

puts "Max Stack Level: #{binary_search}"

What do you think Max Stack Level is?

Running the code reports that the the Max Stack Level is 10080 on my MBP. This is because the stack size of the default Ruby 2.3.1 VM (MRI) is limited to 1MB.

This is very limiting for even a medium-sized dataset. To increase it, one can specify VM stack size for Ruby >= 2.0.0. To check current max stack size, open irb

irb(main):001:0> RubyVM::DEFAULT_PARAMS
{
         :thread_vm_stack_size => 1048576,
    :thread_machine_stack_size => 1048576,
          :fiber_vm_stack_size => 131072,
     :fiber_machine_stack_size => 524288
}

Let's set RUBY_THREAD_VM_STACK_SIZE to 2MB

export RUBY_THREAD_VM_STACK_SIZE=2097152

And running the same code again returns Max Stack Level: 20162, which is roughly 2 * 10080. Setting RUBY_THREAD_VM_STACK_SIZE to 3MB will increase Max Stack Level to 30243. This proves that Max Stack Level is linearly proportional to RUBY_THREAD_VM_STACK_SIZE in Ruby.

Java

public class Solution {
    public static void main(String[] args) {
        System.out.println("Max Stack Level: " + binarySearch());
    }

    private static long binarySearch() {
        long a = 0, b = Long.MAX_VALUE, answer = 0;
        while (a <= b) {
            long mid = (a + b) / 2;
            try {
                recurse(mid);
                answer = mid;
                a = mid + 1;
            } catch (StackOverflowError e) {
                b = mid - 1;
            }
        }
        return answer;
    }

    private static long recurse(long n) {
        if (n == 1) return 1;
        return 1 + recurse(n - 1);
    }
}

Running this code with

java Solution

returns Max Stack Level: 49150. This is because the thead stack size on my MBP defaults to 1MB. The defaults are platform-specific, see here.

JVM has an option ss for setting the thread stack size. Running the code with thread stack size of 2M

java -Xss2M Solution

returns Max Stack Level: 98302, which roughly doubled the stack level. Running it with 4MB gives 196606 and it matches what I thought. Therefore, in Java, Max Stack Level is linearly proportional to ss.

Conclusion

Both Ruby and Java have clearly defined max stack size, which limits the number of levels code can recurse. In Ruby, it's defined by environment variable RUBY_THREAD_VM_STACK_SIZE. While in Java, it's defined by JVM option ss. A clear observation is that when given the same max stack size and the same code, Java performs much better than Ruby. This is not a surprise because Java (on JVM) is a compiled, while Ruby MRI is interpreted.

In my previous posts, I demoed how to orchestrate Docker with Swarm and Kubernetes. They all assume the Docker image chenglong/simple-node is already there and ready to be deployed. But how to develop that image in the first place? How to streamline and automate the process of developing it on local machine, building and testing it on Continuous Integration (CI) server, and finally deploying to production?

This post introduces one possible Docker workflow.

I recommend reading my previous post before this to understand the app architecture and Kubernetes.

Workflow

The flow is summarized as follows:

1. Code on local machine. Using Docker for local development is highly recommended. Find out more from my post Using Docker for Rails Development
2. Push commits to Github
3. Travis is notified and builds new a new Docker image and runs tests against the newly-built image.
4. If all work well, Travis pushes the new image to Docker Hub.
5. Travis deploys the new image to Google Container Engine (GKE). Prerequisite: you need to setup a container cluster on GKE first and get the app up and running. Find out more from my post Orchestrating Docker with Kubernetes.
6. Kubernetes starts rolling update.

This flow works pretty well except that some hacks are necessary for Travis to deploy to GKE. I will explain the hacks and why they are necessary in details in the subsequent section.

Please note that the tools used in this flow can be easily replaced with their respective equivalents. For example, Github can be replaced with Gitlab, Bitbucket or whatever you like. Travis can be replaced by CircleCI, Shippable, or any CI that supports Docker. Docker Hub can be replaced by your private registry. GKE can be replaced by Amazon EC2 Container Service or any Containers as a Service (CaaS).

All source code can be found in my repo.

The devil is in the detail

sudo: required
language: node_js
node_js:
  - '5'

services:
  - docker

env:
  global:
    - secure: AoSvVfpX77AtMBpXKyHH67wTKWCN9xdXoMWroU2TwewBRToebQmLUesT+6gP2rCVoQ282IpESkZkIUumj38rvCGWpL3fLU67Fb1Fa/SqKQ++OYri3NNmoOLltkHMCHHz2bl7B8/72KJQ8e+sl8KCmKBTaLp1g2/36+DZwL9KZjFOqsQg2pwv3zjOUkZte6v6Igsl7lV1pbLkg9Fq1KEvOZl8D6bHgtNuHJGYZZrHCUDHMXxXeQ8/wL0v7GpUkZAe85Ve+fkPoX7AXYumbi5SsxbwVYG64j4zdU2ydlhmMRfOrT4BIbOHUO1kmP5uSf4/xX+fGJZuGDlVT1P5RTZyvu1dA+X10a0t2pOYHAjTjJp6CNRIKuJ4izBvxCZaLAUd42FfU9BAFyUPlViaDzNZAW3DCoapfnS0xM+UY1U8Z7bs+lIXycqIE6OR4KZXfKtpoKakmh8d0eY8LpUNu81VS9Z2mxfRg0xsYhp/1E/X+LZM/YCzPpOhQCc0ZGJEtbqj1/Qebp/GRBIx9oabBH+JxldqrZApIF0MsGiaIT8Q0LJ8wNVVc7QQqhjwdtvP1Wh2pxOvnzEyAdB4AeKrYv0SMwa3OxpIDtJEP1AvXRTZWXPObSSgAhK2VPnOK9Y++N2BlSKJh9k1nlnzn0FQsHlPUV/r9Ui62tmQbzOyL69xltQ=
    - secure: KLweub+lnStvWmKY/XoDy8nXo4v5lYTHapXtewfDFrnWPL6UQV//kklUAswzGEWp4id9pOolZB4RNfucCBzB1Kb1KgSp23wz9mj+HWY+qUQRsLyIxYwsZf/o9H/MMkesG8jOF/vU35Ne3e7MdSmpNuqEqErIqc25691rBMLSDxi5YS3jcB+vGkv38xtlcDilnUrSneKuNEzuI5Q7MKr6vqIVaW3vbwEs5hEh2oPSAgVno67s9NyfUi2Dl4sIDDwzV+iXc9FNk4Ww6Vkm8uzZvIXhZDuyVnFWSJhgfNgElUna7XfqnWOyPdZQ0rh2iOtKzO9uZQiOoNq8JuT9VstFcFlhsfAtlynB5Xgp/EH9NUIZvUiueRhs0bOH9SKRUijIeTQ/OimuX0y8EZdz7TKSigZkc7iVqD0g2E2kLtv3avTCodwts54V8bX7//r8Y2FE7DFkZRGL7Khf5LPjRj7xVEb9OhEkuSYfs0oEhqyXCSMUNyov3BDFbZ/AWh0smQmKYv36U2JvMfPghNlgAE+b/Xc4F/sJMmNSDdLBmGwoJDJtVA5iB3RCfkZfEeqhMoj2C8uu8s8IASzxh2HnaaO5IQKy7kGwAzzRfazTq6JhvLPfteOKStvRYrLLBll7l2DPazLJK8ctKZ6CTFxoycU+R73IliwwpqLCAhasD9N7Q8w=
    - secure: htu5A6fjnQvJTvinumWI1u1unoDMgY5FleGM8JwVLBQl6Srr1GU/2ulii1LSbypG/JimwMzT6BIRMfNgCzAeBc6aWM8xbZUsLl7p+WFyYNxOW26mVm8Z5R+2JZXm3vZoevjHD35gWIBMReqVySIRLXanZ9SOVRC1IcX2Om1uOoQHAqwrFh4KERWnzepAXylfUtFatqmRmCRczH4m9MKF7OgbZD/7xKHHoJxXLHjmWaCQrLmucb01/e/vl3C/LwOvc+A6fwXhhMFdL5UUn9iVWLBhTYknULjOKDF2AgvahvStxGUEAcscaW4qATW32Ylamn3K4l5X+3ic+fLKQHOR+oCaVmaqcpExbxuBfLHGQkV7DiUePQsS1D5NLfAHBdbd8MJCwxQRlSIt/z/aemkx6uWEBu053c47yuuKltFhAm6B7Z52UkHC8fpFfuy03xgorjNTayghPlYDeFZRzItkibTEY1EH4Sh9yVkAIhv5hqomMQMAkxaEFgmBkK7/8+UHB+sMlBupMiejTqIgZOzIhj1uqDGkokqI2UqlNx1wuqzGJ/KnQWzsgFsnoU0Q1zIzw0i1RAiDu8qj5vSnioqw8pvZWuBDuRltE5U4fup529ZtFSNx4Ceb3ECathc6+3Em8OvtqvBOr44xNBW8XCFbi82Ys3Dd/SEHGwlcwYoT4/A=
    - secure: W1CzoMk6zUyhKhx+Q4mX/O1rh7GKFTrMNpkJ+QqN4WN56K3v49t6kotABXbFU4K8ZP84DhUkQYtm6DbEgCkxeNaT0k0gzKyG+CZo5sI6yqx15CwmR/qCkFruDQqUG/2hZNTP5ZP/7hMQwNCxATxuzu2GOcnwk1YuIJsJ5by9+RdWwKBYYhSG/BokNq6JaYxZx8NeMTa5W/CJXtu/5qFsMDQ07Syg4MgAnRHbRB9uCxpUOtJ44HDhZFOOY48qIDVpYhVpTg0XVzw5VkFNgOuczIYWw/SaWyOyoKgt8FtLhUPDsHtvueFqSJffYlyoh81Uxu+LNAtZoact5wFOwkDM+mzqRe/lzm2hcYY0nc4Dxc1ito43GedQmFnadZSozZfKAEmRV0m+1W6iE6fi+DpHXm3JCmgvolB9khLiYVTHkgaSzbfZDfdBmoXPEcaDuKnY6KTB410sGnadO6IT3Z5x8FtMQ/In/dB7XQr/G1aXve3KjUz8/oLZbS+IWEH8eQEOAR8V6w6HpvQIZE6ccwEzmLILlygne7TSAnOPQX1hiEDdUYYOtoyVEv/NUzd/OpGqNl6edYsXZB1bcY7FWy2YhmJD4mqe5h6mp4yRXYyuFbdC3OIBGpr1vnNaF9uUdayZ59pZ4kIojCQL2SKZKqOYdezyV/h0ohP93FIboi/nL7Q=
    - secure: jxY+/q6svho30zk1RMtnwHhHkYMw+NqQS4n7dOILpsI6zKwC4n2c+xjD0MoOYjVsd/SiJucXeeNR4BcS4+OglsVxja68rBWqV4F3xloGwtMZebqMhNGSV4kitgAkdDNktQid2fQAwUsoCtEfq3A6ijMdOgoehZwxXqFg5+5cRwjI58vVOfdFDGPUO6KXTDhwwXIgTi9zN8nXSNCgHUZfrIGw6jTdFvL2NGkejbX4AqutethTlXlA7lVeE/O1SUW4H8MRl59MyKUwWm/j7qkmM/TX+PUaXSwXgoR0Nzbt2DKTbY56qff1olHaUMuV5c8A5gYlThvVYbMnGRTK7JYOzmchyvRsqc1inxIwU7bi/J1DnQpCX4qF06NBU+OLbx8+qDc9E6l9XGdQ09HFVPOctlNrkfcPDgGVfRYwomU8+gb8MykgBxtUqLIpJN0pr9sNZN4L4jGuv58ThkCfoaH6YgiLKTITJ7cLAE2tNmHj5Kw+vHNvJcUltRdMOSaDI6TyAGkVpUSHaTEOvdvsj5y5wFS3TdIHnTQOWMpBiYSHY7D4trvv6KRagDjoB7DKaAohYPwataJdu3Wu+A0UIsukYPG6uVn4RPJmQHBMT8vTtf9q9eSpoRU68zSnhAL5aXtq1vqtfy4859QQDzpH7JMA87WtJqCQ4Fp8MHOg3uvb+ZM=
    - secure: FXEj1gwSi+xjpc4GFFS4ehxy6Q4wpmmPq/rlfWijOAYsNwukfkL2gCmnzjTT6ZZVHM47bNtHlTdqTM8bmBjWVeVEPKwvULq6i2neiGI7OI0vh2dsbCMIZ7bCMuRL0/WRq/A8YPsXIK76YIlMg8sSzdGdAmvZpjTbLywSl9KoSlH3S56e2Nz1kb8DhP4emeFcxdXul463JUdOBpCZa8oJLopOaX5+PypOk6XOjOyWkxN938F4w5/4MzhgghcrTRwtOQCPHyJR2zg08MWm5COKi3L2EmgPyctXGGr31YVeRTC9Gg3SJ+Mt8OfsX8f7TnUzhnfA3IszJpxw4cF1snpVmXZED2J1Aa6Gi7mSIhb1CdYNUxUKx6BVnaUGDRfmcxtGOt8Uy0A2sRl3FGhVVH6xQaaTmG3nzU0fm44cO1RwxqXfP4kXqUNj+LlCUC+3hIRrWRMfxjIWwGA0LZIqnVlou2TRLkxdM3IHx46cFqEfpD6Do/NXOCUZYRcE95FqvK6ZrobsZapkI0WkmoO4YdzCkuW9nSc8Y121OWlP9urkzaU4X7tO7FbLshKmjYNL4vLqyJ0gkN21yVJTIJWE6l/TGGi+RxwFF2jzN8oRjOJrWESmUSPa5WhNvuIOu1t0RObwRup0yjArRuUIXxuQt99c1tkfsn52fZWZih3t4bM0bxE=
    - CLOUDSDK_CORE_DISABLE_PROMPTS=1
    - COMMIT=${TRAVIS_COMMIT::8}
    - IMAGE=chenglong/simple-node
    - IMAGE_VERSION=$IMAGE:$COMMIT
    - DEPLOYMENT=frontend
    - CONTAINER_NAME=nodejs

before_script:
  - docker build -t $IMAGE:$COMMIT app
  - docker tag $IMAGE:$COMMIT $IMAGE:latest
  - docker tag $IMAGE:$COMMIT $IMAGE:travis-$TRAVIS_BUILD_NUMBER

script:
  - docker version
  - docker-compose version
  - docker-compose -f docker-compose.ci.yml run test

after_success:
  - docker login -e="$DOCKER_EMAIL" -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD"
  - docker push $IMAGE
  - curl https://sdk.cloud.google.com | bash
  - source /home/travis/.bashrc
  - gcloud components update kubectl
  - kubectl config set-credentials default --username=${GKE_USERNAME} --password=${GKE_PASSWORD}
  - kubectl config set-cluster default --server=${GKE_SERVER} --insecure-skip-tls-verify=true
  - kubectl config set-context default --cluster=default --user=default
  - kubectl config use-context default
  - kubectl patch deployment $DEPLOYMENT -p '{"spec":{"template":{"spec":{"containers":[{"name":"'"$CONTAINER_NAME"'","image":"'"$IMAGE_VERSION"'"}]}}}}'

This is the .travis.yml that instructs Travis to

• build the Docker image chenglong/simple-node with 3 tags: latest, ${TRAVIS_COMMIT::8} and travis-$TRAVIS_BUILD_NUMBER
• run tests against the newly-built image chenglong/simple-node:latest. Note that the tests are run in a container named test. The tests basicially verify that the app gives expected response. See test code here.
• push the image to Docker Hub if tests pass
• install latest Google Cloud SDK because the one on Travis is too old
• install kubectl
• config kubectl to connect to the a pre-existing cluster on GKE
• rolling update deployment frontend with the newly-built image chenglong/simple-node

A few key points:

• chenglong/simple-node is ONLY built once. It's the same image that is tested against and deployed to GKE. See the logs for a build here.
• Tests are run in a container named test, which simply starts app and redis, fires a few requests and verifies the response. To run the tests, I used docker-compose -f docker-compose.ci.yml run test. See docker-compose.ci.yml
• At the time of writing, Travis CI only has docker-compose 1.4.2, which does not support compose file version 2. This requires the compose file to be in version 1, unless you want to install docker-compose 1.6+.
• Travis doesn't support deploying to GKE yet, which requires some hacks to install the latest Google Cloud SDK and kubectl, so that I can do kubectl patch deployment.

Conclusion

I hope this simple demo gives you a better idea of how Docker can help in your day to day work. A few obvious advantages of this Docker workflow:

• It significantly improves how we ship software. The artifact from CI is NOT anymore a tar ball, jar file, war file, nor ear file etc. It's simply a Docker image. A Docker image has not only the source code baked in, but also a complete running environment. Therefore, it removes the need to first provision server before deploying source code. Do you see Chef, Puppet or Ansible in the demo? In the age of containers, all you need is a server that runs Docker.
• Since it's the same image that is tested in CI gets deployed to production (or maybe staging), it's guaranteed that it works in exactly the same way.
• By containerizing the app, each component of the app is run as a container. And each container can be individually updated and deployed. Does this ring a bell? If not, read Microservices.

With multi-host networking ready for production and the announcement of Swarm 1.0, I think it's time to give Docker a serious try. This post details the steps I took to orchestrate a multi-host and multi-container app.

Architecture

This is a simple Node app that uses Redis as database, load balanced by Nginx. Each blue box is a Docker host, which runs several containers. All hosts talk to Consul for service discovery. The cluster has 5 nodes, each serves a specific purpose. We want to easily scale up and down the number of app containers.

To achieve this, we will use the following stack

• Docker Compose >= 1.6
• Docker Machine >= 0.6
• Docker Swarm >= 1.0
• Consul
• Registrator

All scripts and the compose file for this demo are available here.

1. Create and Run Consul

Consul is an excellent tool for service discovery. It works great with Docker. You could also use etcd.

Let's create a Docker host for Consul

docker-machine create -d virtualbox consul

This will create a new Docker host named consul on my local virtualbox. It's certainly possible to create a Docker host on a supported cloud provider, e.g. AWS, Digital Ocean, Google Compute Engine, etc. The difference is the driver. Checkout driver options and arguments.

Once consul is created, connect to it

eval $(docker-machine env consul)

Run progrium/consul in background.

docker run -d -p 8500:8500 -h consul --restart always gliderlabs/consul-server -bootstrap

Verify Consul is working

curl $(docker-machine ip consul):8500/v1/catalog/services

You can also go to Consul's web UI http://$(docker-machine ip consul):8500/ui

2. Create The Swarm

Create the Swarm master

docker-machine create \
    -d virtualbox \
    --swarm \
    --swarm-master \
    --swarm-discovery="consul://$(docker-machine ip consul):8500"\
    --engine-opt="cluster-store=consul://$(docker-machine ip consul):8500" \
    --engine-opt="cluster-advertise=eth1:2376" \
    swarm-master

This is a very long command. Let's take a detailed look.

• -d virtualbox indicates the driver is virtualbox
• --swarm configs the newly created machine with Swarm
• --swarm-master dictates the newly created machine as the Swarm master
• --swarm-discovery="consul://$(docker-machine ip consul):8500" designates Consul as the discovery service
• --engine-opt="cluster-store=consul://$(docker-machine ip consul):8500" designates Consul as the distributed KV storage backend for the cluster
• --engine-opt="cluster-advertise=eth1:2376" advertises the machine on the network

Create a node for load balancer

docker-machine create \
	-d virtualbox \
    --swarm \
    --swarm-discovery="consul://$(docker-machine ip consul):8500"\
    --engine-opt="cluster-store=consul://$(docker-machine ip consul):8500" \
    --engine-opt="cluster-advertise=eth1:2376" \
    --engine-label host=load-balancer \
    load-balancer

Note that we give this machine a label host with value load-balancer, which will be used for scheduling later.

Create app server 1

docker-machine create \
	-d virtualbox \
    --swarm \
    --swarm-discovery="consul://$(docker-machine ip consul):8500"\
    --engine-opt="cluster-store=consul://$(docker-machine ip consul):8500" \
    --engine-opt="cluster-advertise=eth1:2376" \
    --engine-label host=app-server \
    --virtualbox-cpu-count "2" \
    --virtualbox-memory "2048" \
    app-server-1

Note the two added parameters --virtualbox-cpu-count "2" and --virtualbox-memory "2048", which gives the app server more resources. You should adjust these values according to your needs.

Create app server 2

docker-machine create \
	-d virtualbox \
    --swarm \
    --swarm-discovery="consul://$(docker-machine ip consul):8500"\
    --engine-opt="cluster-store=consul://$(docker-machine ip consul):8500" \
    --engine-opt="cluster-advertise=eth1:2376" \
    --engine-label host=app-server \
    --virtualbox-cpu-count "2" \
    --virtualbox-memory "2048" \
    app-server-2

It's best practice to put your app servers in different availability zones or even different cloud providers to achieve high availability. Checkout the drivers reference for options.

Create database node

docker-machine create \
	-d virtualbox \
    --swarm \
    --swarm-discovery="consul://$(docker-machine ip consul):8500"\
    --engine-opt="cluster-store=consul://$(docker-machine ip consul):8500" \
    --engine-opt="cluster-advertise=eth1:2376" \
    --engine-label host=database \
    --virtualbox-disk-size "40000" \
    database

Note that we specify --virtualbox-disk-size "40000" because it's for running database. You can adjust this according to your needs.

Connect to the Swarm

eval $(docker-machine env -swarm swarm-master)

Check cluster info

docker info

You should see 5 nodes in the cluster, namely swarm-master, load-balancer, database, app-server-1 and app-server-2.

You can also run

docker run --rm swarm list consul://$(docker-machine ip consul):8500

to find out all nodes in the cluster.

3. Run registrator in each host

To automatically register and deregister services for all Docker containers in each host, we need to run gliderlabs/registrator in each node of the cluster. Registrator will also use Consul as the KV store.

Run the following script

After this, the infrastructure is ready. It's time to deploy.

4. Docker Compose

We will use the following compose file to start the app

A few important points:

• chenglong/nginx-consul-template is a simple image that use NGINX and consul-template to load balance any service as instructed. In this case, it's service myapp. It requires a running Consul for service discovery. Environment variable CONSUL_URL should be set.
• chenglong/simple-node is a simple Node image that uses redis as database. All it does is to display the hostname and the number of times the page is visited.
• constraint:host==load-balancer, constraint:host==app-server and constraint:host==database are node filters which tell the Swarm master to schedule the corresponding containers on matching hosts. So in this case, the load-balancer container will be scheduled to run on Docker host load-balancer. All app containers will be scheduled to run on either app-server-1 or app-server-2.
• There are two overlay networks frontend and backend. This is what makes multi-host networking possible, i.e. load-balancer can talk to app and app can talk to redis.

Set CONSUL_URL

export CONSUL_URL=$(docker-machine ip consul):8500

Start the app

docker-compose up -d

Verify that all containers are running

docker-compose ps

Test the app works by repeating curl $(docker-machine ip load-balancer) a few times. You should see the counter running.

5. Scaling

Now you can easily scale the number of app containers

docker-compose scale app=4

This creates 3 more app containers.

By default, Swarm use spread scheduling strategy. So all app containers will be shared evenly by app-server-1 and app-server-2.

Test load balancing works by repeating curl $(docker-machine ip load-balancer) a few times. You should see the hostname cycles the 4 containers and the counter is running.

Scaling down is easy too

docker-compose scale app=2

Summary

With Docker Swarm, Machine, Compose and Consul, it's not hard to scale and schedule Docker containers to a cluster of nodes. Although I demoed with virtualbox, you could do the same with Digital Ocean, AWS or any supported cloud provider to deploy Swarm clusters for production.

Since Swarm is native for Docker and it follows "batteries included but swappable" principle, I feel it's much more natural and easier to use than Kubernetes. I will try to do a post about Swarm vs Kubernetes in the near future.

Many people use Gmail as personal emails. Companies use Google Apps for Work. Everyone is happy using it. Not until you hit its limits.

I recently learnt its limits (shown below) in a hard way.

I will hightlight the most restrictive constraints from the above:

• You can send max 2000 emails per day from one account, 500 for trial accounts
• 3000 unique recipients per day, 500 for trial accounts

From what I understand, normal Gmail accounts have the same limits as trial accounts. What are trial accounts? When you sign up Google Apps for Work, all users are in trial period for 30 days. And it's free during this period. After trial, you have to pay ~$5/user/month.

The catch is What if you want to send > 500 emails during trial period? This is the problem that I had.

According to Google,

At the end of your free trial period, your sending limits will be automatically increased when your domain is cumulatively billed for at least $30 USD (or the same amount in your currency). To expedite this process, you can manually prepay this amount as suggested here. It will take up to 48 hours to upgrade your sending limits after you submit the manual payment. Note: while you're still in your trial period, your sending limits will not be increased.

So I signed up Google Apps and setup Email with my custom domain. Made manual payment of $31. After ~550 emails, it stopped with following erorr

Daily user sending quota exceeded. s197sm16092990pfs.62 - gsmtp

I contacted online chat support. Surprisingly, I got reply immediately. It turned out that even though I had made the payment of $31, it DOES NOT automatically end my trial period. So the limit of 500 emails still applies. The solution is Google support staff sent me a new Google Apps contract via email. Unpon agreeing to its T&C, it effectively expired my trial period and changed my account to paid subscription. I was told that it may take 48 hours for the new limit of 2000 emails/day to be effective.

So far, contacting Google support seems to be the only way to upgrade from trial limits within trial period. I haven't found any setting that I can do it on my own, which is weird. If customers don't want to enjoy the free month, they should have the capability to do so.

If you are doing Rails, chances are that you have RSpec and Cucumber tests. They are great tools to give you the confidence that your software is working as expected. But as project grows, you may find that your tests are getting slower and slower, expecially the Cucumber tests. It not only slows down the speed of your local development, but also your Continuous Integration pipeline (Test is probably the first stage in your CI). We don't want to waste time waiting tests to finish in either local or CI. We want to develop and integrate faster. Here is how.

1. Install parallel_tests

parallel_tests is a great gem to run Test::Unit, RSpec, Cucumber and Spinach tests parallel on multiple CPU cores.

Update your Gemfile

group :development do
	gem "parallel_tests"
    # other gems...
end

Update config/database.yml

test:
	database: db/test<%= ENV['TEST_ENV_NUMBER'] %>.sqlite3

If you are not using sqlite3 for test db, update it accordingly.
Each CPU core will run its share of tests using its own test db. So if you have 8 CPU cores, you will see 8 sqlite3 files generated in db when running tests.

Then

bin/bundle install

2. Use Binstubs to Run Parallel Tests

If you don't know what is binstubs and why you should use it for all commands, take a look at this and this.

Generate binstubs only for parallel_tests

bin/bundle binstubs parallel_tests

It will generate 4 binstubs in bin

• parallel_cucumber
• parallel_rspec
• parallel_spinach
• parallel_test

You don't need to keep all of them. Only keep the ones that you need. In my case, I only use RSpec and Cucumber. So I deleted parallel_spinach and parallel_test. Add the rest to version control.

3. Run Tests in Parallel

Run all specs in parallel

bin/parallel_rspec spec

Run all features in parallel

bin/parallel_cucumber features

This will launch N browsers, where N is the number of your CPU cores.

Be amazed by how long it takes to run all tests now. Theoretically, the test running time can be reduced to T/N, where T is the time to run in serial and N is number of CPU cores. But that's not achievable in practice because the number of tests may not be split evenly among the processes. And even if they are split evenly, they may not take equal time to run. Besides, there are overheads to coordinate the processes and collate results. However, it's very easy and common to achieve 20% ~ 50% reduction in test running time, which can potentially save 5 ~ 10 mins in a medium size project. In a long run, it's a huge time saving.

4. Tips

Not all tests are equal. Some tests are for must-have features and some are for nice-to-have features. Some tests fail more often than others. To get feedback faster, it's a good practice to group tests into categories and run them in order, e.g.

bin/parallel_cucumber features -o '-t @smoke'
bin/parallel_cucumber features -o '-t @flaky'
bin/parallel_cucumber features -o '-t ~@smoke,~@flaky'

The above example will

1. Run all smoke features. More on smoke tests
2. Run all flaky features
3. Run the rest

This way, it avoids running the smoke tests and flaky tests at the very end.

Summary

It's quite easy to setup parallel tests. It not only speeds up your local development, but also CI pipeline. And the time saving is tremendous in the long run.

Life is Short. Run Tests in Parallel!

What is HTTP/2 and Why

HTTP/1.1 has been serving most part of the Web since 1997. As websites get more and more sophisticated and resource intensive, it starts to show its limitations, e.g. one outstanding request per TCP connection. So its next-generation emerged: HTTP/2.

HTTP/2 FAQ does a great job explaining the background and specifications. Highly recommended. Here is an executive summary, HTTP/2:

• is specifically designed to improve performance
• is based on SPDY
• is binary, instead of textual
• is fully multiplexed, instead of ordered and blocking
• can therefore use one connection for parallelism
• uses header compression to reduce overhead
• allows servers to push responses proactively into client caches
• is backward-compatible, designed to be drop-in replacement for HTTP/1.1
• is supported by most broswers over TLS

HttpWatch reported good performance improvement by using HTTP/2.

How to Upgrade

Upgrading from HTTP/1.1 to HTTP/2 is quite easy. You just need to make sure that your web server supports HTTP/2 and "turn it on". I will use NGINX as an example.

Install NGINX 1.9.5+

In the case of NGINX, only 1.9.5+ supports HTTP/2.

Check your NGINX version

nginx -V

If it's lower than 1.9.5, you need to upgrade NGINX first. Otherwise, head over to Turn On HTTP/2.
At the time of writing, the latest stable release of NGINX is 1.8.1, which is lower than 1.9.5. So you need to install NGINX Mainline version. Don't worry that Mainline version is not stable. It's actually better than Stable version because it has the latest bug fixes.

On Ubuntu,

Install NGINX signing key

wget http://nginx.org/keys/nginx_signing.key
sudo apt-key add nginx_signing.key

Add the following in /etc/apt/sources.list

deb http://nginx.org/packages/mainline/ubuntu/ <codename> nginx
deb-src http://nginx.org/packages/mainline/ubuntu/ <codename> nginx

Note that <codename> should be the result of lsb_release -c | cut -f2.

Then install

sudo apt-get update
sudo apt-get install nginx

Compile NGINX from Source to Support ALPN (Optional)

Depending on the OpenSSL version that your NGINX is built with, it may not support Application Layer Protocol Negotiation (ALPN). More details here. Besides, starting from May 15th 2016, Chrome will ONLY support ALPN, which means supporting ALPN is necessary for HTTP/2 to work fully.

Find out NGINX build details

nginx -V

If it says built with OpenSSL 1.0.2f 28 Jan 2016, you don't need to compile NGINX from source. Jump to Turn On HTTP/2.

If it's built with OpenSSL lower than 1.0.2f, e.g. 1.0.1f. You need to compile NGINX from source with OpenSSL 1.0.2f. The detailed steps can be found here.

Turn on HTTP/2

In site.conf,

server {
	listen 443 ssl http2;
    ...
}

That's all you need to turn on HTTP/2!

Please note that although HTTP/2 doesn't require HTTPS, most web browsers only support HTTP/2 via TLS. So you do need to serve your site via HTTPS in order to use HTTP/2. If your site isn't using HTTPS yet, check out my post on how to use Let's Encrypt to make it HTTPS.

Reload NGINX

sudo nginx -s reload

Refresh your site and inspect the page, you should see that the assets from your site are loaded via protocal HTTP/2 (h2).

This blog is using HTTP/2. You can inspect this page to see HTTP/2 in action.

Another way is to let KeyCDN do the test.

If you prefer command line

echo | openssl s_client -alpn h2 -connect yourserver.com:443 | grep ALPN

You should see ALPN protocol: h2.

Summary

Although HTTP/2 is only out for about one year, it has very good adoption thanks to its backward-compatibility, easy of upgrading and performance benefits. I'm convinced that HTTP/2 is the future of the Web.

Wait no more, upgrade!

Reference

• HTTP/2 Home Page
• HTTP/2 Specifications RFC7540 and RFC7541
  
• http2 explained