Posts on Random Musings

Let's Encrypt Intranet

Sun, 03 Sep 2017 14:33:24 +0000

Let’s Encrypt (LE) has been a popular choice to get certs for public websites. Because it’s free and automated. But how to get certs for private websites, which are common in company’s intranet?

Problem

There’s a web app in your company’s intranet.
The web app has a fully qualified domain name (FQDN), e.g. foo.example.com, not an internal one like foo.internal.
It only resolves to a private IP behind VPN. Therefore, it’s inaccessible without a valid VPN.
You want to add an extra layer of security by enabling HTTPS.

How to get a cert for it? And how to automate it and get it for free?

Multirepo vs Monorepo

Thu, 20 Jul 2017 09:32:00 +0000

Here’s a conversion I keep hearing recently:

A: Let’s put all our projects in one repo.

B: Why?

A: Because Google and Facebook do monorepo.

Whenever I hear this, I’m very tempted to show this picture.

Finding the origin of this picture is left as an exercise for the readers. On a more serious note, I want to write down my thoughts on multirepo vs monorepo.

What is multirepo?

One project one repository. Each project is an independent working unit. It can be a mobile app, frontend app, backend service or standalone CLI app.

Be wary of http/client.go

Sat, 25 Mar 2017 09:30:00 +0000

Recently, I found out an interesting problem in Go. The problem can be reduced to a simple client request to a HTTP server.

Suppose we have a HTTP server, which serves only one rooted path /foo/.

package main

import (
	"io"
	"log"
	"net/http"
	"net/http/httputil"
)

func handleFoo(w http.ResponseWriter, req *http.Request) {
	// request details
	dump, _ := httputil.DumpRequest(req, true)
	log.Println(string(dump))

	if auth := req.Header.Get("Authorization"); auth != "Bearer GoodToken" {
		http.Error(w, "401 Unauthorized", http.StatusUnauthorized)
		return
	}

	io.WriteString(w, "Hello World!")
}

func main() {
	http.HandleFunc("/foo/", handleFoo)
	log.Fatal(http.ListenAndServe(":12345", nil))
}

handleFoo simplily verifies that correct token is sent in the Authorization header. Otherwise, it returns 401 Unauthorized.

Keeping configurations sane for multiple projects on Google Container Engine

Sat, 18 Feb 2017 09:27:00 +0000

In my previous post, I present the easist and most secure way to get kubectl working for one project. But what about mutiple projects? Juggle mutiple projects on Google Container Engine (GKE) can be hard, especially when its configurations are admittedly quirky. This post describes the best practice, in my opinion, to keep configurations sane and easy to switch.

Problem

Suppose you have an awesome app that runs on GKE. You probably want to have two different environments staging and production, and the environments should be completely isolated. So you create two projects on GKE, awesome-app-staging and awesome-app-production, and provisioned resources for each. Now the question is how to effectively switch between the two projects on command line without repeating these commands over and over again.

kubectl Authentication Made Simple

Mon, 30 Jan 2017 09:25:00 +0000

While working on a continuous delivery pipeline to automate deployment to Google Container Engine (GKE), I found that getting kubectl to work is very complex and convoluted, especially when it needs to be noninteractive. So I want to find out the easiest way to get kubectl working noninteractively.

Assuming that gcloud and kubectl are already installed but not necessarily setup, ONLY two commands are needed to get kubectl working noninteractively (verified with Google Cloud SDK 141.0.0 and kubectl 1.5.2)

Speed Up SSH

Sun, 04 Dec 2016 09:23:00 +0000

Recently I worked on a project where executing remote commands is very slow to start. This is expected because an SSH connection has to be established first.

$ time ssh server exit
ssh server exit 0.04s user 0.01s system 0% cpu 7.901 total

It took nearly 8 seconds to ssh into server.

That’s slow!

After a bit of googling, there is an elegant way to speed this up. Simply put this at the bottom of your ~/.ssh/config.

Ruby and Java Stack Level

Sat, 05 Nov 2016 09:17:00 +0000

While coding for an algorithmic problem, I discovered that Ruby’s stack level is much shallower than Java. This caused a recursive DFS solution written in Ruby failed due to stack level too deep (SystemStackError), while the same code written in Java passed. Whether recursion or tail recursion should be used is not the point of this post. This post is to find out what the max stack level is and what limits the stack level.

Docker Workflow

Wed, 25 May 2016 09:07:00 +0000

In my previous posts, I demoed how to orchestrate Docker with Swarm and Kubernetes. They all assume the Docker image chenglong/simple-node is already there and ready to be deployed. But how to develop that image in the first place? How to streamline and automate the process of developing it on local machine, building and testing it on Continuous Integration (CI) server, and finally deploying to production?

This post introduces one possible Docker workflow.

Orchestrating Docker with Kubernetes

Sun, 15 May 2016 09:04:00 +0000

This is my second post on Docker orchestration. In the first post, I demonstrated orchestrating Docker with Swarm, Machine, Compose and Consul. This post is to demonstrate orchestrating the same app with Kubernetes and draw comparisons between them. I recommend reading that post before this.

Introduction to Kubernetes

Kubernetes is an open-source system for automating deployment, operations, and scaling of containerized applications.

It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon a decade and a half of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community.

Orchestrating Docker with Swarm, Machine, Compose and Consul

Fri, 15 Apr 2016 09:01:00 +0000

With multi-host networking ready for production and the announcement of Swarm 1.0, I think it’s time to give Docker a serious try. This post details the steps I took to orchestrate a multi-host and multi-container app.

Architecture

This is a simple Node app that uses Redis as database, load balanced by Nginx. Each blue box is a Docker host, which runs several containers. All hosts talk to Consul for service discovery. The cluster has 5 nodes, each serves a specific purpose. We want to easily scale up and down the number of app containers.

Gmail Sending Limit

Fri, 08 Apr 2016 08:58:00 +0000

Many people use Gmail as personal emails. Companies use Google Apps for Work. Everyone is happy using it. Not until you hit its limits.

I recently learnt its limits (shown below) in a hard way.

I will hightlight the most restrictive constraints from the above:

You can send max 2000 emails per day from one account, 500 for trial accounts
3000 unique recipients per day, 500 for trial accounts

From what I understand, normal Gmail accounts have the same limits as trial accounts. What are trial accounts? When you sign up Google Apps for Work, all users are in trial period for 30 days. And it’s free during this period. After trial, you have to pay ~$5/user/month.

Life is Short. Run Tests in Parallel

Sun, 20 Mar 2016 08:56:00 +0000

If you are doing Rails, chances are that you have RSpec and Cucumber tests. They are great tools to give you the confidence that your software is working as expected. But as project grows, you may find that your tests are getting slower and slower, expecially the Cucumber tests. It not only slows down the speed of your local development, but also your Continuous Integration pipeline (Test is probably the first stage in your CI). We don’t want to waste time waiting tests to finish in either local or CI. We want to develop and integrate faster. Here is how.

HTTP/2

Mon, 14 Mar 2016 08:53:00 +0000

What is HTTP/2 and Why

HTTP/1.1 has been serving most part of the Web since 1997. As websites get more and more sophisticated and resource intensive, it starts to show its limitations, e.g. one outstanding request per TCP connection. So its next-generation emerged: HTTP/2.

HTTP/2 FAQ does a great job explaining the background and specifications. Highly recommended. Here is an executive summary, HTTP/2:

is specifically designed to improve performance
is based on SPDY
is binary, instead of textual
is fully multiplexed, instead of ordered and blocking
can therefore use one connection for parallelism
uses header compression to reduce overhead
allows servers to push responses proactively into client caches
is backward-compatible, designed to be drop-in replacement for HTTP/1.1
is supported by most broswers over TLS

HttpWatch reported good performance improvement by using HTTP/2.

Using Docker for Rails Development

Sun, 14 Feb 2016 08:49:00 +0000

Why

There are many use cases of Docker. I see people primarily using it for Continous Integration and deployment. But Docker is also good for development. The obvious advantages of using Docker for development are:

No need to install app dependencies on dev machines. App dependencies are built into Docker images. Hence, the dev machines are not messed up with crazy dependencies. The only dependency needed on dev machines is Docker, nothing else.
Have a consistent development environment for all developers. No more excuse like “It works on my machine”!
Onboard new developers quickly. No need to spend hours setting up new dev machine and configuring it. You only need docker-compose up and you can start coding.

Prerequisites

This post will show you how to setup a Ruby on Rails development environment using Docker. My dev machine has

Let's Encrypt Nginx

Sun, 31 Jan 2016 08:41:00 +0000

Update [2017 Aug 5]: Certbot has been developed by EFF and others as an easy-to-use automatic client that fetches and deploys SSL/TLS certificates. I would recommend using it.

Why

Since you are here, you probably know what Let’s Encrypt is and why it exists. If not, below is an executive summary (copied from here):

Anyone who has gone through the trouble of setting up a secure website knows what a hassle getting and maintaining a certificate can be. Let’s Encrypt automates away the pain and lets site operators turn on and manage HTTPS with simple commands.

Gradle Offline Build

Sun, 20 Sep 2015 08:38:00 +0000

The Problem

In a recent project, I have to do Gradle build on a CI server in a very constrained environment. The constraints are:

NO Internet connection
NO Gradle installed
NO local Maven repo

The obvious problem that these constraints cause is that the project can’t build because:

Gradle is not installed
Can’t get dependencies

The Solution

This solution will achieve the following:

Use Gradle to build
Use Gradle to manage dependencies

Let’s solve the problem step by step.

Speed up bundle install

Sat, 04 Apr 2015 08:36:00 +0000

TL;DR Next time, when you need bundle install, do bundle install --jobs X, where X is the number of your machine cores. It will save you huge amount of time.

Some blog suggests that setting the number of jobs to X-1 is statistically better than X. But that is not true anymore, at least with bundler >= 1.7.12. To prove this, I tested it on a large Rails project (discourse) on my Mid 2012 MBP with 2.6 GHz Intel Core i7. Below is my test result:

Less frequently used git commands

Sun, 22 Mar 2015 08:30:00 +0000

From time to time, I find myself or my peers not using git correctly or efficiently. This is largely due to the fact that some git commands are less well known and used. Below are a few commands that could be helpful but less frequently used.

git bisect

Suppose you find that commit c100 is failing your tests. But it was running fine at commit c67. You want to know which commit in the range (c67, c100] introduced the bug.

Full Access to Mobile Technologies Is NOT a Previlege, It's a RIGHT

Sat, 28 Dec 2013 08:17:00 +0000

It’s the end of 2013. I would like to share some thoughts on the new mobile trend.

Several events happened in the second half of 2013:

hongmi is released for RMB799 (~$132) and become extremely popular
Moto G is released for $179
Honor 3C is released for RMB798.00 (~$132). First batch was sold out in 1 minute

Look at these phones, they all have decent display, CPU, camera, etc. And considering their prices, I realize that their value to price ratio is ridiculously high.