The path of a software engineer is one of constant learning. We learn things from concepts and processes to languages and tools. Once we have seen them work, we add them to our arsenal and make them our praxis.
Before joining CircleCI, my years of experience led me to believe that I was an engineer with a firm understanding of the technical aspects of the craft, as well as what is considered good practice. CircleCI showed me that I had still more to learn.
At CircleCI, my team uses a short-lived branching model for software development. While this strategy goes against everything I once thought was the ‘right’ way to develop software, in this post, I’ll describe how seeing it in practice has taught me to embrace it.
Our model
The most popular branching model is GitFlow. For the purposes of this post, I’ll simplify it as:
- Use one branch for development work.
- Use one or more branches or tags for different levels of production-readiness, hotfixes, etc.
- Gradually move code from feature branches to development, and then to production as a feature becomes tested and ready.
Original blog post here.
This is the branching model I was familiar with. The one I had experienced in every project, at every company, and it had served me well. With perhaps minor variations, that’s what I thought everyone did.
What a surprise, then, to arrive at CircleCI and find that there is no staging environment, no hotfixes branch, and what is on master goes directly to production. Yes, while we work on a ticket, we create a branch. But this branch often lasts less than a day. It might only partially implement a feature (protected by feature flags or just not user-accessible), get reviewed, and get merged. Then it goes to production. Again, no staging environment! No QA team to check that things are working! No release window timed to your sprint (because, spoiler, there are no sprints)!
Why this model?
Why we do this has a lot to do with CircleCI’s mission to empower teams to deliver value quickly. The VCS branching model you use has a big impact on your delivery cadence. Our model synergizes extremely well with CI/CD best-practices. The characteristics of our flow, like short lived branches, and fast deployment cycles, have been shown to correlate with high performance teams.
How you can do it
If you are accustomed to a more traditional flow like I was, this probably sounds scary. I like to think I write good code, but software has bugs; that’s inevitable. CircleCI is a mission-critical part of hundreds of thousands of developer’s path to production. That carries a lot of responsibility. What if I push a bug to production and there is no QA team to catch it? What if I break a feature? What if I bring the whole thing down?
I worried that my colleagues didn’t seem concerned. Were they so much more experienced than me that they trust their code to be flawless? Are they just naive?
There is an incredible amount of talent at CircleCI. However, the confidence in the software is not the result of hiring mythical 10x engineers. It is the result of working every day to make our continuous integration pipeline better and more reliable. Here are the ways that we accomplish this:
- Testing, in its many forms, is expanded from unit testing and security testing, to load and stress testing.
- Any developer can deploy partially using canaries. This contains the blast radius of faulty code to a very small fraction of traffic.
- We implement mean time to recovery reducing measures, including monitors and meaningful alarms, diagnostic tooling, and the means to quickly roll back (or forward).
- Blameless incident post-mortems are conducted to establish failure modes and minimize future reincidence, providing feedback for this strategy.
- We keep a constant eye on the balance between mean times between failure and time to recovery, between reliability and agility.
There is always room for improvement, but this strategy provides incredible value. We would not be able to deliver value as quickly using a “safer” flow like GitFlow or if we had a dedicated QA team.
Conclusion
It’s been over two years since I joined CircleCI. And while I concede there is no VCS branching model that fits all teams, I am happy to have learned that the option CircleCI chose exists, and to have experienced how it affects our delivery cadence, and the engineering culture we build around our product.
When every developer is responsible for what is in production and treats every commit as a release candidate, it fosters a culture of investing in sustainable software quality, for their own benefit as well as their teammates’. We build this together.
