So, you’ve been coding away for months on a big project, and you can’t wait to share what you’ve created with the world. All the functionality is there, your test coverage is great and every demo you’ve shown ends in a complete success with love at first sight….
But wait–what if it totally breaks in production? That would suck, obviously. But what are we as developers supposed to do about what happens in production?
Isn’t that the job of the sysadmin or release engineer or whatever that person is called who….well…does all that stuff with apps in production? (Hmmm, what DOES that dude do? Maybe you should figure that out!)
In fact, there are some things that developers can do to help bulletproof their code against failures in production. After all, “Worked fine in Dev, Ops problem now” might be a funny meme, but it’s not really hilarious when things really do go down. So unless you want all that excitement to be washed away by a total meltdown when your code goes live, you can run your application through this quick production checklist first (there’s a 1-question mini poll down at the bottom that will determine how much you suck. Kidding.)
- Is environment configuration separated from the application?
- Is the application self-contained, packaged and tested with database migrations and configuration scripts?
- Has the application been tested with more than a few concurrent users?
- Do you have a quick set of acceptance tests to run as part of the deployment?
- Does the application expose key internal metrics?
- Is the application backwards and forwards compatible?
Don’t panic if you answered no to most or even all of the questions, as the majority of companies I’ve peeked at are in the same state. That’s why everyone is always stressed at the end of the project, and why there’s a growing list of companies that you’ve only heard of due to bad press after a failed launch.
The good news is, once you’re aware of the issues, they’re easy to fix and transform releases into more or less a minor activity. I’ll walk you through the reasons why I’ve chosen these questions, and some quick ways to tackle them.
1. Is environment configuration separated from the application?
You don’t want any changes in the application after it’s been built because if it has to be changed or repackaged, you can be sure that it won´t be properly tested and mistakes happen. In the best case, it’s just a lot of time wasted, at the worst, it’ll blow up in production. It’s best to keep everything environment-dependent out of the application package and load it from a file system or JNDI. It’s still a good idea to keep configuration templates with the application.
2. Is the application self-contained, packaged and tested with database migrations and configuration scripts?
All of your fancy functionality won’t mean too much when it doesn’t work due to an out-of-sync database or misconfigured environment. However, in practice it’s way too common to only test functionality and let operations (yep, those guys again…go make friends!) figure out the rest by themselves–which is kind of like handing over a malfunctioning car to a mechanic but saying “Good luck!” while taking the key and maintenance history with you. Use a good database migration tool like Liquibase, and bundle/test the configuration scripts with the application.
3. Has the application been tested with more than a few concurrent users?
While your app runs fine in the demo room and on the development machine, it’s easy to forget that the application will be used hundreds or thousands of concurrent users. A basic stress test with JMeter or Gatling is easy to set up and will relieve some of that post-launch tension. After all, the public at large will end up stress testing your app anyway, so you might as well keep the subsequent meltdown behind closed doors, and not on public view.
4. Do you have a quick set of acceptance tests to run as part of the deployment?
If you’re doing it right, you probably already have acceptance tests ready. Why not run them on the production environment before making the changes visible to the users? It’s not a good idea to leave it all to the guys in Operations, since they won’t know the application (and aren’t expected to), and won’t test it as much when deploying.
5. Does the application expose key internal metrics?
If your application has issues in production, you want to hear it from Operations, not in the hateful tweets from users. Even better, they should tell you after they’ve restarted the problematic component, but you can help them do this by considering about the internal metrics they need to troubleshoot. Think transactions per second, queue sizes and stuck requests. Expose key metrics as an API or easy-to-parse logs, so they can hook it up to their monitoring stack and put the fire out quickly.
6. Is the application backwards and forwards compatible?
This is probably the most controversial one on the list. I’ve seen backwards compatibility a lot, as it enables you to do staged releases, with the old and the new version side by side. However, I would consider the other direction even more important–if you change the database in a way that is unusable by the old version, there’s no retreat. And you do want to retreat quickly when all hell breaks loose.
Unfortunately there are no shortcuts on this one. The best I’ve seen is to keep it in mind in development, test it throughout, and have a separate set of database migrations to make the database pretty once the new version has proven itself.
This 1-question poll will grade your organization
How many of these questions can you answer YES to? Leave your results here, it’s always interesting to see what people say?
Hopefully these are the questions you’re already asking before the release, or plan to include in near future. I’m anxious to see what everyone else has in mind before the big push to live. Share your own checklist in the comments or tweet me at @mirkoadari.
Want to learn more about developers and how they influence what happens to their apps in production? Check out our DevOps vs Traditional IT Ops report, where we get into that. Or download the PDF directly: