Build bridges, not walls
In this blog post, John Arundel of Bitfield Consulting describes why DevOps matters, and offers some practical advice on how your organization can apply its principles. John is a well-known Golang trainer and mentor, Kubernetes consultant, and Puppet expert and author. Say hello@bitfieldconsulting.com!
“When devs and ops collaborate, good stuff happens, so start breaking down the walls and coming out of your boxes. Don’t wait for the management memo. Just start today. ”
We live in a world of walls, unfortunately, and some people would like to build even more of them. Whatever you think about that, the walls between software developers and IT operations staff don't do anybody any favours.
Looking over the wall
If you're a developer, have you ever wondered why ops seem so antagonistic? Here's why: they're fed up with your buggy software that doesn't work in production, and your apparent lack of interest in fixing it.
If you're an ops person, how do you think you're seen by developers? The answer is, they think you're grumpy, unhelpful, resistant to trying new things, and unresponsive to requests for changes. Oh, and you're a real buzzkill about security.
I'm going to tell you something now which will shock you rigid. The fact is, those folk in the other team aren't idiots, and they don't hate you. They're smart, motivated, and professional, and they're focused on doing their jobs. But you're not making it any easier for them. Here are some ideas on how to change that.
“People who make music together cannot be enemies... at least while the music lasts.
”
Getting together
First, empathise. Understand a bit more about what your colleagues in the other team do, what they care about, and why it matters to them. Second, collaborate. When you work closely with someone, you get a great insight into what it's like to do their job.
Software developers, get more involved in how your stuff is deployed and run in production. "Throwing code over the wall" won't fly any more. Your ops friends will help you get a development environment that mirrors production. You can use Vagrant boxes or cloud instances built by the same automation that builds production. No more "it works on my laptop"; when something breaks in production, you will have an identical environment to troubleshoot it.
Ops professionals, you already write and maintain software that runs your infrastructure, so make sure you're using the same workflows and tools as your friends in development. Get them to do code reviews for you. Their entire working lives are focused on good software engineering practice; there's a lot you can learn from them.
When there's a new application or service to be deployed, involve the developers from day one. The software that configures the servers, installs the dependencies, and manages deployments, is as much part of the application as the source code itself.
Do pairing. The best way to collaborate with someone is to pair program with them (or pair sysadmin, depending on the task). That means you're both sitting at the same screen and keyboard, talking about what you're doing and working by consensus. You might be coding, troubleshooting a problem, or anything else that's part of your normal work: you're just doing it together. If a disagreement comes up about what to do, talk it out or take it to a whiteboard. Involve other people if you think they can help. If you don't have the information you need to solve something, find the person who does, and pair with them to solve it.
“I am a man of fixed and unbending principles, the first of which is to be flexible at all times.
”
Re-thinking your work
New or experimental projects often need a lot of flexibility. If IT can't offer this to developers, they'll have to go around IT to get the job done, and that doesn't spell collaboration. If developers need your help to get virtual machines running, make it so; if they need the ability to spin up cloud instances to test things, make sure they have it. Re-think your priorities as an operations engineer. It might seem like answering questions, helping people, and working with developers is taking time away from your real work. Guess again. That is your real work!
Developers, you might think your job ends with a git push. But software that doesn't work in the real world is a waste of bits. You need to understand where your code runs in production, how it gets there, how the servers are built, how the cloud provisioning works, what happens when your stuff breaks, and how to fix it. You might think learning about Linux command lines, TCP/IP, and network latency is a waste of your time. Actually, it's making you a better developer. If you think it's not your job to know this stuff, you misunderstand what your job is.
The truth is there was never a neat line between dev and ops. The overlap is precisely where things get interesting. Lots of important work simply can't be done without having a foot in both worlds, and the way to do that is for dev and ops to share their particular fu. If deploys are fragile and often result in unplanned downtime, work on that together. Building a safe, reliable, easy-to-use deployment system is right in the centre of the Venn diagram between dev and ops. If you get that right, much else will follow. If releases pass tests, but fail intermittently in production due to weird edge cases, you'll need to work together to debug that. If performance is a problem, it takes dev and ops collaboration to fix it.
“Show me a completely smooth operation and I’ll show you someone who’s covering mistakes. Real boats rock.
”
Closing the loop
Finally, monitoring connects it all together. Monitoring tells ops that the services are up, and it tells devs how the software is performing. Good automated monitoring checks don't just test that a webserver is responding: they match text strings that prove it's working; they fetch multiple URLs that exercise different parts of the system; they do queries which verify the whole stack. If the system uses login sessions, the monitoring checks log in and behave like users: searching, filling forms, uploading content. The developers know what needs to be tested, and the ops team know how to write checks that test it. Good monitoring demands empathy and collaboration from dev and ops, and it closes the loop between those who write the software and those who run the software.
"But we already write unit tests!" Great, but monitoring is different. Unit tests demonstrate that your code works in theory. Monitoring tells you whether it's working in practice. Unit tests show what happens in the failure modes you've thought of. The real world will throw you failure modes you won't believe. Unit tests are essential, monitoring is essential, but they're not the same.
When monitoring detects that a service is down, that alert needs to go to the person who wrote the service. This is one area where developers can be surprisingly resistant to change. Some people have got used to the idea that their responsibility ends once the code ships. But that's not the case. If software is breaking in production, fixing it needs to take priority over new features, and that means developers need to get that information directly: ops don't want to spend their time nagging devs about bugs, and it creates a tension between the teams which is unnecessary and unhelpful.
A good way to get developers interested and involved in operations is to set up a highly-visible dashboard screen, showing current system status and uptime. If all is well, the board is green. When there's a problem, that should be visible to all developers, and when there's an outage, developers should be getting paged. It's amazing how being on-call for your own stuff concentrates the mind... on fixing it.
“Outside ideas of right doing and wrong doing there is a field. I’ll meet you there.
”
Tearing down the wall
When devs and ops collaborate, good stuff happens, so start breaking down the walls and coming out of your boxes. The devs learn about how to deploy, run, and monitor services at scale. Ops learn good coding practices, the power of pair programming, and how to build software as a team. We all learn how to be better at our jobs, how to be less defensive, and how to be more empathetic. Don't wait for the management memo. Just start today.
