Not a real developer
From My Horrible Career
They said I threatened someone with a knife. That's an absolute lie. It was a screwdriver.
This is the second of a five-part series about my horrible career: how it started, how it’s going, and what I learned from making every mistake imaginable.
- My horrible career
- Not a real developer
- Master of my domain
- If you need the money, don’t take the job
- Will write for food
The full series is now available as a book: My Horrible Career
In the first part of my chat with the reliably on-point Zack Proser about my horrible career, we heard how a degree in computer science was the easy option for a coding-obsessed geek, but it didn’t translate into a lucrative software engineering job (well, that sucks).
Instead, I toiled in the word mines for a few years as a tech writer, cranking out user manuals warning people not to dunk their personal digital assistants in water. Over to Zack:
John, tech writing is important work, as we all know. But it must have seemed a little unsatisfying to someone who always really wanted to be a programmer. And you’d already been writing code for a long time at this point.
I had: I’d been a hobby programmer all my life, as well as mixing with lots of professionals over the years: that is, people who actually had “software engineer” on their business cards. In many cases, I had as much as two decades more programming experience than they did, which made it all the more frustrating that I couldn’t get anyone to pay me to do it.
There is a big gap between recreational programmers and professionals, to be sure. It’s not so much about the code you write as the way you write it: things like tests, version control, program design, abstractions, APIs, and so on.
I didn’t know any of that stuff back then, but of course I didn’t know what I didn’t know. You never do. All I knew was, I felt like a software engineer trapped in a sysadmin’s body, yearning to be free.
I can relate to that feeling. When I was working in technical marketing, somebody once described me as “not a real developer”, and that burns me to this day.
Gosh, that’s about the meanest thing I’ve ever heard, and I’m on Twitter. There’s no excuse for that kind of gatekeeping.
You should have given him some snappy comeback, like “Oh yeah? Well, the jerk store called. They’re running out of you!”
I wish I’d thought of that.
You will, Zack, you will.
Well, I still managed to learn a lot about programming, partly from going to the pub with my “real” software engineer co-workers. (Sidebar, they drank a lot. I didn’t know why, then, but I think I do now.)
Also, like most sysadmins and infrastructure people, whatever my business cards said, I was writing a ton of software, for users who depended on it.
So, for all those out there suffering from imposter syndrome because your company doesn’t call you a “real developer”, that’s bullshit. If you write software, you’re a software developer.
Thank you! That’s what I’m saying.
For example, I wrote an installer for the company’s main product. That wasn’t trivial, because it was a server-based thing, and it involved checking for things like “do we have the libraries we need, and can we install them,” and “which version of the product works with this operating system,” and so on.
Well, that company eventually tanked (not because of my code, as far as I know). But now that I had some proper developer experience on my CV, I managed to get a job as a proper developer (using Python, as it happened). It was great. I felt like Pinocchio becoming a real boy.
It helped that the company were trying to port their product to Unix, and I was the only applicant who knew what an inode was, or how to fsck a filesystem. Good times!
There’s a bit of a Unix theme emerging, no?
Right, and that knowledge was acquired in an extra-curricular way. I wouldn’t have learned it if I’d stuck to the textbook. When the other kids were learning Pascal, I was the smart-alec who wanted to do the assignment in C, and found a battered old K&R book at the back of some cupboard, like the Half-Blood Prince’s copy of “Advanced Potion Making” by Libatius Borage.
You can’t learn C without learning something about Unix, and vice versa—the two things are just so entwined. That’s true of Go as well, by the way. It has a lot of Unix DNA (and is clearly the spiritual successor of C, to the considerable annoyance of lots of people who don’t like C).
What is the Unix gene, then, would you say?
Richard Gabriel coined the phrase “worse is better”, which sums it up well. Simplicity is paramount. Simple systems are easy to write, easy to port, easy to use, and easy to extend. Simplicity beats even strict correctness, in this view: it’s better to be simple (and handle the easy 90% of cases in a nice way) than to be totally correct (and handle the awkward edge cases, at the expense of making the code much more complex).
That sounds familiar to Go programmers, right? Go is almost brutally pragmatic. It’s not a beautiful language at all, at least on a surface level. Programming language theorists hate it! It may be fine in practice, they say, but it’ll never work in theory.
Go is designed to be easy to parse, quick to compile, and simple to implement, just like Unix itself. It’s not aimed at programmers who want to express themselves in rich and byzantine hierarchies of type systems, or who need the compiler to catch all possible bugs. It’s just a plain, non-fancy language for getting useful programs written and running as quickly as possible. I appreciate that about it.
So you were a Unix sysadmin, slash Perl & Python developer, slash network cable plugger-inner. Where did that take you next?
I found myself in a tricky spot. I enjoyed the actual coding, but it wasn’t a very good company, and I didn’t gel with my co-workers. I thought their way of doing software development was tragically crap, and conversely, they didn’t value any of the things I thought I was bringing to the party.
Like what?
Tests, automation, deployment, source code management. Software quality in general. All the stuff I’d learned from decades of hanging around with “real” professional developers.
It didn’t go well. They didn’t see what would be in it for them. When I explained it would result in a better, more reliable product for their customers, the response was “Why would we care about that? We already have their money.”
So it wasn’t really in the stars, type of thing. Eventually the company and I came to what’s usually described as a “mutual decision to part ways”, or less euphemistically, I was canned. [They said I threatened someone with a knife. That’s an absolute lie. It was a screwdriver.]
I know this next part. You became an internationally-recognised consultant, expert on infrastructure automation, and author of such books as The Puppet Cookbook. Right?
No, I was unemployed for a year, got dropped from the books of every recruiter in England, and finally ended up taking a desperation job working nights in a data centre in East London, plugging in network cables.
Okay, it wasn’t always nights (it just felt that way, as you’ll know if you’ve worked nights yourself), and it wasn’t a completely unskilled job either. There was a little light sysadmin involved, but it was a massive, global company. I was the smallest imaginable cog in a very big and impersonal machine.
It was very like being in the military, I suppose: you’re not there because you’re smart or have good ideas, or even a particular set of skills. You’re there to salute officers, follow their orders without question, and most importantly of all, to keep your boots shiny at all times.
In that environment, having good ideas is actually a bad idea. Suggesting things could be changed implies there’s something wrong with the status quo; a Catch-22 I encountered again later when I became a consultant. You know, there’s the right way, the wrong way, and the Army way. In that firm it was very much the Army way, or, more accurately, the Navy.
So it was an often frustrating experience. And I still wasn’t a real developer.
Huh. Didn’t see you as the saluting type, to be perfectly honest.
That’s more or less what I thought, too. That kind of highly structured environment suits some people, I’m sure, but I’m not one of them. Needless to say, then, I wasn’t a huge success in this job either, though the trench-warfare setting did bring us all very close as co-workers. When you’ve fought alongside someone under intense enemy fire, doing a tricky disk replacement at 5am on a control server for a nuclear power station, you form a bond with that person.
I did learn a lot about big infrastructure, and automation too. Obviously an operation of that size has to be almost entirely automated. When you just have a few servers or clusters, you can manage them more or less by hand, with a few basic scripts. When you have a few thousand servers, it’s essential to automate monitoring, backups, installs, upgrades, and everything else. And economies of scale make that kind of investment worthwhile.
So that’s when you became a successful consultant and internet personality?
No, it’s when I nearly died of malnutrition from living on vending machine food and unbranded instant coffee. Plus the sleep deprivation.
One thing about nights, though, is that not a lot happens, so there’s plenty of time to fill. We used to watch movies. (One guy used to watch pornos, so I was always careful to give the chair a bit of a wipe down when I took over his station.)
Or we would play games, read websites, chat with people in other timezones, grab a few minutes of poor-quality sleep in a chair (this is physiologically impossible, as anyone who has tried it will know). I did a lot of hobby programming.
And you did a little moonlighting, perhaps?
Someone I’d worked with in a previous company got in touch to ask for some sysadmin help, on a contract basis, and I thought “Why not?” After all, I had plenty of free time at nights: essentially, we were just there to weigh down the desks and stop the AC from blowing them away.
So I would do the contract work remotely at nights, while nominally on shift for my real job (sorry, supervisor), and visit my client’s site on weekends or off days to do things like plugging in network cables.
Is that when you decided to take the leap and strike out on your own, then?
No, because I didn’t get any other contract work after that. But when my actual employer and I came to the inevitable parting of the ways (I had my own version of Calvin’s “noodle incident”), I got a job in another company that was a lot nicer to work for (and no night shifts).
For various reasons, most of them to do with evading taxes, everybody at this firm was technically a contractor, not on salary. And, again for reasons specific to UK tax regulations, we were encouraged to take on other contracts at the same time, so that we wouldn’t look quite so much like de facto employees.
I already had my limited company in place—not that you need one to do freelance work, but it’s nice to have—so I reached out to a few contacts, and little bits of work started to come in here and there. A friend and co-worker (who’s now an extremely Distinguished Engineer at a well-known company, but I knew him before he was quite so distinguished) did me what turned out to be a very consequential favour indeed, by introducing me to Puppet.
What’s that?
In the bad old days, managing configuration on servers and networks was anarchy, but anarchy made of a lot of largely non-portable and incorrect shell scripts. Every company just cobbled together some bespoke tools (one was literally called Cobbler) and did their best.
The problem is, you can build a machine with a shell script, but then what? If you make manual changes to the server, and then forget about them, there’s no automated way to revert them.
And you can’t easily roll out a single change to a big farm of heterogeneous machines, all with different operating systems and software versions, in a safe and repeatable way. That’s the problem Puppet solved so well, by turning configuration into software: infrastructure as code, to coin a phrase.
I thought that was called “Ansible”.
Never heard of it. Anyway, managing infrastructure has always been about writing and maintaining software of various kinds, as we discussed, but now there was an actual programming language specifically tailored to the job. What you’d need to get the most out of it would be someone who knew about Unix, servers, and networks, but also how to write good software.
All of a sudden, then, there was a me-shaped niche in the industry.
So that’s when you went fully independent?
No, that happened later, when I got fired… again.
I can see why you didn’t mention any of this in your careers book. Shall we pause there, and pick up the story of what you did next, next time?
Sure. We’ll talk about how I became self-unemployed, and what I learned about running my own business, by my standard method of making every possible horrible mistake and suffering the consequences.
Sounds good, my man. Catch you on the flippity flop!
I have no idea what you just said.
Next: Master of my domain
And you can read the whole series right now in this charming, fun-size ebook: My Horrible Career
