The pious and the blind love nothing more than bemoaning a lack of proactivity after a disaster. It is they who are usually the reason you are not.
We’ve all been in the dreaded post-mortem meeting. The meeting where a (usually) pompous Director solemnly asks “how can we ensure this never happens again?”. Middle management jumps in to give a Buzzfeed-style Top 13 Things We Can Do. Because leadership. Outspoken technical architects chime in with solutions that would make Rube Goldberg proud. The up-and-comers know a simple switch in a config file that would fix 80% of the problem, but 80% is never good enough in the “never happens again” scenario, so it is quickly dismissed if anyone brings it up. Meanwhile, half the room is thinking about the latest crisis that is brewing.
Flash back several months to the person who brought up the current disaster as something that is a very real possibility and some ideas on how to avoid it. Chances are, middle management shot it down because it wasn’t a problem then. There are more pressing issues requiring time today to think about a possible, albeit huge, potential problem somewhere in the future.
I’ve been on both sides of this equation, and I have come to one inescapable conclusion. Fuck proactive.
Management Loves the Diving Catch
Management loves it because it is so incredibly easy to show the value to the effort they (you) put in. “If we hadn’t upgraded to Windows Server 2008 just last week, we could have been hit with the ClusterFuck virus that hit the IT press yesterday! Here, have this promotional ClusterFuck virus t-shirt, because all major viruses today have marketing”. It is much more difficult to describe spending time on a comprehensive OS upgrade and patch process. “It’s working now, and installing patches could cause the application it’s serving to break. We should change as little as possible to maximize uptime.” Cha-ching: ring up some more technical debt.
Think about who in your organization is most visible and/or most likely the recipient of the greatest financial rewards for their work. It’s usually the ones who convince the people above them that they are the thin line between the org and disaster. They are the ones who grandstand at every opportunity. The ones who make others who are probably quietly doing most of the heavy lifting roll their eyes. But do you know why management loves them? They give them what they want — a story about a crisis that IT has solved under their leadership. They create stories that management can easily translate to business leaders. And this brings us to the single biggest problem that IT has today: explaining its non-crisis value to the business units it serves.
Why Do We Pay You So Much?
Here’s a fun conundrum: “We don’t see IT doing a lot really. IT must therefore be really easy. Why do we pay you guys so much?” Or how about “Everything is broken all the time and you guys can’t ever seem to make it work. Why do we pay you guys so much?”. The fundamental problem is that most of the consumers of IT services have no way of appreciating the proactive work you did to prevent a problem because in their minds, that’s just your job. IT managers at virtually every company that is not a startup have a hard time explaining why what they do is important and has a direct impact on the business’ ability to get their jobs done. IT struggles to explain its value while the business leaders steadfastly refuse even to try to understand. Hell, most of the time, upper management has a hard time understanding how to connect to a printer let alone understand the value of deploying a CM system like Puppet. Here’s how that conversation goes:
CFO: You want $500k for support for something called “Puppet”? What does it do?
Director: Puppet will allow us to automate the deployment of new systems as well as to manage the configuration of our existing fleet. We’ll take it from days to minutes to deploy new systems. Patch management will become simpler. We will be able to treat clusters of nodes similarly and better catch one-off configuration problems. We’ll be able to better manage overall risk compliance too.
CFO: So you won’t need as many staff to manage all those servers then?
Director: Well, uhh. Technically that is correct, but we can redeploy our already scarce resources towards the development of new and innovative solutions rather than spending the majority of our time on break/fix.
CFO: You mean finding more tools like this Puppet thing that cost $500k a pop?
Director: Umm. Not all of our development will involve paid support. Actually, puppet has an open source version and we could deploy that, but the support contract will help us with training and give us the ability to get support questions answered quickly.
CFO: So this open source tool has no documentation unless you pay for support?
Director: No, there is community documentation, but…
CFO: Marvelous then. OK, you can either install the open source version and read the documentation yourself or I’ll approve the $500k and you have to let go two people.
Director: <choking> I suppose we’ll go the open source route.
And just like that, you’re totally on your own. The project dies a little each day because no one has the time to get to it because they are busy dealing with the tyranny of the urgent.
The Tyranny of the Urgent
‘What is important is seldom urgent and what is urgent is seldom important.’
-Dwight D. Eisenhower
Every day, each of us makes conscious decisions about which balls we’re going to let drop. There are never enough people to do everything that “the business” wants. And there are customers who are quite adept at making their voice heard above the din, regardless of how important what they think they need really is. Stephen Covey made a mint off of The Seven Habits of Highly Effective People. Why? Because people have a hard time prioritizing. They particularly have a hard time prioritizing when they have people literally screaming at them that if they don’t get this done, the company is going to go bankrupt.
You get emails marked with that annoying exclamation mark for urgent. You get emails with a subject line that say “URGENT”. People CC’ing directors, VPs, and God Almighty, making their problem your problem. These people are too busy and important to put in a proper ticket and instead conduct a ground war in your email box. The problem is simply too urgent to give you enough information to solve it. Just fix it!
And, since we all know that production comes before development if you are in IT, that’s your day. That proactive project you’ve been trying to get to? Gone by the wayside, thanks to the tyranny of the urgent. The irony almost always is that if you could work on that strategic project, either the problem(s) at hand wouldn’t have happened or they could be fixed much more efficiently.
And thus you create the kludges which are the glue that holds your environment together. Most IT environments are grand ponzi schemes just waiting for a single node or script to error out to cause a system-wide failure because of the tyranny of the urgent.
Appeals to management to reprioritize your time are met either with indifference or sympathy, but rarely with approval.
The Conclusion? Fuck Proactive
That’s right. Embrace the inevitable. Fuck it. Under no circumstances try to be proactive if you find yourself in this situation. They key is to present your solution as something that solves a burning, seething cauldron of fail that is already happening.
Pull out some graphs that show bad things (in red, of course) going up and to the right and good things (in green) going down and to the right. Paint the picture that the environment isn’t “about” to blow up, it IS blowing up right now. Subtly poke your most vocal users and lead them down the garden path that there is a problem that they didn’t know they were experiencing. The key is for them to think that this is their idea. That you let slip some IT secret you shouldn’t have and that THEY are the ones who have found out about it. Then sit back, grab some popcorn and watch them absolutely go to town on your management. Once you have “business buy-in” that there’s a problem, just try and stop a manager from fixing it. Most IT managers got where they are by being “fix it” people. It’s their native tongue, and when provided a problem an internal customer thinks is burning, they’ll move heaven and earth to fix it. They’ll also take credit for it most likely, but baby steps here, baby steps. Right now, we’re busy tricking customers and management into thinking that a strategic problem is really a tactical one that must be solved now.
While everyone wants you to be proactive, very few are willing to make the investments required for that to become a reality. Trying to be overtly proactive will usually get you nowhere. You must instead embrace the reality that managers tend to allocate more resources to keeping customers from screaming today than keeping them from screaming weeks or months from now.