I'm Production Support... which is why I'm up at 4am.

As I mentioned in my first post I do Production Support for a large company. Unlike "Developer" or "Project Manager" - both widely used and understood titles - this probably needs some explanation. In our case "Production Support" is the team that's called when production applications have problems. I'm one of several "Incident Managers" who manage the incident by engaging dedicated production support resources, other enterprise support teams, application development teams or whoever else is needed to restore service.

My team supports nearly 400 custom applications - both mainframe and distributed. The mainframe is most likely similar to any other mainframe: an illogical progression of code over nearly three decades with tendrils into almost every moving part of the company. In theory we've "standardized" on either AIX/WebSphere or Windows/Dot Net on the distributed side. But that was only recently and of course no corporate money goes to rewriting working applications. So in practice we've got an unholy soup of engines, languages and protocols. If it's ever said "Hello World" it's probably in an application we support.

So my day consists of working up processes, investigating prior incidents and keeping my fingers crossed that things will hold together just long enough for me to get just a little bit ahead. They never do. When incidents arise they may be fast and easy or slow and hard. The easy ones might be sent to another team, have known solutions or not severe enough for major escalation. The hard ones... they can take many forms. There's the "happens every other day for months" or the "must be fixed before we sleep" and even the rare "Holy Shit, if this doesn't get fixed we'll lose millions!"

Incidents often take hundreds of man hours to fully solve - 10 to 30 people on conference calls for literally days on end. Much of the actual boot-on-the-ground work is done by a large offshore team (a mixed blessing at best and something I'll be talking about frequently) and across many offices here in the States. Conference calls backed by instant messaging are the only real option for collaboration and more often than not English is a second language making communication a non-trivial issue.

The work is hard the hours are long and I'm positive that my situation is far from unusual.