I’ve been doing more than my fair share of deployments lately.
These deployments aren’t the one-click-then-beer variety you find on Heroku or in some seriously agile shops. These deployments are arduous day-long journeys, caravanning eight unique technologies, traversing multiple connected systems both up- and down-stream. And bad things do live in the water.
Seriously, it has been a lot of hard work and some difficult times reliving past mistakes. So here’s my list of freshened lessons that make big deployments suck less:
1. One group chat and one conference line. Don’t use email for the deployment team. It’s slow. terrible at threading conversations. And, somehow, people always get dropped off the address list – or appended to the cc field 37 times. A dozen individual chats and phone calls are terrible for anything but proving why Europe mothballed the semaphore back in the 19th century.
2. A deploy with fewer people is better. An all-hands-on-deck deploy may sound comforting. In reality. that’s just more people in the way, arguing over the best corrective path, and slowly burning out while standing at the ready. Deploys with fewer people – with clear responsibilities and fully prepared for the tasks of the day – beat mob deploys any day of the week.
3. Have your support people at the ready. Just because your core deploy team is fully briefed and prepared doesn’t mean you have all the expertise you’ll ever need. Know who else you’ll need if the fit hits the shan. Make sure that each member of this auxiliary group is ready to drop in at a phone call’s notice. Collect and distribute the best contact information for that auxiliary group to each member of your core group prior to deploy day.
4. Clear communication channel. Who owns outward communication and in what direction? People who need deploy status updates could include stakeholders, upper management, cooperate support teams, and auxiliary team members. They will not all want or need the same status reports. (Hint: you don’t want to send a 100-line deploy plan every hour to your stakeholders.) You need to decide in advance who owns which channel, the frequency of status updates, and the content of those updates.
5. Clear decision-making. When your app server and the database stop talking the last thing you need is a team that descends into bickering over which server to kick first. A deploy will involve a variety of experts, but there should be a single person in charge who – once all voices have been heard – calls the shots.
6. Clear accountability. The entire deploy team should know who is responsible for what well before the deploy begins. For me, that means one owner for each discrete part of the deploy. This separation may be by system, technology or product subteam – whatever makes sense for the deploy. Finally, only core deploy team members own deploy tasks. Sometimes a non-team-member is responsible for completing a task, but we still want a core deploy team member accountable to see that it gets done.
No Comments »
Posted in Thoughts
We have so many things to fret over when building software:
Requirements. Quality. Deadlines. Budgets. Scope creep.
Dependencies. Audits. Deployment plans. Disaster recovery.
Single points of failure.
Other systems. Other teams. Other everything.
Backouts. Blackouts. Timeouts. Burnout.
On call. Fire call. Five nines. Server down.
Bad managers. Weak programmers. Ill-suited customers.
The random bus.
Perhaps we hold fast to the long-term course. Fortunate, still, our leaders flare near-terms into the night. Even so our endeavor rolls on as involuntary peregrination.
Though it never was how can we return to the way it should be? Create and inspire. Band together for common purpose and tantalizing gain. Strive and fall and lift one-anther higher. Set collective effort on uncommon aspiration. How do we forget to remember? To make something awesome.
No Comments »
I’ve seen it happen several times. The business requests an upfront, full-cost estimate for some feature or functionality. A business person writes something up – far simpler than the normal story-writing or requirements-drafting process. The development team reviews the write-up. Questions, clarifications and adjustments happen. An estimate is assembled and then KA-BLAM! – everyone is blown over by the sheer size of it.
When even the development team is left scratching its head saying: “Yeah, that looks way big to us too, but … ” you know there are problems bigger than whatever comes after that ellipses.
Granted, there are mitigating circumstances. The requirements – such as they are – may delve into a new and unknown domain. The technology may be novel to both the team and the corporate IT ecosystem. But even in these cases, if the team cannot mount a strong justification for an eye-popping estimate, you have to ask whether there are other issues lurking in the shadows.
In these cases, the two most common issues I’ve set alight are:
1. The delivery team lacks confidence in its own technical ability (due to a skills mismatch, lack of training, or other reasons). This issue goes beyond the time it might take to ramp up on new technologies, adjust to a new domain, or proof a novel design approach. Teams can readily explain all this in their estimates. But an intractable problem arises – with a huge estimate in tow – when the development team fears that it’ll never fully tackle a new technology, domain, or approach. Or, sometimes, it’s an unconquered legacy technology, domain, or approach frustrating future development.
2. The delivery teams lacks confidence in its business partner’s ability to honestly collaborate and openly negotiate with the team. It’s okay if the business partner doesn’t get the requirements solidified on the first, the second or third try. Agile teams know this can be difficult, and agile processes are geared to accommodate. However, when a customer repeatedly demands prompt and accurate delivery on half-baked requirements, even an agile team will take a tanker-truck of spay foam to the delivery estimate.
While the causes are dramatically different, the end result for each issue is the same. The team’s huge estimate reflects an inability to adjust and adapt. In one case the team lacks confidence in its own ability; in the other case the team lacks confidence in its customer. Regardless of the cause, these are issues that need to be addressed, as they impede far more than the team’s ability to deliver an estimate.
Sometimes a huge estimate is just a huge estimate. Sometimes it’s something much more. Next time you see a jaw-dropper of an estimate wrapped in flimsy explanation, it might be worth taking the time to find out why.
No Comments »
The “Maps” application on my iPhone used to work great. It pulled up the number of a nearby joint so fast I never bothered to save those numbers to my contacts list.
Then, about a month back, something annoying happened. I’d search for a local joint – say Gino’s East, to order pizza – and my normal search result appeared different.
While I noticed a difference – my restaurant listing is now a “sponsored link” – I didn’t think much of it. Then the call was answered by GrubHub and not Gino’s East. I hung up.
After further inspection, I discovered my around-the-corner pizzeria is now the unassuming red pin next to the big glaring advertisement.
The math here is pretty obvious – had I completed the call. GrubHub takes a cut from my Pizzeria. Google takes ad dollars from GrubHub. Apple, I’m sure, gets some form of compensation.
Now, let’s talk unintended consequences.
What happens if I succumb to mobile advertising Nirvana and click the default GrubHub ad on every search? I was going to order takeout from my Pizzeria anyway. Similarly, I was going to make a reservation at La Gondola on Saturday regardless of a GrubHub ad. The upshot here is that either my local restaurant takes less profit or raises its prices – one of us ultimately pays for it.
Conversely, what happens if I evade the GrubHub ad? I can – and do – click around the ad to ring my local joint directly. It’s only one extra tap. But it’s a precise tap – while cradling a baby, while walking down the sidewalk, while hanging for dear life inside a CTA train.
Perhaps the precedent is more important. I consider “Maps” an application that I paid for when I bought my iPhone. It came preinstalled. I couldn’t remove it if I wanted to. And the phone certainly wasn’t free.
It’s only one extra tap, but this is the first time I’ve been annoyed by – or ever seen – an ad in a preinstalled application on ANY phone I’ve owned. And this isn’t just any advertising. This advertising affects how I use the phone. Personally, I will pay to avoid being assaulted by advertising. That’s one reason I don’t have an Android phone. That’s the major reason I’m selective about my use of Google.
What’s the upshot?
For me, I see a behavior change. One by one, I’ll add local shops and restaurants to my contact list. Hopefully it’ll take Google a couple more years to find its way there.
For mobile usability, this is a loss, as the extra ad dollar is clearly more important than the paying customer – even on an iPhone.
For my local joints, it’s one more national player trying to take a bite out of their pie. That’s also bad for me.
No Comments »
Most project managers want to track actual effort and dollars toward the completion of their projects and deliverables. The goal is obvious and laudable. By knowing the actual cost of something we can provide more precise estimates the next time around.
However, I don’t find actuals useful. I have little faith in the quality of the data that gets collected. Low-quality data is, at best, not useful. At worst, it will lead you to wrong conclusions and bad decisions.
Four issues seriously degrade the reliability of actuals.
- We, as individuals, remember pain. And we are much more inclined to remember when we took 100 percent longer to complete a one-day task than when we went 50 percent under on a two day task.
- Professionals prefer producing results over pushing paperwork. Sometimes we don’t put any mental cycles to tracking and remembering our effort against discrete activities. When the time come to push out the paperwork we can do no more than look back at our estimates.
- Sometimes we don’t work in sequence. Tasks get entangled. Priorities shift. We deviate from our plan to help other team members. The work gets done, but there’s no accounting for where the effort went. Again, our only fallback is our estimates.
- There is an expectation that each of us contribute a minimum number of hours per week, regardless of whether one is an employee, consultant or contractor. Will there be a problem if my actuals don’t add up to 40 hours (or more) a week? What about meetings, email, chores and helping other teams — where do those things go? No matter what reassurances you provide, too many of us will ensure that our weekly actuals total to the company work week, regardless of our actual effort toward shippable product.
Is it really worth spending your team’s limited resources on this?
5 Comments »
I’ve always been puzzled why Kanban attracts so much attention in the agile community. At its essence Kanban is workqueue on a wall. Why would any team, delivering within a healthy iterative cadence, switch to a method that does nothing to support a sustainable pace for team members nor provide real delivery planning and predictability to the business?
During a conversation with Rebecca Porterfield, of Solstice Consulting, I realized that I have been looking at Kanban from the wrong perspective. Rebecca has recently rolled out Kanban with a client that had no existing process. The client’s team fell in love with Kanban instantly. It was a welcome relief to the interrupt-driven, everything-is-a-priority world to which the client was accustomed.
This conversation made me realize that Kanban is ideal for teams that have yet to take the agile plunge. Kanban is a quick hit of simple rules that fills a process void. It appeals especially to those who dwell in process-free (or chaotic) environments.
Kanban can be mundane and underpowered for the experienced agilist. It doesn’t provide me with much assistance on my current 30 person, multiple-system, heavy-process-mandated, regulation-driven, do-or-die project.
For the uninitiated, however, Kanban is a grand welcome into the agile community. Many teams (although not all) can invest a little effort in Kanban and reap significant reward. Then the teams get comfortable and even excited about adopting additional agile practices. This makes Kanban the ideal gateway drug to agile.
2 Comments »