What if someone told you that your job could be fun again? What if they said you could learn at the rate you did when you were green? What if they told you that you could reduce your process related documentation and red tape by massive factors and that you could deliver your product faster and of higher quality? What if they told you that you could write code twice as fast with half the keyboards that you currently use on your team? And if they said you would be able to massively redesign systems with ease after all these gains in delivery were realized? Now what if they claimed that deployment, configuration and testing were easier? All this sounds like a pipe dream. It sounds like someone trying to sell something to you. Actually, they (we) are just trying to sell you on something. And its free.
There is plenty of literature on agile methodologies. I am just getting started digesting them. One of my favorite articles, The New Methodology by Martin Fowler, is a great place to start. It lays the ground work for why we need to consider a different approach. From there you can just about launch in a million directions. He does a good job of describing some of the various flavors of Agile.
Why? What is the magic of Agile? Its small, controlled movements. Its like snowboarding. If you want to be sure you won't fall down, you might visit the mountain in the summer to survey the landscape (early business requirements gathering and analysis). You'd make a map of the trails and get some measurements for how steep the inclines are and what trails are most likely to provide the safest, most enjoyable decent. Then you'd return a few months before your first run. You'd talk about how you plan to go down the hill with experts and interested parties. Everyone would evaluate the plan. You'd have to compile a document with definitions and reference for those who aren't intimately familiar with snowboarding. This would be when you'd provide your functional analysis and high level design document. Pundits would make their critique and help revise the document so their concerns were met. Once you'd gotten sign off, you might actually walk some of the trails to prove your early estimations and expectations. Once you had a very clear idea of which trail you'd be taking, how fast, what kind of board, at what time of day and under what specific weather conditions (detailed design document) and had it signed off by the experts and decision makers, you'd ride the lift to the chosen trail, jump off, and engage the mountain exactly as you'd planned to do. Chances are you'd find some variance from your evaluation. New equipment might have become popular but you hadn't included it in your spec so you aren't at liberty to use it. You might even realize you have no idea how to snowboard and hurt yourself really, really bad. You say, "I know exactly what I am doing!" If you knew exactly, you'd not need to evaluate and plan. You'd just pick up the time cards from the last time you did it and be right on the money. Regardless, in the event of disaster, as long as you adhered to your document then you could reference the detailed specifications that were signed off upon and you would not be responsible for your folly. You'd still be in massive pain but it would be someone else's fault.
Now consider how life really works in an unpredictable environment. You'd strap on the board that the most knowledgeable expert you could find recommends. You'd ride the chair lift to the bunny hill and you'd watch the six year olds tearing up the trail. You'd fall. You'd fall alot. However, they would be slow, controlled falls. Each small stumble teaches you what to do next. If you're smart, you'll get the knowledge you need to avoid the common mistakes that can be prevented with acquired skills and appropriate planning. For example, you might enlist the assistance of a more experience boarder. She would have to slow down a little while you work your way through the basics but she'd learn in the process of teaching you. She'd identify quickly the things that would provide the quickest gains and help avoid the bad habits you might develop without advice. After a short time, and with a few reviews of your technique by experts and sponsors, you'd move up to the intermediate hills. At this point you have some stable experience and technique to leverage and you can build up quickly. At any point that anyone realizes you have taken the wrong path and are in trouble, you can quickly adjust to recover safety and security. After a while, you'll be showing other newbies how to take on the powder. Every step of this learning and growing process happens to be fun as well.
Its a silly analogy but it has merit. Planning is important but planning is not the end goal. The end goal is results. Planning should be specifically focused to accomplish results. Process should get you to the desired end point, not provide a scape goat should things not come out smelling of roses.
In my professional world, I find a unit test is like a helmet. I don't plan to hit a tree but if I zig when I should have zagged or some other skier clips my toes and I head butt a tall pine, my unit test will protect my product from a costly concussion. Pair programming is like taking a run that's a little beyond your means with someone who could carve it like a thanksgiving turkey. You're mere presence keeps the hotshot cautious and attentive while their expertise is training in process. Both of you come out less bruised and having had more fun. When its software, there are fewer bugs and better code structure overall. Lastly, having frequent delivery and reflective review of both product and process likens to showing off your mad skills to your peers and constituents as you get better and better.