Software Development Live Cycle

There's a lot of variants of a Software Development Live Cycle. And the methodology has changed over the years... but mostly that about terminology or details, not the basics which seem to remain fairly constant, because physics, human nature, resource management, and the concepts behind writing software itself don't really change. So while companies (and some people) like to claim they're adherents to some methodology, if they are, they're idiots. The truth is every methodology adapts to the team/company, or that reflects the managements lack of a clue (or flexibility).

Some people have been asking about the Development Process, how it works, and the different stages. Being that I'm an Engineer, and have been around a long time, I'm completely jaded, and more than just a smidgen sarcastic and cynical; it comes with the territory. It is also a requirement of the job (engineer) to believe that you can do everyone else's jobs better than they can and that the world is full of boneheads (not because they can't think, but because most of them choose not to). So with that in mind, here's my snide views on how the whole engineering process and how the Software Development Live Cycle works.

Remember, there are dozens of different methodologies: waterfall, iterative, extreme programming, agile, and so on. Those are processes on implementation, but the basics are still all the same.

Requirements - Analysis

First, you do the requirements. In theory, the company (or specifically the marketing department or the business analysts) gather information and do the initial requirements -- then work with engineering to flesh them out and design things that are practical and can deliver the best results in the least time. This sets goals for the design; the embodiment of the requirements nailed down to specifics. This is also called the "conception phase" probably because someone is on the giving end, and someone else on the receiving end and someone (the engineering department) walks away feeling screwed.

In practice, it takes too much time to communicate with engineering and so marketing or analysts usually sit around drinking and shmoozing and put everything into a list that a bunch of half-informed but creative lushes can think of in their inebriated state. They are amazingly imaginative when liquored up. Their ideas are not based on what is practical, in fact, I wonder if any of them have heard of that word, or of words like "tradeoff", "time" or "work".

They are paid to create or sell, so they create the requirements based on what would sell, or would have more features than the competition. No regard is given to what is possible to create. Requirements that come out are nothing vaguely related to goals that could be met, and instead often have things like "pure matter to energy conversion for under $5.00, a fastest computing device in existence, fits in a shirt pocket, and doubles as a toaster". The engineers have this annoying habit of arguing with those requirements with petty little complaints like "that can't be done in this millennium", "if I don't know the meaning of life, how can I code the answer into this program" or "why the hell does a fork have to survive unassisted reentry into the atmosphere from a high earth orbit?" This is known as obstructionism so engineers are usually labeled early on as troublemakers, whiners, and "not team players".

Depending on the quality of the analytical, marketing and business staff, it can take the engineering staff somewhere between a few days to a few years to actually convince the "higher-ups" in the company of the facts about what is producible. The end results are almost always that the analysis/requirements process takes longer than expected, say 80% of the time originally allotted to the entire project. That doesn't mean that the schedule will slip, it just means engineers have more to do in less time. This helps the project start off on the right foot (by being way behind, to begin with) and management calls this "motivation".

Since requirements take so long, no one can afford to wait for them. Many products progress without any functional requirements until right before shipping. Not having the requirements will almost always cost more time than having requirements -- it is just that you push all the time from the requirements/collection phase of the project into all the other phases. Whether you have no requirements or bad ones it still results in the same conclusion -- Engineering has more work to do. In case you didn't notice, there are little direct costs to those that create the requirements for doing them poorly -- and ironically, there are actually rewarded instead (since they get to watch everyone else turn into cannibalistic feral wolves fighting over what they thought something meant). So the few people in companies that have actually seen requirements before, often don't have good memories about them -- and opt to go on without them. Accordingly, many companies don't do any (real) requirements so that everyone is shooting in the dark (usually at each other) while having some nice corporate mission like "designing the highest quality product possible". Which they are actually doing -- it is just that they are each designing a different "highest quality" product -- which they will only figure out (and go to war over) at some later date (usually right before shipping).

When they get to the phase where they realize that everyone has designed and implemented a different product -- each person masterfully points the fickle finger of blame at everyone else. Somehow, when this starts going on, marketing and business analysts just know (psychically) to be out of the building (and preferably out of the country), probably at some bar or on a golf course with the CEO, badmouthing "those troublesome engineers".

This process explains the rich and diverse feature set of products like Microsoft Office -- where you want to go over there and slap some product managers around and ask, "Who thought a dancing little wisenheimer cartoon 'assistant' was a good idea or had anything to do with word processing?" I mean if I really wanted some sarcastic heckling assistant telling me that I was doing everything wrong I'd just ask my mother to help me write an article.

Design (Definition)

The next part of the process is the design or definition phase. This is where you take the ethereal requirements and try to stuff them into something that is actually doable -- not unlike trying to stuff the entire smell of a city sewage treatment plant into a single two-liter coke bottle. The theory is that the engineers will go into a room, calmly discuss how the "requirements" will be implemented, and come out a few minutes later with nicely typed and clearly defined design that will let everyone on the team (now and in the future) know exactly what they are supposed to be doing for the next 1 to 10 years. It will also discuss all parts of a product, how they interact, where potential pitfalls might be (with solutions) and solutions for curing the problems of world hunger.

Usually, the process is somewhat less harmonious and takes a lot longer than expected. In fact, this usually this becomes the scenes of turf guarding and tantrums the likes of which are seldom seen outside of a nursery school play room. A few quiet engineers are in the corners eating paste, napping, or soiling themselves -- while most of the rest are having simultaneous petty squabbles over things that may have nothing at all to do with the project. The cries, whines, hair pulling and occasionally biting are all allowed in order to achieve your objective -- that objective seems to be to make the others on your team hate you and swear to their gods that they will pay you back for your tyrannically controlling demonstrations in the design meetings. Management calls this team building.

Surprisingly something vaguely resembling a plan can come out of this process. There is only a slim chance that the design will have anything to do with the requirements -- but engineers have come to accept this. (Along with the fact that marketing, management, sales, and business teams are all morons -- and in response, everyone else in the company thinks that engineers are arrogant, know-it-all, prima donnas.) Fortunately management, marketing, sales, and analysis (business) often have very short memories and don't notice the design has no relation to their requirements -- probably because of alcohol-pickled synapses, or because they are so tired of engineering's long unintelligible explanations to simple questions that they just ignore them altogether. Of course, there are the paranoids that think they let it ride because they just know that they will pay to engineer back (via blame) when the product is shipped late.

Now designs come in two flavors. Too specific or too vague.

The "too specific" designs take about 300% of the total time allotted to the product -- and try to tell everyone how to do everything, and they never work because someone will inevitably forget something critical like a power cord or open file command. But everyone will be so focused on their narrow little piece of the pie that they will forget all about it until final testing. This is how Government, Aerospace or big corporation way of developing products (and sometimes we could add in education, biomedical and some others) -- and the odds of this happening seem to magically double if you work for NASA.

The "too vague" designs are more common -- just think of this as "commercial" development. This style of design only takes another 80% of the total time allotted to the project (meaning that the total schedule is already 60% over-budget without getting started yet). Sadly these designs don't really nail much down nor are they of much value.

Think of this -- the later a project is, the less will be defined in design (as the morale of the team is inversely related to scheduling and pressure). Since we've already used up 80% of our time in the requirements phase, and the clock is ticking, you should have an idea of how much time and morale is left for good design.

Good design is also an iterative process (one that keeps getting added to as the product is being implemented) -- but that takes more time that engineers don't have (since they are already way late and feeling the pressure). So most of the time there is a feeble effort at the design early in the process, then no one has the time or motivation (or has too many open wounds from the initial design meetings) to further the design as the product progresses. Resulting in brilliant statements like "the Product is the design", or there is the ever popular idea of documenting the product after it is shipping, and calling that the design. This is like watching someone accidentally do something stupid, then hearing them say, "yeah, I meant to do that". Or my personal favorite design strategy is when "they" just tell engineering to save time by copying a previous (award losing) version of the product -- in the hopes that if you keep doing the same thing over and over again (in the same way) that it will result in something new and different.

This last concept is the old business idea that if you keep trying to copy and sell a really bad product, that eventually you will miss your goals and make something that is accidentally a high quality product. I think of this like imagining Ford management asking for the Edsel II and engineering accidentally delivering the Mustang instead. Of course 99 out of 100 times, they actually deliver exactly what was asked for (or not asked for) -- which is something that was built without a design -- things like a two legged table, the five wheeled car, cube shaped bearings, and Rube Goldberg solutions come to mind.

Of course the only thing more expensive (in time and effort) than having a design is the alternative of not having one. Remember, the magic key to understanding "why this would ever happen", is that the costs and blame of not having a design will be "someone else's problem". In fact, that motive explains why most things happen in a company. Most of the smart engineers and managers will shift projects immediately after the design phase -- because it only goes down hill from there.

Implementation (Engineering)

Now that we're way over schedule and budget, we are ready to get started implementing. Remember, we either have a design that tries to account for everything in the universe (but still leave out critical details) -- or the company doesn't have any design or requirements at all and tells people to start implementing without them.

Now I know there are theoretically a lot of possibilities between no time spent on design, and an infinite amount of time on design -- but we can forget about them in the real world. Life is sometimes binary. Companies either get process and design -- and beat you mercilessly with it, as some cruel torture for forcing them to learn the process in the first place -- or companies don't get the concepts at all and you can give up on the idea of teaching them. They say the squeeky wheel gets the grease -- but in most companies it just gets kicked until it learns to shut up (or breaks completely). So engineering will begin with no documentation and not a clue as to what they will be doing -- or mountains of documentation (which they will be buried under, but will never be able to filter through to find anything of value), and it will still miss key items so they will still start without a clue.

So with those two alternatives in mind, most experienced engineers just ignore the requirements and do what they want. One programmer will run off and code the ultimate control or feature because he thinks it would be cool. Another will waste a month adding the ultimate Easter Egg, and so on. The electrical or mechanical engineers are doing their same thing -- one is designing the ultimate power supply (for some other product), another is designing a well ventilated case (for something that is supposed to be submergible), and so on. This is snatching chaos out of the jaws of order. It is easy to find motivated engineers -- just hard to find ones that are motivated to do what you want them to.

We varitably fly through the implementation phase. This is where coders code, builders build, circuits get built, and things get put together. And before you know it (but still long after it was due) -- the basics of implementation are done -- and the fun begins late in the process at what is called "integration". It should be called dis-integration since this is where the team once again falls apart (and turns on itself). Everyone takes all the square pegs and round holes, and tries to make them fit together. There will be teams of people that may not have talked in 6 months (or more), all trying to make their components work together (all created in complete isolation). If you are lucky, maybe 3 or 4 people in the company have actually ever read the design documents -- and they are the idiots who didn't know any better. And to keep it professional, every piece and widget is the pride of someone's professional life -- and any effort to make it work with anything else will be seen as a personal slight and offense to the artistic nature of its very existance. The mayhem that results often makes the design meetings seem like an afternoon tea party at a cricket match. You'd think that someone would have talked to someone else during all that development -- but there is usually such animosity left over after the design meetings that everyone refused to deal with anyone else and they all spit in each others coffee when others aren't looking.

So now a small minority of the team (say two or three), will go through what is being loosely being called a product and try to make something that the rest of the world would call "A" product (instead of a junk-traders collection). This process requires a chain saw, crowbar and some crazy glue. Features and functions will come flying out left and right, and whatever can't be hacked in will be torn off and thrown to the wolves. So for 20 peoples effort there will only be a few peoples work that will ever see the light of day. Since the rest of the team is slighted by having their babies called ugly (or worse, thown away), the serious producers are usually the first to go in the next layoff.

Test (validation)

Testing is another process that is supposed to be going on during most of the product life cycle. But this almost never happens (read the next section "Repeat if necessary" to understand why). Instead, testing is something that will be pushed to the very end -- because we all know that true quality doesn't require any time and must be an accident. By the time you get into testing the entire product is about 300% over schedule -- and everyone's raw and bleeding from the process. Now the barely hacked together results of 3 years of labor (in which everyone is feeling burned over something), is handed over to QA (Testing) -- three weeks after it was supposed to ship in the third revised schedule.

Poor, poor, QA (Quality Assurance). They have the thankless job of pointing out "issues" -- like pushing this button accidentally will electrocute the operator, or like pressing <F1> (help) on Microsoft Word will instantly crash the program (the first version of Office95 really did that). Everyone is so frazzled by this point that they often treat QA with the same respect and compassion as they would show a biting fly. Management is pressuring QA to "just ship it" -- and all the previous groups and individuals that have been previously fighting tooth and nail over everything are now are given a common enemy -- TESTING! Burn the witches! Not only that, testing is usually held in such high regard in companies that they are paid dirt and forced to hire high school rejects and hung-over college students to get the job done. There are usually two or three really sharp QA people --- but most are in way over there head, and getting gang-raped by the rest of the company.

Normally there are stages of testing and release: Development, Alpha, Beta, FC (Final Candidate), Golden Master and then release. Normally these have particular meanings in a product development cycle.

• Development version means highly unstable first passes -- barely working, not showable outside the company (other than specific concepts in controlled environment). So marketing is usually giving these out to impress their friends.

• Alpha means working and usable skeleton, not feature complete but could be shown outside (with bugs -- some show stoppers)

• Beta means feature complete (fewer bugs, and very few show stoppers) and you can actually allow this to some outside testing groups. Since it is supposed to be feature complete, this is where marketing and sales start doing their jobs (of asking the customer what they want), and they start adding all sorts of features to the requirements for this version of the product, right after Beta is achieved.

• FC means that it is going through the final rounds of testing and unless there are show stoppers that pop up, it is going to go golden.

• Golden Master is just your last final candidate (the one that survived to test without anything new popping up) -- it is what is going to ship. Murphy's law of releases means that there will always be more than one golden master -- which shows you how much meaning that term has.

All these stages are supposed to be achieved based on quality, reliability, features and so on. There arewell known criteria and metrics for achieving each stage. Usually, all that is just ignored. Instead, many companies opt to pick levels of release on some secret criteria only known to management -- like a date. Usually, the company is so pissed off at the ever-slipping schedule that they just say "screw it" and ignore QA's whines, and decide that on March 15th this product will be a Beta version because they need to ship on April 15th -- ready or not. And on April 15th the product WILL ship because some PR flack or CEO said it would. This explains why 1.0 versions of products are usually such garbage.

There is one ace up the sleeve for Q.A. though -- it is called "Apathy". If they just don't care, they win. They can say "good enough" at any time, and the product will ship -- or put up with management's date based criteria (and not quality based). After being mercilessly beaten, they finally figure out that they just aren't paid enough to care -- so they just sign off, and then go out and have a release party -- where they crack bittersweet jokes and have a belly-laugh about how screwed tech-support is going to be when this product hits the streets.

Repeat if necessary (product management)

The development cycle is actually supposed to be a complementary loop -- with lots of feedback in each stage to improve the product. Communication is the key -- and the more times you go through the process, the better the product will get (in theory). From this knowledge and experience evolved two forms of development -- cascade (spiral) and waterfall .

Engineering is best done as an iterative process, with you going through all stages of the cycle multiple times to deliver a single product. This is the "practice makes perfect" form of development --known as spiral development (or cascade). This is where you keep circling the goal, getting ever closer without ever really hitting it -- think of a turd going down the toilet bowl. You keep doing the requirement, design, implementation (and integrating) and testing phases in a series of small baby steps. Each version getting closer to what the customer wants and getting better. By doing the stages many times you keep the hate and resentment of each phase fresh in people's minds, and you keep changing who has the control and power -- knowing full well that each person/group will take their opportunity with the power to bludgeon all the others for transgressions in some previous phase. It may take a product two or three times through the wringer to get something completed and all the needed functionality in (with any one phase taking the entire time allotted to all three) -- but it will get done (eventually).

But if you think that the process is bad, you should see the more popular one. The more common old-school philosophy of product development is called "waterfall" development. This is where the entire project team gets one single pass to do it all. Only one try at it -- since everyone does everything right the first time. Everything has to come together at the end. This is called waterfall because it is like sticking the entire team in a life-raft, with popsicle sticks as oars, and send them towards the waterfall. Since everything will integrate perfectly at the last second you save time on all those pesky iterations. Of course, resulting in something that just works how everyone expects it to.... then everyone wins the lottery, twice, and lives happily ever after. Usually, there are more tragic results! Everyone not smart enough to get out of the boat before it goes over the falls will get crushed on the jagged rocks below. The trick in these projects is to get as far downstream as possible before bailing out so that you have less distance to walk downstream to pick at the remains of your fallen comrades (and their project). The less fortunate ones (and naive) just go along for the ride.

Naturally, because humanity has dozens of years of experience at using this bloody "waterfall" design philosophy it is the one that management seems to like the best. I think it is called malevolent denial or payback (since management knows that everyone thinks they are too morons who are too old to cut it as engineers any more). Or maybe waterfall is used because everyone is so wounded by the process the first time that they don't want to go through it more times than necessary. This whole thing reminds me of a polish sex manual -- which contains one page that reads, "In. Out. Repeat if necessary". Many don't believe it necessary to repeat the process more than once for a single product.

Conclusion

This whole process leads to the engineering axiom, "There is never time to do it right but there is always time to do it over". Once this whole mess is complete -- you are now ready to either start a 2.0 version of the product, to fix all the failings in the 1.0 version (without altering anything in the process itself). Or things are so bad that they will start over and do a new product altogether (again, not altering the processes or people that led to the first fiasco). And the best anyone can hope for is that the company gets so irate at what happened, that they completely swing to the opposite extreme, and get the same results by making the exact opposite mistakes (or same mistakes in the other extreme).

So I hope this rather long-winded article helps you understand how development is done -- and why products almost always ship late, under-featured, buggy and everyone walks away feeling sore. To paraphrase our first lady, this should also show you that to truly screw something up "takes a village".

• If you are an engineer, enjoy your role as a rodent stuck in a cog of the machine (getting repeatedly crushed in the gears).

• If you were thinking of becoming an engineer, at least this was a warning on what you are getting into!

• If you didn't understand the engineering Product Development Cycle, I hope you walk away with a new understanding -- and maybe even appreciate the thankless tasks in an engineering department, and what engineers (and others) have to go through to make that product that you mercilessly ridicule.

Even though this was an extremely dark and cynical look at the process -- there were elements of truth in all of it. I seriously hope that by poking fun at all the ways in which the process is screwed up, it can help people learn how things are done (and why), and help avoid some of those mistakes in the future. So laugh, have fun, and rant cathartic -- I know I do

Written 1999.09.29