Are software companies knowingly releasing buggy, defect-ridden software intentionally? In the words of Sarah Palin, “You betcha!” I’m not saying that they release bad software with malice. It’s more about the cost equation associated with fixing the defects. I was talking with Tom Poppendieck at the ADP Conference last week and here’s how he explains the costs associated with fixing defects:
- Fix it now: The effort to fix a defect as soon as your developer types the wrong code is pressing Ctrl-Z (UNDO!). Cost is essentially ZERO. And if you’re pair programming, you’re very likely to catch the defect at this point.
- We’ll fix it next week: The cost of fixing a defect one week after it happens is a fix on the incorrect code, plus the time refactoring one week of code developed on top of the bad code. Probably not too much, we’re talking about a few hours max.
- We’ll fix it at the end of our iteration: The cost of fixing the defect after a two-week iteration is probably about twice as much as after one week. Cost is up to one day.
- Don’t worry, we’ll get to it later, we don’t have enough QA budget for it now: Here we are 6 months later at the release date. The cost of fixing one error in the code is now exponential if you have to refactor 6 months of code based on the initial defect. Cost: Potentially HUGE!
Now, imagine you’re one of the bean-counting managers when it’s time to release the buggy software. Here’s your argument: “What! We have a ton of fixes to do? Marketing already ran the ads announcing the release date! We have to beat Foo, Inc. to market on this or we’re dead in the water! Plus, it’s going to cost us way too much to fix everything now! We have no choice, release it now, we’ll fix it later”
Fast forward to your customers using your new buggy software. They’re disappointed. They start ripping your product (and maybe even you) on their blogs. Word gets out, your stuff sucks! You do your best to put out patches and service packs. It doesn’t matter, word of mouth has spread the bad news, your product still sucks!
Now ask yourself, was it worth it. Releasing low quality software based on the argument above makes no sense. Your company probably lost market share to Foo, Inc. even though you beat them to release because of the bad press. And you still had to spend the money to fix the defects through patches and service packs. By now, you’re probably way in the hole, much worse than if you had taken the time and money to fix the defect when it happened or very shortly thereafter. I’ve already made the argument for the value of pair programming and continuous QA/testing, but it’s worth repeating. Spend the money, spend the time, use your resources and build defect free software NOW! The longer you wait, the more it costs. And you may end up releasing buggy software intentionally, and that’s the worst thing you can do. Maybe David Rice, author of Geekonomics: The Real Cost of Insecure Software, hit it on the head in his interview with PRI: We should start taxing buggy software. That would do the trick!
Related posts:
- Pairing for Quality OK, so if you didn’t guess it, quality software...
- Best of EdgeHopper Here’s a quick list of some of the most popular...
- QA and Testing in an Agile Environment In the past few weeks I’ve been asked about and...
- Are CIO’s Indifferent Towards Quality Software Last night I read through the results of a study...
- The Bug Bash Sprint In every Sprint we do, we try very hard...
Related posts brought to you by Yet Another Related Posts Plugin.
91 Responses
November 20th, 2008 at 5:37 am
I do not think any vendor would leave defects on purpose, if they do they will never get repeat business. The root cause lies in Design Skills - My experience is that hardly 5% people (of all who claim) know design principles (e.g. Open Closed principle, etc.), OOAD, Patterns etc, really know these concepts. Most designers/developers use OO languages but still do functional programming and end up creating tightly coupled and rigid code. Which is difficult to maintain - a small change can impact other tightly coupled areas in the code. Most of the time because of timelines designers are not given enough time to come up with a good futureproof design and the workable design with flaws is put on production. Later on the ad-hoc changes make the code buggy and defect-ridden. Clients always want a quick and cheap fix because of never ending cost-cutting! Conclusion - you get what you pay for.
November 20th, 2008 at 8:48 am
Chris, there are a lot of factors that contribute to the bugginess of today’s software. Tom identified a key contributing factor, but there are many others, including deadlines, budgets, low resources and scope creep.
While there are some organizations that do not try hard enough to mitigate enough of these factors, there are others who do rather well.
What is needed is more of a focus on the issue so that the right level of attention will be brought to bear on the problem from both the standpoint of vendors and their consumers.
-ASB
FAST, CHEAP, SECURE: Pick Any TWO
———————————————————————————–
http://Home.ASBzone.com/ASB/
http://www.linkedin.com/in/AndrewBaker
November 20th, 2008 at 9:29 am
Without a doubt, yes. Management typically doesn’t understand what technical debt is and the ramifications of high technical debt.
It’s a combination of the business pushing too hard for “more features faster” and team caving to the pressure.
Obviously I’m over simplifying, there are many more factors involved here including, as you mentioned, bad practices, not enough QA, inadequate skills etc, but by and large management needs to understand technical debt and how to eliminate it.
November 20th, 2008 at 9:41 am
One of the problems I have encountered is the difficulty in finding good, experienced, committed QA staff. I have found that developers are just not the right people to entrust testing to; it requires a different mindset and a different approach, and a different set of skills. Sadly, good QA staff are hard to come by for a variety of different reasons, not least I believe the lack of any good reward for them - great developers get career progression, extra money, and some kudos. Great testers get bored, and find something else to do.
This problem afflicts a project whether you run it Agile or Waterfall or AnyOtherThing.
November 20th, 2008 at 9:43 am
Hi Chris, what kind of defect? You refer to plain bugs, process gaps, non-sense features, or what? BR, Daniele
November 20th, 2008 at 9:45 am
Even the most jaded purchaser of software knows that software companies are not intentionally releasing buggy software. The REAL issue is that software, especially enterprise software, has become incredibly complex. We are at the point where it is impossible to test for all conditions, to find all bugs, and to think of everything that might go wrong in a very complex environment. Software companies manage this by either placing tight restrictions on the environment (such as Apple) or understanding that there will be some condition they can’t predict.
Here’s a real life example:
I bought an HP Deskjet printer the other day for my home. It is a simple, no nonsense affair. I hooked it up to my Maxtor Shared Storage drive that uses some older version of Windows Storage Server. The printer driver cannot print the composite black necessary when only the color cartridge is in. Works fine when direct attached, not when attached to this device. In all fairness, could HP have thought of this scenario? A printer hooked up to a five year old NAS device? That is a layer of complexity not accounted for.
So the short answer to you question is “none of the above”. It is the complexity of modern software and modern software environments that make 100% defect free software a silly goal. Most software works as advertised out of the box. You simply cannot test for everything in the world.
November 20th, 2008 at 9:58 am
Taking a step back this happens because businesses find their own balance between time-to-market, cost, functionality and quality that works commercially. Sometimes software houses get this wrong and release software with too-low quality. Companies that do this repeatedly will be punished over time.
The balance that works commercially is sometimes more toward low quality than many would like. But some who wish for less bugs don’t have a commercial viewpoint.
Is software like making a movie or like building a bridge? I believe the answer is dependent on the type of software. Consumer software has far lower quality demands and the rigidness of the development process is somewhere between movie making and bridge building. Critical systems like flight control or medical life support systems have much higher quality demands and the processes are much stricter in terms of configuration management and testing for that reason. Also the standards are dictated by legislation not just commercial factors. If the same high standards were always applied to all software we wouldn’t be having this conversation on a social networking website in 2008. We would never have heard of businesses developing at “internet speed”. I’m irritated by buggy software as much as the next person but there is the big picture.
Darren
November 20th, 2008 at 10:34 am
Chris, having been in this racket for more years than I care to admit, the view of software has changed overtime from carefully crafted functionality to almost commodity status. The expectations of what software should be able to do in a single package has made it exponentially more complex, which means that the opportunity for introducing errors is skyrocketing while the ability to test or even preventively test errors is diminishing. It is not our knowledge that stops us from delivering perfect results, it is the sheer amount of effort necessary to achieve perfect results.
I have worked on projects to develop an air traffic control system, to overhaul nuclear reactors, credit/debit/smart card transaction processing, to name a few where quality was simply not subject to any compromise. It can be achieved, but the costs are such that it can only be achieved when the end-users see the value and are willing to pay for it. What you need to realize is that on a basis of errors per number of lines of code we have achieved incredible improvements: based on conflicting requirements we are often unable to test all combinations and permutations to even be able to detect the errors resulting from that complexity. I look forward to seeing SOA take hold as a general strategy for formalizing the interaction of functionalities within software as it is within an infrastructure, because isolating functions in manageable chunks as well as formalizing data exchange between chunks is probably one of the few viable approaches to deal with this complexity.
Microsoft is often chided for bugs: I have had to find workaround more than I care for. At the same time, people don’t give them credit for stalling releases to hammer out more bugs where possible. There is a point where pundits are mocking the company and the risk of releasing code with bugs is less than a perception of inability to manage planned project deliveries. There was a time people lined up to be early adopters, now they take a wait-and-see approach. Microsoft, and most other vendors, generally own up to problems and deliver fixes. Translating this to internal development projects, most companies do not have this on-going support mentality for in-house projects. Outsourcing is only going to make matters worse, because now they have to contract out all bug-fixes as well, which often translates into sucking it up and learning to live with the problems.
Are we contributing to the problem as end-users? In the words of Sarah Palin, “You betcha!” - the more companies try to cut back on in-house staff competent to deal with these problems the worse it will get. Outsourcing on the basis of cost advantages means cost cutting: you know that QA will be the first casualty of that, with change management a close second. Cost is also a factor for vendors: you can only invest so much in a product if you are interested in earning a profit. Walmart is living proof that consumers have a taste for penny pinching until it turns out that the children have poisenous toys to play with, because the only way Walmart can deliver to that demand is to find suppliers in areas where quality and safety are incidentals. Of course, at that point public opinion blames Walmart (or its competitors) for the problem.
I think we can safely say that with the exception of malicious hackers people do not intentionally create buggy code, but corporate consumers drive them hard to cut corners: the results are predictable and evident. So, perhaps we should ask “Are you prepared to pay a premium for more reliable software?”
November 20th, 2008 at 10:41 am
I believe its all that you’ve listed with possibly the exception of poor engineering. Engineers code to what the spec indicates. If poor engineering comes into play, its due to poor analysis. We all been exposed to the “meet the deadline” scenario, so cuts in QA and especially UAT are made to meet the date. What also comes into play is the the old “80-20″ rule - well we have 80% of the functinality, so lets go so as to avoid the hit to the timeline. We’ll address the remaining 20% in the next release or phase.
Your comment regarding the cost of fixing the defect being high is also well taken. It’s a statiscal fact that fixing a defect after the fact is much more expensive. This comes from poor upfront analysis in my opinion.
Poor upfront analysis is most probably the root cause of most defects.
November 20th, 2008 at 11:00 am
I think buggy software release are due in part to two issues.
The development team is typically over-scoped and are rushed to meet an unreasonable due date in relation to the size, competency and aptitude of a given team. Scope changes, scope creep, unforseen issues will come up, and the allocated time buffer, if any,will be consumed and even cutting things out, will not make the due date any more reasonable.
Also the QA, if any, is more a Failure QA, and will often miss defects due to limited QA resources, staffing, and QA software capabilities to detect and catch all bugs. The answer may not always be more QA resources, but smarter and more efficient QA. Sometimes this involves providing tools to QA to become efficient.
November 20th, 2008 at 11:02 am
I feel its Time-Quality trade off keeping ROI in mind
November 20th, 2008 at 12:12 pm
In addition to being a mix of the reason syou stated, there is one that you may have missed(intentionally?). That is, there is so much competition in the market today that even a day’s delay in releasing the software may mean millions in losses. If this means releasing, as you called it, buggy defect-ridden software, so be it! You have to realize that most of these organizations are about as big as mine (10-15 people) who find it hard to hire somebody to system-test the software prior to releasing it.
The companies are aware that there are problems but will try to get someone to give them a contract for the concept prior to fixing these problems. Some times that works, but most often it results in just crappy software!!!
November 20th, 2008 at 12:33 pm
The problem is far too often the developers build product for developers and not users, the net result is products that are inside out verse market driven….then the problems start
November 20th, 2008 at 12:56 pm
I think the answer to your question is - of course, they do. The goal of a business is profit. Period. Good business calculates the point when extra testing or extra conditions on the development process stops bring any more value and releases the product at that point. What considerations and use models go into this calculations is obviously up to a given company and usually constitutes a trade secret. Bad software companies miscalculate that point often, good ones - rarely.
November 20th, 2008 at 1:21 pm
I can offer a unique perspective as a technology escrow company that receives the code and does some level of evaluation and/or verifications in many cases.
A vendor in the last year actually admitted to making an empty deposit because they needed the business . They knew that by the time the client figured it out they would have the code ready and be able to make a valid deposit. How scary is that?
We are seeing an unprecedented amount of releases because software companies are unable to weather the current storm…This week I had several calls because vendors were filing bankruptcy and are offering to put their code in escrow believing they will come out of it and not wanting to lose the business they have.
I look forward to reading other perspectives…Karen
November 20th, 2008 at 1:28 pm
Far too often testing gets short changed because its pushed to the back end of the project. My teams don’t produce buggy software. We start QA the first day we write code. We use several techniques to create a high quality product: automated unit testing, code reviews, ad-hoc testing, automation of build processes, continuous integration, frequent releases to the customer.
If people are releasing buggy code on purpose, I would suggest finding a new software vendor.
November 20th, 2008 at 1:31 pm
I think there has to be a balance between the need to get the software out the door and having good quality software.
Some companies are willing to take the risk of releasing software that has bugs in the effort to start receiving revenue from the product.
The quality has be directed from the top with buy-in from the Finance group.
It’s all a partnership
Ian
November 20th, 2008 at 1:58 pm
I don’t think companies do this intentionally. Having lead product management in software development I can tell you that we would love to have error free software on the first release. The reality is, it very unlikely. There are always unforeseen obstacles and you cannot QC test every possibility.
Also, you sometimes just miss things, despite doing user acceptance testing, performance testing and the like. The last point is funding. In most cases you have a budget and timetable to get the software released. You don’t have unlimited funds and an open timetable to design and develop the solution.
That all leads to releasing “buggy” software. What happens after that tells you if the company is truly committed to developing and supporting the solution. If big issues are not addressed immediately it probably tells you they are underfunded (if it is a small firm), or not committed (if it is large corporation).
November 20th, 2008 at 2:03 pm
I tend to agree with BV on the trade off thing, however since improving the maturity of an organisation normally raises the quality of the products and at the same time reduces the time used for development it might not be as simple.
In my opinion buggy software is released intentionally, but it’s often due to poor requirements, estimation based on nothing but gut feeling, insufficient planning and lack of communication just to name a few.
November 20th, 2008 at 2:08 pm
>>>Is it poor engineering practices? Not enough QA? Or is it that so much technical debt has accumulated that the cost of fixing the defects is just too high by the time the release date comes?
Yes, yes, and yes and many times a combination of two or three of them.
>>>Is it poor engineering practices?
In some cases, companies aren’t even aware of their practices, some know its not so good, and some think its good when its not really.
>>>Not enough QA?
I think you may find a very few companies that have enough QA. However, the question is, what’s enough? If its a release 1 product and the only one of its kind or cheap enough, then let the people in prod (aka the consumer) be the QA. However, when its a tight market of competitors, then its a matter of applying the most QA for the budget.
>>>Or is it that so much technical debt has accumulated that the cost of fixing the defects is just too high by the time the release date comes?
Since refactoring is still not done by many, then this may be the perception by many. Also, when you lose some of the key spaghetti coders, then it does become hard to do. But in many cases, I don’t think people realize their technical debt in teh first place.
The other reason may be that people are used to a certain threshold of bugs and just live with it. As long as the sev 1s & 2s are taken care of (in the most part), then most people are content. Hey, we live in a Sev 3 & Sev 4 world…
November 20th, 2008 at 2:11 pm
Technology moves forward and by the time Release One is in test the management has a decision to make: getting bugs out of the almost obsolete Release One or put its development staff to work on Release Two. Smaller companies can’t afford both. BTW, how many of you skipped “even-numbered” releases of Adobe products?
November 20th, 2008 at 2:13 pm
These are all good answers; one that I have not seen is a “race to market” situation. Markets are highly competitive and companies are ever striving to gain the competitive advantage of being the first to market. This kind of attitude fosters the “just get it out the door, we’ll fix the bugs in the next release” mentality. This requires a certain amount of risk analysis to determine that there are no critical/fatal bugs being release. There is also a risk of damage to brand image. If a company determines that the rewards out weigh the risk they may be willing to release buggy software. In the end the bugs usually get pushed aside for new feature and may remain in the system for a long time.
No amount of designing and testing can prevent businesses from racing a product to market. Companies often face a choice of releasing a more stable, less functional, non-differentiated product or a less stable, all the latest functionality one of a kind product. If they go with the first choice it usually takes a few releases to get to a stable, latest functionality version. In that time competitors may have improved their product reducing the differentiation. The other choice leads to the situation I described earlier.
So the question now becomes how do you deliver high quality, stable systems in the shortest time possible while remaining ahead of the game?
November 20th, 2008 at 2:24 pm
My experience [36 years] indicates that not much has really changed in this area. Software has always been released that is not 100% perfect, at least partially because it is very difficult to know (and therefore test) all of the possible ways the software has ben used. I also believe that the potential for a “bug” is O(k^n) [k being constant, and n being the complexity of the code]. When this is combined with the rapid expansion of the amount of software being released, the results should not be a surprise.
I also agree in principle with Atul’s coments above (although I think the number is much higher than 5% if you look at development groups vs individuals [a team of 10 people with an experienced lead who is properly "leading" the group and has a thorough background in the design principles mentions should ensure that the work of the entire group conforms to these principles].
November 20th, 2008 at 2:37 pm
You have answered your own question with your commentary. It is expensive and time-consuming to release “bug free” software, and if you miss a ship date in most commercial situations it won’t matter how good your software product was. “First out the door” beats “best out the door” almost every time.
We are literally managing billions of details in software with minds that can only keep track of 5-8 details at once. The most brilliant people in the field may be able to track 8-12 details at once. All of our design principles, programming languages, IDEs, and methodologies are simply tools to extend that reach.
Software development is error-prone because we have yet to work out a full set of tools that gracefully matches our limited mental abilities with the complexity of the problems we are working on. We also keep working on increasingly difficult problems.
November 20th, 2008 at 4:06 pm
No… they just don’t have their act together… young and transient staff and over optimistic sales & wall street pressured deadlines. No collective priority to do a good job or to capture long term market share.
November 20th, 2008 at 4:50 pm
I guess the ‘real’ question is, “is there a perfect software”?
November 20th, 2008 at 4:54 pm
I think google certainly introduced the concept of perpetual beta and that hasn’t helped. However I really think the culprit is that software is getting more complex and development and release cycles (due to customer demand) are getting shorter. More bugs is an inevitable result.
November 20th, 2008 at 4:57 pm
Chris, I suggest you have a look at Eric Sink’s column http://www.ericsink.com/articles/Four_Questions.html . That may help.
November 20th, 2008 at 5:40 pm
Hello ? Do we read the disconnect: “first to market” verses “Is there a perfect software”. You know I have worked in this business too long, and I laugh again to see that only men are involved in this converstaion.
I think, yes we release buggy software, because we trip up in the functionality, verses good specifications and developers seeking to have perfect code. We sometimes/often do bad design at that outset because we assume what the customer needs based on shallow marketing data and someones money.
November 20th, 2008 at 6:02 pm
In my experience it’s (c) “…the cost of fixing the defects is just too high by the time the release date…”
Seeing typos or other oh-so-easy-to-fix defects in software I’ve worked on being packaged to send out the door used to drive me up the wall as a UX professional (and former editor). But ultimately I was not “the decider,” and as you and others pointed out, the impending ship date became the overriding factor, above the seeming embarrassment, perception of poor quality, and relative ease of fix that I or others might have felt.
November 20th, 2008 at 6:16 pm
To build on Frits and Ian the time to market can be a factor in the success of your new product, so companies will scrimp on due process and just get it out the door. This is aggravated by the premise that first to market will lead the market. (some seem to have forgotten that the product has to deliver value too not just the idea)
The scrimping will affect the scoping and design; Just get it done and we can deal with that later, but when do you get a chance to rewrite when new features need to be added to maintain the edge. As now several other shops have seen the opportunity and are now throwing their own team at the problem, building on your design and the response of the market.
This is one reason why Agile seems to be a good thing, it allows for the Just get it done while addressing the testing, value and applicability issues as well, as long as the software guys have access to the customer or some good facsimile.
So to agree with your question it is Yes companies do intentionally release buggy code, but do intend to rectify this later given time once they have solve all these other issues first. The Intentions are good but fail due to the need to show profit and new features, hence leads to the technical debt answer.
It is not poor engineering but the focus is on adding features and not on making the product seamless. The argument goes if there is a workaround then it does not need to be dealt with, just add the script to the support desk knowledge base.
November 20th, 2008 at 6:31 pm
My experience has been that most software companies use the “good enough” approach to releasing software. Even the best companies with the best process, quality, architecture, etc. follow this approach since it is impossible to test every combination of desktop software, hardware, mobile, etc. People have to know how to truly use the good enough approach and how to do object and data based quality assurance correctly to make sure they are truly getting to that point.
That being said, I have seen some companies use this as an excuse to release the software before it’s time, but that is due to poor management or one’s without the guts to say you have to miss the target date due to quality problems. This is especially true of what I call release (or basically beta software). Some people use the real world to test out the defects versus taking the time to do it right.
I ran the technology areas for two software companies in the 80’s and developed commercial software. Back then we still used the good enough approach but we tested the heck out of a well designed system first before it left the building. We also used a small client base and proper regression testing to make sure some of the obvious performance, data and programming issues were resolved early in the date.
That’s my two cents
November 20th, 2008 at 6:52 pm
Of course companies intentionally release buggy software. First off, there’s no way to know when you’ve fixed the last bug. Most companies establish release criteria that specify no more than x serious known bugs, y not-so-serious known bugs, and z live-with-it known bugs. And then, in the weeks before the planned release date, they/we argue over the severity of the bugs.
November 20th, 2008 at 7:12 pm
Nothing is ever working perfectly before launch. Microsoft is a perfect example. They release patches for their various OS all the time.
November 20th, 2008 at 8:27 pm
Bill Gates has been doing it for years. Its all part of the service.
November 20th, 2008 at 9:59 pm
I have to second what Frits has said. My personal experience is similar. When people begin to fall behind, QA is always the first area to get “fast tracked”.
I have worked in environments where QA did not exist and it was based on the CLIENT to give feedback! I worked hard to fix that environment, the excuse I use to hear was that there will be hardly any bugs because I know I coded it correctly, I use to rip apart that level of arrogance.
Another issue I have run into is poorly written contract - SOW. I have been in environments where the person that sold the project, didn’t understand what they sold to begin with. So the project manager is left scratching their head.
Lastly is the sheer lack of common sense! Non intuitive set ups, missing buttons, and broken functionality or not well thought out. I don’t know how or why some people became software engineers and business analysts.
November 20th, 2008 at 10:34 pm
Hi All,
In my humble opinion, software tends to be error prone as,
1. Requirements analysis is not done correctly( The client wants an orange and he gets an apple)
2. People are inclined towards showing something or the other by just rushing towards development
3. There is no clear bigger picture(Architecture) available
4. Last but not the least unreasonable pressures for delivery.
All these factors lend to error prone software. Also companies are more inclined towards using ‘Cutting Edge’ technologies where the business problem could be solved by using simple technology.
Net the effect percolates from top to bottom which results in error prone software.
Regards,
Amit
November 20th, 2008 at 10:53 pm
To answer Chris’ question I would have to agree that software companies are throwing immature software out into the market place. It is cheaper for them to do this than throw extra time, development $$$, resources in doing full Q and A. The best bet for a software company is to sell the product and let the (unsuspecting) public/ users find the bugs whilst obtaining a positive cash flow on sale, realising their development costs spent so far.
As mentioned above I’m sure that software vendors do not purposely issue software with bugs, but it becomes a break-even discussion as to whether to put the product on the market or commit further $$$ to further development and bug fix refinement.
November 21st, 2008 at 5:02 am
Bugs are not introduced intentionally. You can view bug as a human error in the software development. If the bug is identified during some testing and if the severity is critical, then it will be definitely fixed. If the bug is not so severe, then the same is mentioned in the release notes/changelog. In some cases the bugs would not have been caught during any of the testing and it goes along with the software. You cannot then say its a buggy software. You need to understand the environment in which the softwares were developed, how it was tested etc.. before saying “software companies knowingly releasing buggy”.
November 21st, 2008 at 5:20 am
I hate to say it, but the long tradition of making things poorly for recurring profits has influenced the software, and webware, industries. We’re dealing with the repercussions of the industrial era, where quality usually went out the window in the name of exactly what were talking about here, speed, cost, and namely, who’s involved, who owns it, versus, and this one really hurts, who’s receiving it. Not a very user centric approach, fairly tycoon in it’s approach, unfortunately.
I’d like to see government built software, just for the argument, not thinking it would be any better overall, but it would be interesting.
November 21st, 2008 at 7:12 am
Because the ‘business’ forces (cost, schedule), and the ‘environment’ forces (resources, complexity) are stronger than the ‘development’ forces (adaptability, scalability, and reliability). Unfortunately, there is collateral damage, ‘buggy’ code. *sigh*
November 21st, 2008 at 7:49 am
All of the above are great answers! Being a software developer myself, I must admit to being a victim of scope creep, deadlines, budget and resource constraints.
That being said, most software developers take great pride in the products that they create. Core functionality is typically very well tested by the developement and QA teams, but it is impossible to create and test the infinite hardware and software combinations that can be found outside of the development and test labs. Add to that an equally infinite number of combinations that yield sequence of events and/or timing errors and I’m sure you can begin to appreciate how difficult the development and testing process can become.
For instance, I know of some clients that have recently upgraded to Windows 2000. That is 8 year old software that is not even supported by Microsoft anymore, yet I have to build and maintain software that can run in that operating system. If their software is that old, what does that say about the hardware that it is running on? Does that mean that I “dumb down” the feature set to support the ancient hardware and software? This is a major consideration for most commercial software developers, and also the reason behind some of the “Windows Bloat’. Microsoft attempts to support older hardware and software interfaces by carrying forward legacy API’s. Yet these same API’s cannot be fully tested in today’s complex environment because of the immense effort required to do so. At what point can a software maker safely deprecate legacy API’s without angering other software vendors (keep in mind that such a decision means that they would have to retool their software as well or stop support on it)?
As Software continues to become more complex and steadily pushes for greater, more streamlined interfaces into other applications, at some point, all a project manager can do is make a good faith effort to test on what would be considered representative environments, meaning certain configurations are completely untested. Furthermore, most QA teams work off of test scripts that can only test a finite combination of event sequences.
When errors are found either through code analysis, peer reviews, QA or any other avenue, they are reported to the project manager who assesses the impact of the error and weighs it against cost estimates to repair and the real or perceived value returned.
There may be a handful of known defects that return little or no value at a high cost - these defects are usually not corrected because the impact would be negligible (especially if there is a known workaround) yet the effort required to correct it is immense.
Ultimately, Anton and David speak the truth. Software is a highly competitive business. What good is there in putting a 100% defect free software package to market two or three years late with only the barest feature set while competitors release feature laden software that actually provides real value to their users? Sure, some customers may experience bugs, but >90% are unaffected and/or never experience the bug.
“If a tree falls in a forest and no one is around to hear it, does it make a sound?”
“If a defect exists in software, but nobody encounters it, is it really there?’
Finally, I assume that you actually report all software defects that you find, right?
November 21st, 2008 at 7:57 am
My two cents: bad engineering and bug fixing costs apart, sometimes clients get what they ask for.
Forcing release dates on the developers with the excuse of time to market makes for things to be released buggy. And knowingly so.
Also, administrators and project managers tend to follow the “client is always right” creed and end up getting themselves into some holes they have no other way of escaping other than releasing buggy software. Gain the client today, worry about the bugs tomorrow, seems to be the most practiced engineering technique today.
November 21st, 2008 at 8:10 am
I have worked with and tried to help design “government software.” It is NO better.
November 21st, 2008 at 8:48 am
I guess Shelley-Ann take on your quest. These days “bugs” are calling as patches or updates or new versions. Knowingly or unknowingly, not only software companies do but buyers too do keep their all the requirements in one phase.
November 21st, 2008 at 9:02 am
The answer is simple… accountability and ownership. People don’t like to measure themselves and each other, so they often wait for QA to do it… and it is too late by then.
Yes, I am an architect and a big fan of analysis, design, and agile methods, but none of these work without automated regression testing. Software developers must develop automated regression tests for themselves and each other’s code during development (if not test-first) to ensure it continues to work as originally expected as it evolves… and people must “own” there code and the regression test.
System integration testing is different, and that’s essential too… but it doesn’t impact ownership and accountability, it perpetuates the problem if it is the only QA practice within the organization.
November 21st, 2008 at 9:50 am
Too often there is an attempt to push engineering too hard too fast due to promises the business has made to its customers or partners based on early estimates or just arbitrary selection. Engineering tends to drop testing faster than lightning when the amount of work greatly exceeds the amount of time to accomplish it. Often, they will want to show their management they are working hard and fast by marking things as done, all the time assuming QA will find their defects. In these cases, it is common to find that the QA staffing is insufficient, the QA time allotment is too low, and the QA environment not up to par. Throw in the final match — an iron triangle of scope, time, and cost — and it’s no wonder folks are just letting things get to production.
My postulate is that product quality drops exponentially with a drop in perceived economic strength of the business environment. Thus I would expect that these days you will see some of the worst quality software you have ever seen making it to production.
November 21st, 2008 at 10:12 am
I don’ think any software company intentially release buggy software and even any developer wants to write buggy code. Ofcourse sometimes unknowingly developers do that. I also feel that there is need of good professional training to the developer. Sometimes, client is also responsible for that giving change request at the 11th hour and asking for delivery on time.
But nothing is intentional either from Management, Developer or QA side.
November 21st, 2008 at 10:52 am
It is a combination of so many factors … and this will differ with the priorities of each group.
Time & the SHIP DATE drives this the most I feel, coupled with the lack of foresight (in not being able to list and therefore test for every possible scenario - the knowledge factor) and In the end, the need to solve the bigger issue shadows the minor manageable ones (the ones that don’t “BREAK” the solution) that eventually translate into the deliverable.
November 21st, 2008 at 11:56 am
A company will decide that a certain level of bugginess is acceptable for a release - that’s a commercial decision.
The amount of resource and effort one puts into QA displays a decreasing rate of return - drawing the “appropriate” cut-off is also a commercial decision.
November 21st, 2008 at 1:36 pm
So many good points have been made. There are some additional points that I think have also been left out.
I have been exposed to a wide spectrum of software development environments. I have worked for small startups and large corporations; I have been a consultant as well as a permanent employee; I have been a developer, a project leader and an architect; I have helped to build in-house software solutions as well as commercial software (I am currently working for a company that sells appliances); I have also been exposed to multiple vertical industries.
In every case - without exception - the same problems keep re-appearing. In order to avoid bugs, you need to have the following:
1. A clear, well-considered set of requirements that focus on the user’s goals and fully explore all potential failure paths. These requirements should make every effort to avoid discussing the implementation. More importantly, there must be a real effort to promote blue-sky thinking. This is critically important. In order to create a solid design that can adapt to change, the designers need to have a complete view. This is similar to statistics - the more data you have, the better the result. With design, the broader the perspective you have (i.e., the bigger the picture you can see), the better the resulting design will be. By limiting the scope before the designers are involved, projects risk having brittle, myopic designs that take longer to implement because designers are unable to find the common elements that would have pointed the way to a strong, robust framework.
2. A solid design / architecture that follows an established set of core principles that are understood by the development team. As I pointed out, a solid design is predicated on the creation of strong and broad requirements (a rare creature, at best). However, I have also seen many instances where insufficient time was given to the design process. Most of the time, coding is fast and easy - even boring - once a solid design has been put in place. Most applications are not rocket science and the devil is in the design. Make the time for extensive group design and use agile principles to constantly revisit the design when necessary. My experience is that developers always improve when they are exposed to group design sessions.
3. A commitment to automated testing and established target goals for code coverage. For example, any server-side component should achieve 99% code coverage during automated testing, since it should be easy to isolate such code for testing purposes. UI code, on the other hand, should have a lower threshold since a lot of display-layer testing can be hard to automate. In every case where this has not been implemented, I have seen higher rates of regression bugs and a tendency for developers to become afraid of old code (the “monster-in-the-attic” syndrome). That tendency simply leads to even more bugs as old code is improperly fixed when bugs eventually crop up.
4. Sufficient code-level documentation to avoid the “monster-in-the-attic” syndrome. Too often, working code becomes broken code because the developer is too intimidated by undocumented code to make the modifications required for a new feature, opting instead to make a superficial change that increases fragility and introduces new bugs or inconsistencies.
5. String QA testing process. This should involve a proper bug tracking tool (yes, this still needs to be said), a QA team that does not consist of the developers that wrote the code and a method of measuring the quality of a release (e.g., an index value that assigns a weight to each bug severity). When this is not in place, it becomes impossible to know whether you are release strong code or a piece of garbage. Assessments are made based on anecdotal evidence coming from the developers themselves (and they have no vested interest in being honest about this).
November 21st, 2008 at 2:58 pm
I propose the answer comes in two parts: Yes and No.
That is, some software companies do know that a given release has defects (and what they are) but takes a commercial decision to ship anyways, presumably calculating that the damage to their reputation will be less in total than the damage to their revenues, market share, what have you from *not* shipping.
OTOH undoubtedly some companies have *no idea* about the status of their proposed release, how many defects exist, what functionality or features those defects impact, and so on. These folks ship on a wing and a prayer…
Thirdly, there are some companies that ship when the known defect count has reduced to zero, or zero in some categories (such as “critical”), or some “acceptable” threshold figure (a more considered, even planned version of the first case, above).
And just for completeness, of course there are a very few companies that know there are no defects in their release (and I mean definitively no defects, not just none found or none known).
Actually, the above categories also very much apply to in-house development organisations releasing into their own host organisations, too - although the thresholds are often higher than one might expect from commercial software vendors.
So, that’s all well and good, but your subsidiary question is more interesting I think: Why do these companies ship buggy software? Simple answer: because their market tolerates it. This *can* be a good thing (people get software cheaper, quicker than if it was heavily gold plated).
Actually, I hold that there’s as much problem with shipping software that’s too GOOD (low defect counts or defect-free) as with shipping software thats too BAD. Truly competent engineering teams should be able to build a product to *specified* (quantified) quality criteria (Cue apocryphal story about Japanese semiconductor manufacturer - please ask if you haven’t heard it already…)
For the past 15 years I have run projects using Tom Gilb’s approach to *quantifying* the interesting and important qualitative aspects of the system under construction, with high and low bounds for each such dimension (along with planned, current and actual). Tom has continually evolved this approach with e.g. Planguage - for those with an interest I recommend taking a look.
BTW Mature Agile projects can also select *different* levels for various qualitative dimensions (a.k.a. non-functional requirements) along the project time-line (i.e at successive milestones or inch-pebbles). Simplistically: buggy early, less buggy later - we have done this from time to time with good response from customers and sponsors.
I also remember Ed Yourdon’s emphasis on “good enough” software as a desirable mindset for all developers. And Goldilocks and the Three Bears’ porridge, too
- Bob
November 21st, 2008 at 5:31 pm
Sheo, I respectfully disagree. Look at the endless Microsoft offerings that are fraught with easy to find yet still malodorous bugs simply oozing out where anyone, much less a talented QA person, can easily find them.
VISTA? XP before it’s first service pack?
Tha fact is, folks like you and I (Early Adopters) have caused these companies to release essentially Beta code as production, get paid for the dev, then fix issues that the user base finds as unpaid QA folk.
I have a QA company- AlphaBetas Inc. We have tested 100’s of games and apps, but we have been turned DOWN by 100’s of dev houses- and we were charging $5k for a complete run, end to end. the fact is, developers believe they are best suited for the testing, but as we are all aware, they are not.
My .02…
November 21st, 2008 at 7:56 pm
Chris,
I would suggest that it is more about getting the product out to market to take advantage of the product enhancements, which are designed to drive sales. If the product spends to much time in Q & A or Development, your competitors have time to release your features early or you have customers waiting to make a decision because the sales team have informed them of the expected new features.
November 22nd, 2008 at 8:25 am
Disclaimer: The following points are strictly my opinions and observations in different places and contexts and they strictly DO NOT reflect the opinions, practices ,principles,beliefs of the companies which I currently work/or have worked for in the past.
In my opinion, the schedule always drives what to do ,what not to do,as Glenn pointed out.
1.The developers do not get time to write and prepare UT cases, which cover atleast 90% of the code.
2.The QA team does not get the time which it needs to analyse the system in detail,understand the requirements, dissect the design and come out with test cases, which I think is not available or not used properly ( if available).I have observed many times in my experience across different companies, that some of the QA members are not really worried about knowing the system in detail and expect inputs to be fed to them,rather than doing their own study and understanding , by themselves, thus buying the perspective of the Architects and the developers. This is due to the fact that the QA team works in isolation sometimesm,as Jonathan pointed out.
3.With the schedule driving everything, the QA team is not allowed to focus on automating and developing automation test tools.Most of the time such efforts are not taken into account while preparing the effort estimates,where all the seeds of deep troubles are sown.
4.Undocumented and tightly coupled complex code,where the future bunch just plans for a work around,rather than getting in and fixing those bugs in those so-called “hard to touch” code, which is backed by the management to save the time on impact analysis and regression testing,again as pointed out by Jonathan.
As David pointed out, the people who lead the team should have their hands red in the principles of software development and testing, so that ,those convictions can be passed to the team.
So it is not with the companies only to blame,but the people who form/lead those respective teams.
November 22nd, 2008 at 8:57 am
It is true that a lot of software escapes instead of getting released!
Years ago, when I was getting my MS in computers, one class showed that bug free software is possible only if the time and expense involved in mathematically proving your algorithms is undertaken. The proofs would have to be on all algorithms from software to OS to firmware to the logic used in chips. The theory being if all algorithms have been proven, different hardware and software combinations won’t matter. Belief in that is up to your discretion, but the fact remains: Proving algorithms is too costly in both time and money so it is not done.
To answer the original question, yes companies are knowingly releasing buggy software. This is a fine heuristic if statistical quality control is part of the testing and release process. It keeps prices down and creates fairly reliable software. Just remember the old adage buyer beware applies.
November 22nd, 2008 at 9:52 am
“why does so much defect ridden software make it out the door these days?”
I would like to comment in purist sense[ideal case= bugfree systems].I think there are two aspects to this question,
One deals with tools/processes/techniques and people who are using them.
We do not have tools [that includes hardware and languages/platforms that we work upon] are not perfected and
Many times it is also found that people who are working on software are also do not have mastered art [seems cutting edge platforms are playing role here].
And,on any imperfect base,we could not build perfect thing.
The second aspect is of dimensions,as a developers/Architect,we know and had to build a system with trade-offs between abilities.
It is like the principle that “Any individual can not be present in two places at a given time”.
The trade-off decisions we had made,has inbuilt resonance which somewhere resist the stability factor to go to 100% hence we get those buggy system.
November 22nd, 2008 at 10:09 am
I think that how QA was integrated into the classic waterfall development process is to blame for it being a step child. The agile mindset where QA is part of development early and often is new to many of the software companies and their product/project managers who are responsible for rolling out buggy software. I have noticed that there is a passive (and in many cases very active) resistance to adopting good TDD as well as active QA involvment during development in both dev and QA teams.
As to whether the software companies do it intentionally or are slaves to old habits is still an open question. IMHO.
November 22nd, 2008 at 11:46 am
Yes! There are four reasons for the answer.
Darren gave the first: businesses make release decisions based on what makes them look good to investors, customers (internal and/or external) and/or market (think analysts, competition, media). This rarely aligns with the readiness of the technology.
The second is that while everyone knows how a good project should run, a very small number know how to execute. This is because a small part of project management success falls on the technical side. What sets exceptional PMs appart from the others are the people skills. Project Management is a contact sport and you need these soft skills to come out ahead:
- Negotiation
- Leadership
- Teamwork
- Conflict management
- Communication
- Relationship building
- Consensus building
- Emotional intelligence
- Motivating individuals
Coming in third is corporate culture. No matter how good you are, or your team is, the status quo can break the project. Now I’m not saying it’s OK to point the finger at someone else (because you always have three other fingers pointing back at you), but when you are a cog in the machine there’s other machinery that can snap defeat from the jaws of success.
Finally, and probably most important, customers are A-OK with bugs in their software. They expect it and still pay money for it. If customers stopped opening their wallets for poor quality you’ll notice right away that QA’s timeline on the schedule will no longer be the FIRST target to cut when milestones start getting missed. squeezed
November 22nd, 2008 at 12:07 pm
It seems like the software companies are getting both an economic and strategic benefit from this practice. The economic benefit comes by way of the free integration testing they are able to get from the user community. The strategic benefit comes from the quality of the testing. Users who are doing things with your software that you may not have imagined, integrating it in ways you never intended will find more issues, will document them better, will be able to recreate them, will work with you to test the fix (for free) and will monitor the progress until the fix is applied successfully. It seems like the most dedicated testers are the ones who are using the software.
November 22nd, 2008 at 1:43 pm
I must agree with the first answer given by Atul about the root cause of proliferation of bug-ridden software. Except from my personal experience he is being too gracious speaking about 5%, it’s more like 0.5%.
Though many other reasons exist, human factor is absolutely the most important one. It’s impossible by using any clever techniques, processes, software create high quality software with low quality developers. Somebody studying great achievers concluded that they all spent at least 10000 hours actively perfecting their skills. In software field it would mean that a person should have at least experience writing 100000 lines of real code, following correct software principles. And this is only the beginning. As Einstein put it, “after a certain high level of technical skill is achieved, science and art tend to coalesce in esthetics, plasticity, and form. The greatest scientists are always artists as well.” The greatest software architects are always artists as well.
Since companies cannot really provide for high level software developers, a lot of procedures that supposed to mitigate that were created. You can reduce visible software defects by testing. In fact it’s essential to have this process in place. But testing can do only so much. As McConnell wrote, “Testing by itself does not improve software quality. Test results are an indicator of quality, but in and of themselves, they don’t improve it. Trying to improve software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often. What you eat before you step onto the scale determines how much you will weigh, and the software development techniques you use determine how many errors testing will find. If you want to lose weight, don’t buy a new scale; change your diet. If you want to improve your software, don’t test more; develop better.” The same goes about many other procedures companies try to employ, they usually increase company’s bureaucracy and expenses without improving software quality.
November 22nd, 2008 at 6:50 pm
When I was in the game industry, they actually had a tongue-in-cheek name for this: “In-Market Testing”. In the waterfall flow, testing (and bug fixing) came last so it simply got cut if the project fell behind schedule. Buggy releases ensued. Fix-as-you-go makes much more sense and should result in less QA time required just before release.
November 22nd, 2008 at 9:08 pm
IMHO, many software companies don’t even know the problem why their software is buggy so it’s not intentional.
Several reasons:
1. Business factor - Customer often wants to keep their cost low squeezing the vendor while some vendors tried to accommodate it by rushing SDLC to keep their time/cost low thus neglecting the quality issues. Vendors just want to get the dirty job done with a ‘hit and run’ mindset and expects the client to sign a maintenance contract for bug fixes later.
2. Management problem - When a company hires people who does not have enough technical experience or software development knowledge on top, problems arise. Some project managers are hired simply because they can “talk” not because they are good. When it comes to managing a software project, will do things their own way outside proper engineering process and push the developers to get things done quick and dirty. Usually these people do not even understand why and how doing so will result the cost to fix or maintain it later is even higher. After all, they will still get appreciation from top management while developers are often blamed for buggy software. Moreover, when developers who are not satisfied with such people managing a software project, they will leave and the cycle repeats. A software project that is moved into too many different developers before it is mature will often result into spaghetti code that is not maintainable and buggy!
3. Bad engineering practice/mindset - There are some technical architects and developers who don’t even practice common unit testing at all and think very short term. I notice a lot of developers nowadays shifted their design effort mindset to 3:3:4 (design:code:debug) approach rather than 6:3:1 (design:code:debug) approach. This is bad especially if they just wanted to show they have done something to the management in early stage but in long run, such practice is not sustainable.
In short, business decision and getting the right people from top to bottom does make a big impact towards the software quality of a company.
November 23rd, 2008 at 10:53 am
All the answers have covered most of the areas / possibilities of defect(s) cropping. This starts from getting an order with final delivery date - (various other phases) - the code written by a programmer - (various other phases) - Testing phase - (various other phases) - Final delivery phase.
Causes for a defects are project duration, skills and talents of resources at various department, Cost/price of the project/product etc. But all these defects are human dependent. By making entire project system dependent (following process, automated testing etc.), to some extent defects can be reduced, but not 100%.
Inspite of following process, programming methodologies, best practices, I would like to highlight one of the best practice (incase of product development), i.e. code re-factoring at regular intervals. Every design has lifecycle, the design / architecture has to be modified or tweaked at regular intervals. Again code re-factoring at various levels from high level design to low design, when ever and where ever required.
Also I would like to high light one more factor, Quality = Time = Price. When ever there is a cut down in time or cost beyond limit, there will be some amout of quality degradation. The reason for the reduction, is either client want final delivery with so and so date and/or vendor want to save cost also or deliver withing agreed date (as there is huge amount of compitition). So any reduction in the time or price beyond range with fewer / less skilled resource has an impact on the final quality.
November 23rd, 2008 at 7:36 pm
Fixed price, Fixed scope so what’s left to vary ?
Quality.
It’s that simple.
November 23rd, 2008 at 8:05 pm
I think the quality of software (and not only software) became really bad lately because business model has changed drastically in the last 10 years. Software developers are considered as commodity, employers are not interested in retaining talents. On contrary, they dump talents in big numbers as soon as “cheaper commodity” is found anywhere. There are at least two major problems that are associated with this approach:
1. The quality of “cheaper commodity” is much worse in many cases
2. Employees do not have any bonds with their employers either and are ready to leave at first convenience. They are not interested to deliver high quality product. QA is not able to find all potential problems (e.g. scalability issues, race conditions, memory leaks, etc.). In other words, software engineers have a “contractor mentality” nowadays that hurts the quality of software big time.
All other issues that have been discussed in this thread have been valid in the past as well. The trend that I’ve described is relatively new.
November 23rd, 2008 at 9:36 pm
I completely go with what Oleg has to say. Besides, how skilled are the Quality resources themselves?? What is the percentage of defect leak attributed to testers? As high as 10-20% defect leak into production environment seems to be a norm with pressures to cut down the costs at “any cost”!
November 23rd, 2008 at 10:13 pm
Bugs are created because building software is hard. All the easy things have already been built, complexity of systems keeps increasing.
At the same time employers are looking for cheapest (rather than best) talent. Most of software engineering processes are “evil” - i.e. cause even good people to produce bad work. For example, people are rewarded for individual achievement when it takes coordinated work of a team to build a “knowledge product” like software.
Buggy products are released because customers accept buggy software - being the first to market beats being bug-free hands down. Big part of that is currently standard ULA - the purchaser/user of software releases the maker from any responsibility for any problems caused by the software.
November 24th, 2008 at 5:41 am
I doubt bugs are intentional, only that vendors are faced with other factors more important like those already mentioned. Gaining clients, meeting deadlines are more pressing than bugs that are either not properly screened or deemed less important than the major features already committed to consumers. Often these software products come out of the door with some level of commitment/usability to intended users who could tolerate certain percentage of error.
November 24th, 2008 at 7:33 am
I agree with Darren´s reasoning to some extent. However, a software house can find themself in a very undesirable position if they increase their technical debt for a number of iterations without paying their mortgages.
Often it is possible to hide poor internal quality from the customer for some time, usually until some kind of tipping point is reached, can be both technical or organisational such as staff replacement. Then the facade starts cracking up, bug after bug shows up, the users are discontent, quick fixes are put into the system, technical debt increases even more, bug fixes introduces more bugs etc. Not a nice position to be in.
My point is, when looking for the quality we need in a business perspective, we must also be pro-active, and not only see to that the users are happy today, but that they are still happy tomorrow, day after that and next week.
November 24th, 2008 at 8:16 am
I don’t think bugs are introduced intentionally for most of the companies but there are some companies I’ve seen that they realize the is