Software Testing in the Real World: Improving the Process

Software testing.\ It is exhilarating. It is exasperating.\ It can be the best of jobs—or the worst.\ It is fundamental to delivering quality software on time within budget.\ \ This book is for anyone interested in improving their test process, including test specialists, product developers, managers, quality assurance specialists, maintainers, project managers, auditors, systems analysts, tool developers, and others whose jobs involve understanding software testing. It offers developers and other people whose work is not primarily in the testing area an opportunity to improve their own specialty through a better understanding of testing.\ To understand testing and to improve testing practice, it is essential to see the software testing process in its broadest terms—as the means by which people, methodology, tools, measurement, and leadership are integrated to test a software product. More than 20 years in testing practice, teaching, and consulting have shown me the following key questions raised again and again by people wanting to improve their testing process:\ \ Methodology questions\ What are the key testing methods, when do we use them, and how do we implement them?\ How do we prevent defect migration and improve defect finding capabilities?\ How can we understand and leverage the relevant test standards and terminology?\ How do we create meaningful test plans and other critical test deliverables?\ How do we identify and prioritize potential software testing process improvements?\ \ \ Leadership questions\ How do we organize the software testing effort so it is measurable and controllable?\ Where can we find additional help regarding software testing?\ How can we ensure that improvements are sustainable in the medium and long term?\ How can we organize testing for the most effective relations between people?\ How can we generally maximize exhilaration and minimizeexasperation?\ \ \ Tools and technology questions\ What are the most significant opportunities for automating the testing process?\ How can we use measurement to understand and control the software test process?\ How can we select test tools and vendors and implement an effective tools program?\ How can we use risk analysis techniques as a basis for decision making in testing?\ \ My experience has shown that a balanced strategy addressing methodology, leadership, and technology is most effective for improving testing.\ Most software development organizations suffer from immature software processes. Given this starting point, practitioners and managers naturally ask: "Yes, but what do I do now?". This book offers a tool-box for effectively improving the software testing process. A tool-box is not a single methodology to be followed from A to Z. It is a menu of techniques that can be taken individually or in combination to provide the means to formulate and reach realistic improvement objectives.\ Part I is for orientation. It describes the six essentials of software testing, the history of testing, and a simple, practical approach to getting started with improving the testing process. It identifies software engineering in general and testing in particular as new disciplines grappling with the escalating demands of an environment expecting miracles.\ Part II establishes practical goals from a "now we are here" position, and explains how these relate to formal and less formal definitions of testing and testing objectives, and to the current standards that are fundamental (and useful) to practitioners in their everyday work.\ The basic forms of the testing process are explained, and the critical real-world issues of cost and risk are fully clarified and integrated in the decision making process of what, when and how to test. In this part we address the items that need to be in place before we dig more deeply into the testing activities of verification and validation. The issues of planning, software engineering maturity goals, configuration management, standards, and tools are explained and positioned.\ Part III explains the basic verification and validation testing tasks, including planning and controlling testing costs, and making the best use of resources. The use of tools as a way of gaining leverage in the various testing activities is explained, together with clarification of tool categories, and practical guidance on the critical questions for tool acquisition. The appropriate and constructive use of measurement is explained, and guidelines given for the proper use of measurement results.\ Part IV explains the organizational and human management issues that surround the test effort and which are key determinants of its success. It surveys some current best practice sources, and then refocuses on how to get sustainable improvements in place in the short and medium term.\ Definitions, references, and examples are integrated within the main text of the book as far as possible. The appendices contain a variety of material that is too detailed for inclusion in the main text of the book, and which is essentially complementary. This includes sample checklists, exercises, planning documents, and sample working documents. There is further detail on official standards and specifications as well as useful help sources such as journals, tools, conferences, newsletters, user groups, and an annotated bibliography.\ Acknowledgments\ Life provides many teachers. Some say all of life is our teacher. Space prohibits acknowledging all who have positively influenced this book, but does allow acknowledgment of the following people due special appreciation.\ Great appreciation is due my wife, Cathy Baldwin, who not only provided continuous moral support and endless editing of the book, but also provided encouragement during the early start-up years of Software Development Technologies. Special thanks to our families for their support over the years.\ I am particularly grateful to David Gelperin and Bill Hetzel: to David for providing the opportunity to teach my first course on software testing nearly a decade ago, and for providing the ongoing friendship and support that kept me going; to Bill for encouraging me to write a book and forwriting the foreword for the book; and to both of them for providing the opportunity to be associated with Software Quality Engineering for many years.\ Appreciation is due Chris Larson, who for 15 years has generously shared his friendship and wealth of software engineering experiences gained from over 25 years with Hewlett Packard, IBM, Tandem, and CDC, and who contributed enormously to the course on which this book is based.\ Appreciation is also due Susannah Finzi for being an outstanding friend and editor, and for keeping the book appealing from a global perspective while patiently enduring seemingly endless rewrites without a single complaint.\ Appreciation is due the reviewers who gave their time to providevaluable comments, especially Bill Hetzel, David Gelperin, Dot Graham, Marty Gipson, Vijay Sarathy, Ray O'Neal, Ellen Brodbine, Trevor Reeve, Cathy Baldwin, Chris Larson, Denise Leigh, and Paul Claydon.\ Finally, appreciation is due to the students who attended the courses upon which this book is based and who in the process became my teachers and provided suggestions for improving the course, especially the participants from Hewlett Packard, Motorola, Microsoft, Apple Computer, US Army, IBM, NCR, Johnson & Johnson, Grumman Data Systems, Australia and New Zealand Bank, ADP, EDS, PeopleSoft, SCO, General Electric, Oracle, NEC, Lloyds Bank, USDA, NetFrame, Cadence, Sears, AT&T, Informix, British Aerospace, Sybase, Tandem, and Octel Communications.\ Ed Kit Cupertino, September 1995

ForewordPrefaceCh. 1The six essentials of software testing3Ch. 2The state of the art and the state of the practice7Ch. 3The clean-sheet approach to getting started12Ch. 4Establishing a practical perspective17Ch. 5Critical choices: what, when, and how to test26Ch. 6Critical disciplines: frameworks for testing36Ch. 7Verification testing57Ch. 8Validation testing77Ch. 9Controlling validation costs110Ch. 10Testing tasks, deliverables, and chronology118Ch. 11Software testing tools144Ch. 12Measurement154Ch. 13Organizational approaches to testing163Ch. 14Current practices, trends, challenges176Ch. 15Getting sustainable gains in place181Appendices187A Standards relevant to software engineering and testing189B Verification checklists194C Verification exercises210D Validation exercises (solutions)225E Bibliography228F Sources: conferences, journals, newsletters, DOD specifications232G Specific tools and tool selection235H Sample lists of improvements to be implemented239Index243

\ From Barnes & Noble\ \ Fatbrain Review\ Focuses on the testing process and improvements to it. An entire section is devoted to working with an "immature organization" seeking to improve testing in the here and now, and to put structures in place for ongoing improvements. Questions regarding methodology, leadership, and tools & technology are responded to with practical solutions. E.g. "How do we identify and prioritize potential software testing process improvements?" "How do we organize the software testing effort so it is measurable and controllable?" "What are the most significant opportunities for automating the testing process?" Addresses defect migration, quality of the process, testing tools, taking responsibility and more. This is an ACM Press book.\ \ \ \ \ BooknewsFor test and quality specialists, project managers, and developers of software, presents techniques and strategies for improving the process of testing software, and for maintaining a sustainable improvement of the process within the organization. Acknowledging the relative immaturity of software engineering, addresses such issues as cost and risk, standards, planning, and tasks and tools for testing. Annotation c. Book News, Inc., Portland, OR (booknews.com)\ \