Software Test Automation: Effective Use of Test Execution Tools
How to structure and build an automated testing regime that will give lasting benefits.\ Software testing is vitally important in the software development process, as illustrated by the growing market for automated testing tools. However, many attempts to automate software testing are not successful.\ At first glance, it seems easy to automate testing: just buy one of the popular test execution tools, record the manual tests, and play them back whenever you want to. Unfortunately, as those...
Search in google:
How to structure and build an automated testing regime that will give lasting benefits. Software testing is vitally important in the software development process, as illustrated by the growing market for automated testing tools. However, many attempts to automate software testing are not successful. At first glance, it seems easy to automate testing: just buy one of the popular test execution tools, record the manual tests, and play them back whenever you want to. Unfortunately, as those who tried it have discovered, it doesn't work like that in practice. Just as there is more to software design than knowing a programming language, there is more to automating testing than knowing a testing tool. "This first comprehensive treatment of software test automation provides the equivalent of 2 or 3 years of on the job experience." James Bach, Test Design Consultant "The most authoritative book on this subject - a 'must read' for every software testing professional." Jeffrey M. Voas, Chief Scientist, Reliable Software Technologies "This book will teach you how to make automated testing tools useful." From the Foreword, by Brian Marick Features: A detailed introduction to the principles of automated testing Advice on choosing and implementing software testing tools Explanation of why capture replay is not test automation Practical techniques for designing a good automated testing regime Detailed methods for increasing automation maturity Example-based illustration of ideas in practice Practical solutions to frequently encountered problems in test automation Case studies of test automation experience in a variety of organizations (including Microsoft) Advice from test automation experts and practitioners world-wide Instruction on how to make it easier to add a new automated test than to run it manually Booknews Describes how to structure and build an automated testing regime that will give lasting benefits in the use of test execution tools to automate testing on a medium to large scale. Offers practical advice for selecting the right tool and for implementing automated testing practices within an organization, and presents an extensive collection of case studies and guest chapters reflecting both good and bad experiences in test automation. Useful for recent purchasers of test automation tools, technical managers, vendors, and consultants. The authors are consultant partners in a company that provides consultancy and training in software testing and test automation. Annotation c. Book News, Inc., Portland, OR (booknews.com)
Preface Part One: Techniques for Automating Test Execution 1 Test automation context 1.1 Introduction 1.2 Testing and test automation are different 1.3 The V-model 1.4 Tool support for life-cycle testing 1.5 The promise of test automation 1.6 Common problems of test automation 1.7 Test activities 1.8 Automate test design? 1.9 The limitations of automating software testing 2 Capture Replay is Not Test Automation 2.1 An example application: Scribble 2.2 The manual test process: what is to be automated 2.3 Automating Test Execution: inputs 2.4 Automating Test Result Comparison 2.5 The next steps in evolving test automation 2.6 Conclusion: Automated is not automatic 3 Scripting techniques 3.1 Introduction 3.2 Scripting techniques 3.3 Script pre-processing 4 Automated comparison 4.1 Verification, comparison and automation 4.2 What do comparators do? 4.3 Dynamic comparison 4.4 Post-execution comparison 4.5 Simple comparison 4.6 Complex comparison 4.7 Test sensitivity 4.8 Comparing different types of outcome 4.9 Comparison filters 4.10 Comparison guidelines 5 Testware Architecture 5.1 What is testware architecture? 5.2 Key issues to be resolved 5.3 An Approach 5.4 Might this be Overkill? 6 Automating Pre- and Post-Processing 6.1 What are Pre- and Post-Processing? 6.2 Pre- and Post Processing 6.3 What should happen after test case execution 6.4 Implementation Issues 7 Building maintainable tests 7.1 Problems in maintaining automated tests 7.2 Attributes of test maintenance 7.3 The conspiracy 7.4 Strategy and tactics 8 Metrics 8.1 Why measure testing and test automation? 8.2 What can we measure? 8.3 Objectives for testing and test automation 8.4 Attributes of software testing 8.5 Attributes of test automation 8.6 Which is the best test automation regime? 8.7 Should I really measure all these? 8.8 Summary 8.9 Answer to DDP Exercise 9 Other Issues 9.1 Which Tests to Automate (first)? 9.2 Selecting which tests to run when 9.3 Order of test execution 9.4 Test status 9.5 Designing software for (automated) testability 9.6 Synchronization 9.7 Monitoring progress of automated tests 9.8 Tailoring your own regime around your tools 10 Choosing a tool to automate testing 10.1 Introduction to Chapters 10 and 11 10.2 Where to start in selecting tools: your requirements, not the tool market 10.3 The tool selection project 10.4 The tool selection team 10.5 Identifying your requirements 10.6 Identifying your constraints 10.7 Build or buy? 10.8 Identifying what is available on the market 10.9 Evaluating the short listed candidate tools 10.10 Making the decision 11 Implementing tools within the organization 11.1 What could go wrong? 11.2 Importance of managing the implementation process 11.3 Roles in the implementation/change process 11.4 Management commitment 11.5 Preparation 11.6 Pilot project 11.7 Planned phased installation or roll-out 11.8 Special problems in implementing 11.9 People issues 11.10 Conclusion 12 Racal-Redac Case History 12.1 Introduction 12.2 Background 12.3 Solutions 12.4 Integration to Test Automation 12.5 System Test Automation 12.6 The Results Achieved 12.7 Summary of the case history up to 1991 12.8 What happened next? 13 The Evolution of an Automated Software Test System 13.1 Introduction 13.2 Background 13.3 Gremlin 1 13.4 Gremlin 2.0: A Step Beyond Capture/Replay 13.5 Finding The Real Problem 13.6 Lesson Learned 14 Experiences with Test Automation 14.1 Background 14.2 Planning, preparation and eventual success 14.3 Benefits of test automation 14.4 Lessons learned 14.5 The way forward 15 Automating System Testing in a VMS Environment 15.1 Background 15.2 The first attempt at automation 15.3 New tool selection and evaluation 15.4 Implementation of V-Test 15.5 Conclusion 16 Automated Testing of an Electronic Stock Exchange 16.1 Background 16.2 The System and Testing 16.3 Test Automation Requirements 16.4 Test tool selection 16.5 Implementation 16.6 Maturity and Maintenance 16.7 Our results 17 Insurance quotation systems tested automatically every month 17.1 Background: the UK insurance industry 17.2 The Brief, or how I became involved 17.3 Why automation? 17.4 Our testing strategy 17.5 Selecting a test automation tool 17.6 Some decisions about our test automation plans 17.7 The Test Plan 17.8 Some additional issues we encountered 17.9 A telling tale: tester versus automator 17.10 In Summary 18 Three Generations of Test Automation at ISS 18.1 Introduction 18.2 The Software Under Test 18.3 First Generation 18.4 Second Generation 18.5 Third Generation 18.6 Three Generations - A summary 19 Test Automation Failures - Lessons to be Learning 19.1 Introduction 19.2 The projects 19.3 Problems 19.4 Recommendations 19.5 Pilot Project 19.6 Epilogue 20 An Unexpected Application of Test Automation 20.1 Introduction and Background 20.2 Helping the Background 20.3 Doing the testing 20.4 Automated Testing 20.5 The results 21 Implementing test automation in an Independent Test Unit 21.1 Introduction and Background 21.2 The evaluation process 21.3 The implementation phase 21.4 The Deployment of the tool 21.5 How QARun has been used 21.6 Problems we have experienced 21.7 The benefits achieved in two years 21.8 Conclusion 22 Testing with Action Words 22.1 Introduction and Background 22.2 Test clusters 22.3 The navigation 22.4 The test development life cycle 22.5 Applicability for other types of tests 22.6 Templates: meta clusters 23 Regression testing at ABN AMRO Bank Development International 23.1 Background 23.2 Problems with conventional testing 23.3 Pilot project using TestFrame 23.4 Regression test project 23.5 Spin-offs 23.6 Future 24 A Test Automation Journey 24.1 Introduction 24.2 The five generations of testware development 24.3 Radstar 24.4 Window-centric Scenario Libraries 24.5 Business Object Scenarios 24.6 Mixing Business Object Scenarios with existing tests 24.7 Re-use versus Repeatability 24.8 Conclusion 25 A Test Automation Journey 25.1 Introduction 25.2 First Steps 25.3 An Off-the-Shelf Automated Test Foundation: RadStar( 25.4 How we have implemented automated testing using RadStar( 25.5 Payback 26 Extracts from The Automated Testing Handbook 26.1 Introduction to this chapter 26.2 Introduction to the Handbook 26.3 Fundamentals of Test Automation 26.4 Test Process and People 26.5 Test Execution: Analyzing Results 26.6 Test Metrics 26.7 More information about the Handbook 27 Building Maintainable GUI Tests 27.1 Introduction and Background 27.2 Cost Drivers 27.3 Test Planning and Design 27.4 Well Behaved Test Cases 27.5 Encapsulated Test Set-up 27.6 Putting it All Together 28 Test Automation Experience at Microsoft 28.1 History 28.2 Batch Files 28.3 Capture/Playback tools 28.4 Scripting Language 28.5 Cosmetic Dialog Box Testing 28.6 Help testing tool 28.7 Tools to randomize test execution 28.8 What should I automate first? 28.9 My Top Ten list for a successful test automation strategy References Glossary Index
\ From the Publisher“This is the first comprehensive treatment of software test automation issues, strategies and tactics ever published. It provides the equivalent of two or three years of on the job experience. Every important aspect of test automation is covered, with enough information to help the reader approach the subject with the right balance of caution and confidence. I'm delighted with what Fewster & Graham have done.”\ —James Bach, Test Design Consultant\ “The most authoritative book available on this subject, a must read for every software testing professional!”\ —Jeffrey M. Voas, Chief Scientist, Reliable Software Technologies, VA\ \ \ \ \ \ BooknewsDescribes how to structure and build an automated testing regime that will give lasting benefits in the use of test execution tools to automate testing on a medium to large scale. Offers practical advice for selecting the right tool and for implementing automated testing practices within an organization, and presents an extensive collection of case studies and guest chapters reflecting both good and bad experiences in test automation. Useful for recent purchasers of test automation tools, technical managers, vendors, and consultants. The authors are consultant partners in a company that provides consultancy and training in software testing and test automation. Annotation c. Book News, Inc., Portland, OR booknews.com\ \
![Advanced Windows Debugging [Addison-Wesley Microsoft Technology Series]](/application/data/covers/44/62/9780321374462.jpg "Advanced Windows Debugging [Addison-Wesley Microsoft Technology Series]")
