Home

Rapid Test Planning - Cem Kaner, JD, Ph.D.

1. Inthe second column list all the values of the variable skip the line list the values etc For example if variable 2 s possible values are X Y then the table looks like this so far Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 93 Combination Testing Building an all pairs combination table Each section of the third column think of AA as defining a section BB as defining another etc will have to contain every value of variable 3 Order the values such that the variables also make all pairs with variable 2 Suppose variable 2 can be 1 0 The third section can be filled in either way and you might highlight it so that you can reverse it later The decision say 1 0 is arbitrary Now that we ve solved the 3 column exercise let s try adding more variables Each of them will have two R UME Sanning Copyright 2001 Cem Kaner and James Bach All rights reserved 94 Combination Testing The 4th column went in easily note that we started by making sure we hit all pairs of values of column 4 and column 2 then all pairs of column 4 and column 3 Watch this first attempt on column 5 We achieve all pairs of GH with columns 1 2 and 3 but miss it for column 4 The most recent arbitrary choice was HG in the 2nd section Once that was determined we picked HG for the third in order to pair H with a 1 in the third column eSo we will erase th
2. Our working definition of equivalence Two test cases are equivalent if you expect the same result from each This is fundamentally subjective It depends on what you expect And what you expect depends on what errors you can anticipate Two test cases can only be equivalent by reference to a specifiable risk Two different testers will have different theories about how programs can fail and therefore they will come up with different classes A boundary case in this system is a best representative A best representative of an equivalence class is a test that is at least as likely to expose a fault as every other member of the class Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 122 Domain Testing OO e Strengths Find highest probability errors with a relatively small set of tests Intuitively clear approach generalizes well e Blind spots Errors that are not at boundaries or in obvious special cases Also the actual domains are often unknowable Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 123 Domain Testing Interesting Papers ee e Thomas Ostrand amp Mark Balcer The Category partition Method For Specifying And Generating Functional Tests Communications of the ACM Vol 31 No 6 1988 e Debra Richardson et al A Close Look at Domain Testing IEEE Transactions On Software Engineering Vol SE 8 NO 4
3. Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 3 0 Product Elements A product is Loo as sl t s s st An experience or solution provided to a custom r Everything that comes in the box plus the bo Functions and data executed on a platfor that serve a purpose for a user 1 A software product is much more than code 2 It involves a purpose platform and user 3 It consists of many interdependent elements Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 3 l Product Elements _ _ a_ sas OO e Structures Everything that comprises the physical product Code the code structures that comprise the product from executables to individual routines Interfaces points of connection and communication between subsystems Hardware hardware components integral to the product Non executable files any files other than programs such as text files sample data help files etc Alternate Media anything beyond software and hardware such as paper documents web links and content packaging license agreements etc Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 32 Product Elements oo e Functions Everything that the product does User Interface functions that mediate the exchange of data with the user System Interface functions that exchange data with something other than th
4. Test4 Pass Fail Fail Pas Test5 Fail___ Pass_ Fail_ Pass Pass This matrix records the results of a series of tests against the 6 standard configurations that we defined in the Configuration Planning Table In this table Config 1 has passed 3 tests failed 1 and hasn t yet been tested with Test 2 Config 6 is still untested Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 86 Testing Variables in Combination TT Interesting papers Cohen Dalal Parelius Patton The Combinatorial Design Approach to Automatic Test Generation IEEE Software Sept 96 http computer org 80 software so1996 sStoc htm Cohen Dalal Fredman Patton The AETG System An Approach to Testing Based on Combinatorial Design IEEE Trans on SW Eng Vol 23 7 July 97 http computer org 80 tse ts 1997 e7toc htm OnLine requires IEEE membership for free access See http www computer org epub Several other papers on AETG are available at https aetgweb tipandring com AboutAETGweb html Also interesting http www stsc hill af mil CrossTalk 1997 oct planning html Jorgenson Software Testing A Craftsman s Approach Brian Marick Multi Generating test ideas from expressions with booleans and relational operators http www testing com tools multi README html Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 87 Ye The
5. the number of possible combinations is 12 3 x 2 x 2 Building a simple combination table Label the columns with the variable names listing variables in descending order of number of possible values Each column before the last will have repetition Suppose that A B and C are in column K of N columns To determine how many times rows in which to repeat A before creating a row for B multiply the number of variable values in columns K 1 K 2 N Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 9 1 Combination Testing O Building an all pairs combination table Label the columns with the variable names listing variables in descending order of number of possible values Ifthe variable in column 1 has V1 possible values and the variable in column 2 has V2 possible values then there will be at least V1 x V2 rows draw the table this way but leave a blank row or two between repetition groups in column 1 Fill in the table one column at a time The first column repeats each of its elements V2 times skips a line and then starts the repetition of the next element For example if variable 1 s possible values are A B C and V2 is 2 then column would contain A A blank row B B blank row C C blank row Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 92 Combination Testing O Building an all pairs combination table
6. 75 Routine Case Matrices CeP NNTTrTYxY T You can often re use a matrix like this across products and projects You can create matrices like this for a wide range of problems Whenever you can specify multiple tests to be done on one class of object and you expect to test several such objects you can put the multiple tests on the matrix Mark a cell green if you ran the test and the program passed it Mark the cell red if the program failed Write the bug number of the bug report for this bug Write in the cell the automation number or identifier or file name if the test case has been automated Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 76 Routine Case Matrices V PNnNVNY PNrr e Problems What if your thinking gets out of date What if this program poses new issues not covered by the standard tests Do you need to execute every test every time or ever What if the automation ID number changes We still have a maintenance problem but it is not as obscure Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved T1 Complex Data Relationships Options Viet General Edik Print Save Spelling Grammar Track Changes User Information Compatibilik _ File bocation Sompatibilicy options For Document Font Substitution s re icra OF E ord oy prions Combine tabl
7. Kaner and James Bach All rights reserved 66 Myers Boundary Table Variable Valid Case Invalid Case Boundaries Equivalence Equivalence and Special Classes Classes 99 to 99 gt 99 lt 99 non number expressions Second same as first sameasfirst same number 198 to 198 Are there other sources of data for this variable Ways to feed it bad data Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 67 Revised Boundary Analysis Table Equivalence Alternate Boundaries Equivalence and Special Class 99 to 99 gt 99 lt 99 99 100 digits non digits 0 9 leading spaces or Os expressions Null entry Second same as first same as first same number 198 to 198 22 2 Are there other 127 to 127 198 to 128 127 128 127 Sources of data for 128 to 198 128 this variable Ways to feed it bad data Note that we ve dropped the issue of valid and invalid This lets us generalize to partitioning strategies that don t have the concept of valid for example printer equivalence classes Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 68 Equivalence Classes A Broad Concept OO The notion of equivalence class is much broader than numeric ranges Here are some examples Membership in a common group e such as employees vs non employees Note that not all classes have shar
8. Satisfice Heuristic Test Strategy Model at http www satisfice com tools satisfice tsm 4p pdf You can assume not always correctly but usually that every sentence in the spec is meant to convey information The information will probably be about e the project and how it is structured funded or timed or e about the product what it is and how it works or e about the quality criteria that you should evaluate the product against 129 Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Scenario Testing _ s ww O Tag lines Do something useful and interesting Do one thing after another Fundamental question or goal Challenging cases that reflect real use Paradigmatic case s Appraise product against business rules customer data competitors output Life history testing Hans Buwalda s soap opera testing Use cases are a simpler form often derived from product capabilities and user model rather than from naturalistic observation of systems of this kind Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 3 0 Scenario Testing rr e The ideal scenario has several characteristics It is realistic e g it comes from actual customer or competitor situations There is no ambiguity about whether a test passed or failed The test is complex that is it uses several f
9. User testing Beta testing Subject matter experts Coverage e Function testing Domain testing State based testing Path testing Statement coverage Configuration coverage Potential problems e Input output computation storage constraints Risk based testing Activities e Exploratory testing Scenario testing Load testing Performance testing Evaluation e Oracle based testing Comparison with saved results e These examples are not definitive how you classify a testing approach depends on what you think is most central to it For example is load testing problem oriented denial of service or activity oriented e The important thing is to conscious manage the 5 dimensions Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 56 General Test Techniques Oe e Function e Regression e Domain driven e Stress driven e Specification driven e Risk driven e Scenario use case transaction flow e User testing e Exploratory e Random statistical All of these have been used as the dominant technique in some companies How can approaches so different yield good overall results We think that the answer is that each of these fixes only one of the dimensions for testing techniques For example function testing speaks to coverage but not to testers risks activities or evaluation You can vary all four of these and still be doing function test
10. and James Bach All rights reserved 18 Context Free Questions Defining the Problem _ OOO Where are the boundaries of the problem What product elements does it apply to How does this problem relate to the quality criteria Can you separate the various parts of the problem Can you write them down What are the relationships of the parts of the problem What are the constants things that can t be changed of the problem What are your critical assumptions about this problem Have you seen this problem before Have you seen this problem in a slightly different form Do you know a related problem Try to think of a familiar problem having the same or a similar unknown Suppose you find a problem related to yours that has already been solved Can you use it Can you use its method Can you restate your problem How many different ways can you restate it More general More specific Can the rules be changed What are the best worst and most probable cases you can imagine Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 19 Context Free Questions Doo o Context free process questions A jee Who is the client sampile o What is a successful solution worth to this client patentee What is the real underlying reason for wanting to solve this rs sed on problem Gause amp Who can help solve the
11. e Whittaker amp Jorgenson How to Break Software Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 126 Specification Driven Testing Oooo e Tag line Verify every claim e Fundamental question or goal Check the product s conformance with every statement in every spec requirements document etc e Paradigmatic case s Traceability matrix tracks test cases associated with each specification item User documentation testing Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 127 Specification Driven Testing Oooo e Strengths Critical defense against warranty claims fraud charges loss of credibility with customers Effective for managing scope expectations of regulatory driven testing Reduces support costs customer complaints by ensuring that no false or misleading representations are made to customers e Blind spots Any issues not in the specs or treated badly in the specs documentation Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 128 Reviewing a Specification for Completeness _ s_ as _ OO Reading a spec linearly is not a particularly effective way to read the document It s too easy to overlook key missing issues eWe may not have time to walk through this method in this class but the general approach that use is based on James Bach s
12. for the test strategy to be organized around functionality or code leaving it to the testers to concoct test data on the fly Often that indicates that the strategy is too focused on validating capability and not focused enough on reliability Heuristics for Test Plan Evaluation Coo eee 6 Not all testing should be pre A rigid test strategy may make it more specified in detail The test strategy likely that a particular subset of problems should incorporate reasonable variation will be uncovered but in a complex and make use of the testers ability to system it reduces the likelihood that all use situational reasoning to focuse on important problems will be uncovered important but unanticipated problems Reasonable variability in testing such as that which results from interactive exploratory testing increases incidental test coverage without substantially sacrificing essential coverage 7 It is important to test against implied Testing only against explicit written requirements the full extent of what requirements will not reveal all important the requirements mean not just what problems since defined requirements they say are generally incomplete and natural language is inherently ambiguous Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 03 Heuristics for Test Plan Evaluation TT 8 The test project should promote Other teams and stakeholders often collabo
13. mid project What does this do to the net quality of the test documentation and test planning effort e WHAT REQUIREMENTS DOES A STANDARD LIKE THIS FULFILL e WHICH STAKEHOLDERS GAIN A NET BENEFIT FROM IEEE STANDARD DOCUMENTATION e WHAT BENEFITS DO THEY GAIN AND WHY ARE THOSE BENEFITS IMPORTANT TO THEM Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Problems with the allegedly standard approach Loo OO It is essential to understand your requirements for test documentation Unless following a standard helps you meet your requirements it is empty at best anti productive at worst Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 9 Requirements OOO There are many different notions of what a good set of test documentation would include Before spending a substantial amount of time and resources it s worth asking what documentation should be developed and why Test documentation is expensive and it takes a long time to produce If you figure out some of your main requirements first you might be able to do your work in a way that achieves them Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 0 Defining documentation requirements _ _ s OO e Stakeholders interests actions objects Who would use or be affected by test documentation What interests of theirs does do
14. of tests or an algorithm for creating a list a list of diagnostics initial diagnostics at start of testing diagnostics at start of each test diagnostics on detected error and diagnostics at end of session an initial configuration and a list of configuration changes on specified events e The tool runs the tests in random order and outputs results to a standard format log file that defines its own structure so that multiple different analysis tools can interpret the same data Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 60 Random Statistical Testing Loo o e Strengths Regression doesn t depend on same old test every time Partial oracles can find errors in young code quickly and cheaply Less likely to miss internal optimizations that are invisible from outside Can detect failures arising out of long complex chains that would be hard to create as planned tests e Blind spots Need to be able to distinguish pass from failure Too many people think Not crash not fail Executive expectations must be carefully managed These methods will often cover many types of risks but will obscure the need for other tests less amenable to automation Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 6 1 Random Statistical Testing O e Blind spots Testers might spend much mo
15. AETG System An Approach to Testing Base Ble Eat Mew Go Cormunicatar H a Bookmarks Be Wetene https aetqweb tipandring com papers AET Gieeed html The AETG System An Approach to Testing Based on Combinatorial Design on Combinatorial Design Netscape e Appeared in July 1997 issue of IEEE Transactions On Software Engineering Vol 23 No 7 By D M Cohen IDA CCS Work done while at Bellcore S R Dalal Bellcore M L Fredman Rutgers University gt gt Patton Bellcore TABLE OF CONTENTS Abstract 1 Introduction 2 The Basic Combinatorial Design Paradigm 3 Logarithmic Growth for n Way Interaction Testing 4 5 A Heuristic Algorithm AETG Input Language 5 1 Constraints 5 2 Hierarchy and hierarchical testing Experiments Overview of Applications A 7 1 High Level Test Planning 7 2 Test Case Generation amp Related Methods 9 Summary Acknowledgements References Abstract This paper describes a new approach to testing that uses combinatorial designs to generate tests that cover the pair wise triple or n way combinations of a system s test parameters These are the parameters that determine the system s test scenarios Examples are system configuration parameters user inputs and other external events We implemented this new method in the AETG w ar _ ee Copyright c 1997 1999 Cem Kaner All
16. General approach Divide the set of possible values of a field into subsets pick values to represent each subset Typical values will be at boundaries More generally the goal is to find a best representative for each subset and to run tests with these representatives Advanced approach combine tests of several best representatives Several approaches to choosing optimal small set of combinations e Paradigmatic case s Equivalence analysis of a simple numeric field Printer compatibility testing multidimensional variable doesn t map to a simple numeric field but stratified samplin is essential 120 Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Domain Testing Loo o e In classical domain testing Two values single points or n tuples are equivalent if the program would take the same path in response to each e The classical domain strategies all assume that the predicate interpretations are simple linear inequalities the input space is continuous and coincidental correctness is disallowed It is possible to move away from these assumptions but the cost can be high and the emphasis on paths is troublesome because of the high number of possible paths through the program e Clarke Hassell amp Richardson p 388 12 Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Equivalence and Risk OO
17. July 1982 e Michael Deck and James Whittaker Lessons learned from fifteen years of cleanroom testing STAR 97 Proceedings in this paper the authors adopt boundary testing as an adjunct to random sampling e Richard Hamlet amp Ross Taylor Partition Testing Does Not Inspire Confidence Proceedings of the Second Workshop on software Testing Verification and Analysis IEEE Computer Society Press 206 215 July 1988 Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 124 Stress Testing OOOO e Tag line Overwhelm the product e Fundamental question or goal Learn about the capabilities and weaknesses of the product by driving it through failure and beyond What does failure at extremes tell us about changes needed in the program s handling of normal cases e Paradigmatic case s Buffer overflow bugs High volumes of data device connections long transaction chains Low memory conditions device failures viruses other crises e Strengths Expose weaknesses that will arise in the field Expose security risks e Blind spots Weaknesses that are not made more visible by stress Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 125 Stress Testing Interesting Papers ee e Astroman66 Finding and Exploiting Bugs 2600 e Bruce Schneier Crypto Gram May 15 2000 e James A Whittaker and Alan Jorgensen Why Software Fails
18. Rapid Test Planning Loo o Cem Kaner J D Ph D Department of Computer Sciences Florida Institute of Technology James Bach Satisfice Inc October 2001 STAR West Acknowledgments O e These notes outline the test planning chapters in prep for Testing Computer Software 3rd Ed by Cem Kaner James Bach Hung Quoc Nguyen Jack Falk Brian Lawrence amp Bob Johnson They incorporate and adapt materials by these authors The notes are also based on materials developed for Lessons Learned in Software Testing a book just completed by Cem Kaner James Bach and Bret Pettichord e Many of the ideas in these notes were reviewed and refined at the Third Los Altos Workshop on Software Testing LAWST February 7 8 1998 and at the Eleventh LAWST October 28 29 2000 The participants at LAWST 3 were Chris Agruss James Bach Karla Fisher David Gelperin Kenneth Groder Elisabeth Hendrickson Doug Hoffman III recorder Bob Johnson Cem Kaner host Brian Lawrence facilitator Brian Marick Thanga Meenakshi Noel Nyman Jeffery E Payne Bret Pettichord Johanna Rothman Jane Stepak Melora Svoboda Jeremy White and Rodney Wilson The participants at LAWST 11 were Chris Agruss James Bach Hans Buwalda Marge Farrell Sam Guckenheimer Elisabeth Hendrickson Doug Hoffman HI recorder Bob Johnson Karen Johnson Cem Kaner host Brian Lawrence facilitator Alan Myrvold Hung Quoc Nguyen Noel Nyman Neal Reizer Amit
19. Rights Reserved 88 Combinations Exercise Illustration O Find gt lt Find whet flowersesel O Direction Cancel C Up Down T batch case e Here is a simple Find dialog It takes three inputs Find what a text string Match case yesorno Direction up or down Simplify this by considering only three values for the text string lowercase and Mixed Cases and CAPITALS e Note To do a better job we d also choose input documents that would yield a find and a don t find for each case The input document would be another variable or really the intended result Find Don t would be the variable We ll think about that again after the exercise Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 89 Combinations Exercise OO 1 How many combinations of these three variables are possible 2 List ALL the combinations of these three variables 3 Now create combination tests that cover all possible pairs of values but don t try to cover all possible triplets List one such set 4 How many test cases are in this set Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 90 Combination Testing O eImagine a program with 3 variables V1 has 3 possible values V2 has 2 possible values and V3 has 2 possible values elf V1 and V2 and V3 are independent
20. Singh and Melora Svoboda Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 2 Abstract Loo o This workshop has grown out of our dissatisfaction with paper intensive approaches that attempt to provide a seemingly reproducible somewhat mechanical process for planning and managing testing and test documentation Over the past 17 years we have criticized IEEE standard 829 on software test documentation and related approaches as being often inappropriate Colleagues have asked what we would put in IEEE 829 s place To date our responses have been piecemeal This seminar s notes are a draft of our attempt to write a more comprehensive response They start from the premise that the best approach to test documentation depends on the project context For example creating detailed test documentation can be useful for some projects but can get in the way of the development of a high volume automated testing strategy What are the relevant differences between these projects Before adopting an implementation guideline like IEEE 829 we should analyze our requirements There is no point spending a fortune on creating a deliverable here the test documentation set that will not be used or that will interfere with the efficient running of the project Instead we should build a documentation set that will actually satisfy the real needs of the project The notes also reflect our view that testing is a
21. arallel with development work and finding problems worth fixing faster than the developers fix them Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 07 Heuristics for Test Plan Evaluation OO 14 The feedback loop between This is important in order to maximize testers and developers should be the efficiency and speed of quality as tight as possible Test cycles improvement It also helps keep testing should be designed to provide rapid off of the critical path feedback to developers about recent additions and changes they have made before a full regression test is commenced Whenever possible testers and developers should work physically near each other Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved l 08 Heuristics for Test Plan Evaluation OO 15 The test project should employ By examining product quality channels of information about quality information gathered through various other than formal testing in order to help means beyond the test team blind evaluate and adjust the test project spots in the formal test strategy can be Examples of these channels are uncovered inspections field testing or informal testing by people outside of the test team 16 All documentation related to the test Tunnel vision is the great occupational strategy including test cases and hazard of testing Review not only procedures should be underg
22. arl Popper in his famous essay Conjectures and Refutations lays out the proposition that a scientific theory gains credibility by being subjected to and passing harsh tests that are intended to refute the theory We can gain confidence in a program by testing it harshly if it passes the tests Subjecting it to easy tests doesn t tell us much about what will happen to the program in the field Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 3 6 Risk Based Testing Interesting Papers a e Stale Amland Risk Based Testing e James Bach Reframing Requirements Analysis e James Bach Risk and Requirements Based Testing e James Bach James Bach on Risk Based Testing e Stale Amland amp Hans Schaefer Risk based testing a response e Carl Popper Conjectures amp Refutations Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 3 7 User Testing Loo OO e Tag line Strive for realism Let s try this with real humans for a change e Fundamental question or goal Identify failures that will arise in the hands of a person 1 e breakdowns in the overall human machine software system e Paradigmatic case s Beta testing In house experiments using a stratified sample of target market Usability testing Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 3 8 User Testing Loo UU Stre
23. atrices for Routine Issues Loo e After testing a simple numeric input field a few times you may prefer a test matrix to present the same tests more concisely e Use a test matrix to show track a series of test cases that are fundamentally similar For example for most input fields you ll do a series of the same tests checking how the field handles boundaries unexpected characters function keys etc As another example for most files you ll run essentially the same tests on file handling e The matrix is a concise way of showing the repeating tests Put the objects that you re testing on the rows Show the tests on the columns Check off the tests hat you u actuall com leted in the cells 73 Rapid Test Planning pyright 200 aner and James Bach All LHE reserve Reusable Test Matrix Numeric Input Field s bIp uoN seya Jo s ibip jo saquinu AN 40 epis no anjea yJnejap y 4e3 9 pjaly Ajdwi3 s1ey3 10 s 6 p Jo saquinu gn Jeyd 10 s 6ip Jo saquinu g7 aAebon onjea Jo gn anjea jo g7 anjea Jo gn anjea JO g7 Bulujon Copyright 2001 Cem Kaner and James Bach All rights reserved 73 Rapid Test Planning Examples of integer input tests OO e Nothing e Valid value e AtLB of value e At UB of value e AtLB of value 1 e At UB of value 1 e Outside of LB of value e Outsi
24. bout Test Techniques e Analyze the situation e Model the test space A test technique e Select what to cover is arecipe e Determine test oracles for performing these tasks that e Configure the test system e Operate the test system e Observe the test system e Evaluate the test results will reveal something worth reporting Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 53 Thinking About Test Techniques Loo o e What is the difference between User testing Usability testing User interface testing Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 54 Thinking About Test Techniques OO e Testing combines techniques that focus on Testers who does the testing Coverage what gets tested Potential problems why you re testing what risk you re testing for Activities how you test Evaluation how to tell whether the test passed or failed e All testing involves all five dimensions e A technique focuses your attention on one or a few dimensions leaving the others open to your judgment You can combine a technique focused on one dimension with techniques focused on the other dimensions to achieve the result you want Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 5 5 Thinking About Test Techniques Loo o e Examples Testers e
25. cumentation serve or disserve What will they do with the documentation What types of documents are of high or low value e Asking questions e Context free questions e Context free questions specific to test planning e Evaluating a plan Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 l Discovering Requirements CCONOvrVO__ e Requirements Anything that drives or constrains design e Stakeholders Favored disfavored and neutral stakeholders e Stakeholders interests Favored disfavored and neutral interests e Actions Actions support or interfere with interests e Objects Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 12 Exercise Loo UU _ e1 List the Stakeholders Favored Disfavored Neutral stakeholders For each Stakeholder list her Interests Favored Disfavored Neutral interests For each Interest list Actions Actions support an interest Actions interfere with an interest Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 3 Exercise _e TNTNFT TTNNNuuv _ ee Objects The Stuff You Create Such as features data of the product eFor each object what is its relationship to a stakeholder astakeholder s interest or in the actions the stakeholder wants to take or will have taken on her Rapid Test Planning Copyright 2001 Cem Kan
26. d Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 80 Tabular Format for Data Relationships Loo o e THE TABLE S FIELDS Field Create a row for each field Consultant End Date and Start Date are examples of fields Entry Source What dialog boxes can you use to enter data into this field Can you import data into this field Can data be calculated into this field List every way to fill the field every screen etc Display List every dialog box error message window etc that can display the value of this field When you re enter a value into this field will the new entry show up in each screen that displays the field Not always sometimes the program makes local copies of variables and fails to update them Print List all the reports that print the value of this field and any other functions that print the value Related to List every variable that is related to this variable What if you enter a legal value into this variable then change the value of a constraining variable to something that is incompatible with this variable s value Relationship Identify the relationship to the related variable Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 8 1 Tabular Format for Data Relationships Loo O Many relationships among data Independence e Varying one has no effect on the value or permissible values of the other Causal determinat
27. damental question or goal Manage the risks that a a bug fix didn t fix the bug or b the fix or other change had a side effect e Paradigmatic case s Bug regression Show that a bug was not fixed Old fix regression Show that an old bug fix was broken General functional regression Show that a change caused a working area to break Automated GUI regression suites e Strengths Reassuring confidence building regulator friendly Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved l 1 5 Regression lesting OOO e Blind spots weaknesses Anything not covered in the regression series Repeating the same tests means not looking for the bugs that can be found by other tests Pesticide paradox Low yield from automated regression tests Maintenance of this standard list can be costly and distracting from the search for defects Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 l 6 Automating Regression Testing Loo o e This is the most commonly discussed automation approach create a test case run it and inspect the output if the program fails report a bug and try again later if the program passes the test save the resulting outputs in future tests run the program and compare the output to the saved results Report an exception whenever the current output and the saved output don t match Rap
28. de of UB of value e 0 e Negative e At LB number of digits or chars e At UB number of digits or chars e Empty field clear the default value Rapid Test Planning Outside of UB number of digits or chars Non digits Wrong data type e g decimal into integer Expressions Space Non printing char e g Ctrl char DOS filename reserved chars e g Upper ASCII 128 254 Upper case chars Lower case chars Modifiers e g Ctrl Alt Shift Ctrl etc Function key F2 F3 F4 etc Copyright 2001 Cem Kaner and James Bach All rights reserved 74 Error Handling when Writing a File OO e full local disk e almost full local disk e write protected local disk e damaged I O error local disk e unformatted local disk e remove local disk from drive after opening file e timeout waiting for local disk to come back online e keyboard and mouse I O during save to local disk e other interrupt during save to local drive e power out during save to local drive Rapid Test Planning full network disk almost full network disk write protected network disk damaged I O error network disk remove network disk after opening file timeout waiting for network disk keyboard mouse O during save to network disk other interrupt during save to network drive local power out during save to network network power during save to network Copyright 2001 Cem Kaner and James Bach All rights reserved
29. e borders like Ward S c For the Macintosh C Geo Full justification like WordPerfect 6 x For Windows _ Don t add automatic tab stop For hanging indent WC Don t add extra space For raised lowered characters D Don t add leading fextra space between rows of text IC Don t add space For underlines C Don t balance columns For Continuous section starts Il Don t balance SECS characters and OBCS characters C Don t blank the area behind metafile pictures Don t center exact line height lines Don t convert backslash characters into yen signs BeFaul Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 78 Tabular Format for Data Relationships Field Entry Display Print Related Relationship source variable Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 19 Tabular Format for Data Relationships Loo OO Once you identify two variables that are related test them together using boundary values of each or pairs of values that will trigger some other boundary This is not the most powerful process for looking at relationships An approach like Cause Effect Graphing is more powerful if you have or can build a complete specification I started using this chart as an exploratory tool for simplifying my look at relationships in overwhelmingly complex programs There doesn t have to be a lot of complexity to be overwhelming Rapi
30. e last choice and try again Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 95 We flipped the last arbitrary choice column 5 section 2 to GH from HG and erased section 3 We then fill in section 3 by checking for missing pairs GH GH gives us two XG XG pairs so we flip to HP for the third section and have a column 2 X with a column 5 H and a column 2 Y with a column 5 G as needed to obtain all pairs Rapid Test Planning Combination Testing Copyright 2001 Cem Kaner and James Bach All rights reserved 96 Combination Testing O eBut when we add the next column we see that we just can t achieve all pairs with 6 values The first one works up to column 4 but then fails to get pair EJ or Fl The next fails on GJ HI Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 97 Combination Testing When all else fails add rows We need one for GJ and one for HI so add two rows as the last column has values The other values in the two rows are Bly 1 e H J arbitrary leave them blank and fill them in ff Hh as needed when you add new columns At the very end fill the remaining blank ones with arbitrary values Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 98 Combination Testing O elf a variable is continuous but maps to a number line partition and use boundaries as the distinc
31. e user such as with other programs hard disk network printer etc Application functions that define or distinguish the product or fulfill core requirements Error Handling functions that detect and recover from errors including error messages Testability functions provided to help test the product such as diagnostics log files asserts test menus etc e Temporal relationships How the program functions over time Sequential operation state to state transitions Data changes in variables over time System interactions such as synchronization or ordering of events in distributed systems Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 3 3 Product Elements Doo OOO e Data Everything that the product processes Input data that is processed by the product Output data that results from processing by the product Preset data supplied as part of the product or otherwise built into it such as prefab databases default values etc Persistent data stored internally and expected to persist over multiple operations This includes modes or states of the product such as options settings view modes contents of documents etc Temporal data based on time such as date stamps or number of events recorded in a unit of time Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 34 Product Elements oo e Platform Everything o
32. eatures and functions There is a stakeholder who will make a fuss if the program doesn t pass this scenario Strengths Complex realistic events Can handle help with situations that are too complex to model Exposes failures that occur develop over time e Blind spots Single function failures can make this test inefficient Must think carefully to achieve good coverage Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 3 1 Scenario Testing Interesting Papers ee e Hans Buwalda on Soap Operas in the conference proceedings of STAR East 2000 e Kaner A pattern for scenario testing at www testing com e Lots of literature on use cases Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 32 Risk Based Testing _ s s wt OO e Tag line Find big bugs first e Fundamental question or goal Define and refine tests in terms of the kind of problem or risk that you are trying to manage OR prioritize the testing effort in terms of the relative risk of different areas or issues we could test for e Paradigmatic case s Failure Mode and Effects Analysis FMEA Equivalence class analysis reformulated Test in order of frequency of use Musa Stress tests error handling tests security tests tests looking for predicted or feared errors sample from predicted bugs list Rapid Tes
33. ed boundaries Equivalent hardware e such as compatible modems Equivalent event times e such as before timeout and after Equivalent output events e perhaps any report will do to answer a simple the question Will the program print reports Equivalent operating environments e such as French amp English versions of Windows 3 1 Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 69 Variables Well Suited to Equivalence Class Analysis OOO Many types of variables including input output internal hardware and system software configurations and equipment states can be subject to equivalence class analysis Here are some examples ranges of numbers size of a number that you enter character codes number of digits or size of a how many times something is character string done size of a concatenated string e g shareware limit on numberof size of a path specification uses of a product size of a file name e g how many times you can do size in characters of a it before you run out of memory document how many names in a mailing size of a file note special values list records in a database such as exactly 64K exactly 512 variables in a spreadsheet bytes etc bookmarks abbreviations size of the document on the page size of the sum of variables or of compared to page margins some other computed value across different page margins th
34. eliability Usability Quality is Performance value to Installability some perso Compatibility Jerry Supportability Weinberg Testability Maintainability Portability Localizability Efficiency Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 3 fi Quality Criteria O Accessibility Maintainability Quality is value Capability Performance to some person Compatibility Portability Jerry Concurrency Recoverability Weinberg Conformance t gt o Reliability Standards Scalability Efficiency Security Installability Supportability and i minal uy Usability Localizability Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 3 8 Risk T __ _ Hazard A dangerous condition something that could trigger an accident Risk Possibility of suffering loss or harm Accident A hazard is encountered resulting in loss or harm e Useful material available free at http seir sei cmu edu e http www coyotevalley com Brian Lawrence e Good paper by Stale Amland Risk Based Testing and Metrics 16th International Conference on Testing Computer Software Rapid LEP ning Copyright 2001 Cem Kaner and James Bach All rights reserved 3 9 Risk e Project risk management involves Identification of the different risks to the project issues that might cause the project to fail or to fall behind schedule or to cost too muc
35. er and James Bach All rights reserved 14 Testers Questions Does Your Car Work _ s ws CC OO HOW CAN YOU TELL THAT SOMETHING WORKS How do you know your car works Are there situations in which your car would stop working Who else uses your car Do they use it differently than you so that it might work for you but fail for them What facts would cause you to believe that your car doesn t work In what ways could your car not work yet seem to you that it does In what ways could your car work yet seem to you that it doesn t Do you know enough about cars to answer these questions Have you observed your car enough today to answer them Under what circumstances would these questions matter Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved l 5 Questioning CO u u u e Requirements analysis requires information gathering Read books on consulting Gause amp Weinberg Exploring Requirements is an essential source on context free questioning e There are many types of questions Open vs closed Hypothetical vs behavioral Opinion vs factual Historical vs predictive Context dependent and context free Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 16 The classic context free questions TT e The traditional newspaper reporters questions are Who What When Where How Wh
36. esting project Items Under Test Logistics Budget Deliverables Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 2I Project Factors OOOO e Stakeholders Anyone who is a client of the main project Anyone who is a client of the testing project e Includes customers purchasers end users tech support programmers project mgr doc group etc e Processes The tasks and events that comprise the main project e How the overall project is run The tasks and events that comprise the test project e How the testing project is run e Staff Everyone who helps develop the product e Sources of information and assistance Everyone who will perform or support testing e Special talents or experiences of team members e Size of the group e Extent to which they are focused or are multi tasking e Organization collaboration amp coordination of the staff e Is there an independent test lab Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 28 Project Factors OT e Schedules The sequence duration and synchronization of events When will testing start and how long is it expected to take When will specific product elements be available to test When will devices or tools be available to support testing e Equipment Hardware required for testing What devices do we need to test the product with Do we have them e Tools amp Test Mate
37. ests e The individual tests are not all that powerful nor all that compelling e The power of the approach lies in the large number of tests e These broaden the sample and they may test the program over a long period of time giving us insight into longer term issues Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 144 Random Statistical Testing O e Paradigmatic case s Some of us are still wrapping our heads around the richness of work in this field This is a tentative classification e NON STOCHASTIC RANDOM TESTS e STATISTICAL RELIABILITY ESTIMATION e STOCHASTIC TESTS NO MODEL e STOCHASTIC TESTS USING ON A MODEL OF THE SOFTWARE UNDER TEST e STOCHASTIC TESTS USING OTHER ATTRIBUTES OF SOFTWARE UNDER TEST Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 145 Random Statistical Testing Non Stochastic _ _ s CC OO e Fundamental question or goal The computer runs a large set of essentially independent tests The focus is on the results of each test Tests are often designed to minimize sequential interaction among tests e Paradigmatic case s Function equivalence testing Compare two functions e g math functions using the second as an oracle for the first Attempt to demonstrate that they are not equivalent 1 e that the achieve different results from the same set of inputs Other test using fully determi
38. for errors O e Poor design or unmaintainable implementation Some internal design decisions make the code so hard to maintain that fixes consistently cause new problems e Tired programmers long overtime over several weeks or months yields inefficiencies and errors e Other staff issues alcoholic mother died two programmers who won t talk to each other neither will their code e Just slipping it in pet feature not on plan may interact badly with other code N I H external components can cause problems e N I B not in budget Unbudgeted tasks may be done shoddily Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 43 Risks Where to look for errors O e Ambiguity ambiguous descriptions in specs or other docs can lead to incorrect or conflicting implementations e Conflicting requirements ambiguity often hides conflict result is loss of value for some person e Unknown requirements requirements surface throughout development Failure to meet a legitimate requirement is a failure of quality for that stakeholder e Evolving requirements people realize what they want as the product develops Adhering to a start of the project requirements list may meet contract but fail product check out http www agilealliance org Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 44 Risks Where to look for errors O Complexity complex code
39. g what it is supposed to or not Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 149 Stochastic Tests No Model Dumb Monkeys OOO e Paradigmatic case s Executive monkeys Know nothing about the system Push buttons randomly until the system crashes Clever monkeys More careful rules of conduct more knowledge about the system or the environment See Freddy O S compatibility testing No model of the software under test but diagnostics might be available based on the environment the NT example Early qualification testing Life testing Load testing e Notes Can be done at the API or command line just as well as via UI 150 Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Stochastic assert or diagnostics based random tests OO O e Fundamental question or goal High volume random testing using random sequence of fresh or pre defined tests that may or may not self check for pass fail The primary method for detecting pass fail uses assertions diagnostics built into the program or other e g system diagnostics e Paradigmatic case s Telephone example asserts Embedded software example diagnostics Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 5 l Random Testing Stochastic Regression Based _ _ ai O e Fundamental question or
40. gies that will reveal the product and its defects e Paradigmatic case s Skilled exploratory testing of the full product Rapid testing Emergency testing including thrown over the wall test it today testing Third party components Troubleshooting follow up testing of defects Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 14 l Exploratory Testing Loo o e Strengths Customer focused risk focused Takes advantage of each tester s strengths Responsive to changing circumstances Well managed it avoids duplicative analysis and testing High bug find rates e Blind spots The less we know the more we risk missing Limited by each tester s weaknesses can mitigate this with careful management This is skilled work juniors aren t very good at it Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 142 Exploratory Testing Interesting Papers Pn e Chris Agruss amp Bob Johnson Ad Hoc Software Testing Exploring the Controversy of Unstructured Testing e Whittaker amp Jorgenson How to Break Software Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 143 Random Statistical Testing Loo o e Tag line High volume testing with new cases all the time e Fundamental question or goal Have the computer create execute and evaluate huge numbers of t
41. goal High volume random testing using random sequence of pre defined tests that can self check for pass fail e Paradigmatic case s Life testing Search for specific types of long sequence defects Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 52 Random Testing Stochastic Regression Based OOO e Notes Create a series of regression tests Design them so that they don t reinitialize the system or force it to a standard starting state that would erase history The tests are designed so that the automation can identify failures Run the tests in random order over a long sequence This is a low mental overhead alternative to model based testing You get pass fail info for every test but without having to achieve the same depth of understanding of the software Of course you probably have worse coverage less awareness of your actual coverage and less opportunity to stumble over bugs Unless this is very carefully managed there is a serious risk of non reproduceability of failures 1 Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 53 Random Testing Sandboxing the Regression Tests Eee e n a random sequence of standalone tests we might want to qualify each test T1 T2 etc as able to run on its own Then when we test a sequence of these tests we know that errors are due to interactions among them rather than me
42. h or to dissatisfy customers or other stakeholders Analysis of the potential costs associated with each risk Development of plans and actions to reduce the likelihood of the risk or the magnitude of the harm Continuous assessment or monitoring of the risks or the actions taken to manage them Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 40 Risk Based Testing O e Two key dimensions Find errors risk based approach to technical tasks of testing Manage the process of finding errors risk based test management Our focus today is on methods for finding errors efficiently Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 4 Risks Where to look for errors Loo O e Qualities Failure to conform to a quality criterion risk of unreliability risk of unmaintainability etc e New things newer features may fail e New technology new concepts lead to new mistakes e New markets A different customer base will see and use the product differently e Learning Curve mistakes due to ignorance e Changed things changes may break old code e Late changes rushed decisions rushed or demoralized Staff lead to mistakes e Rushed work some tasks or projects are chronically underfunded and all aspects of work quality suffer Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 42 Risks Where to look
43. hierarchy of lists and sublists Such as the testing objectives list later in the course notes Tables A table organizes information in two dimensions showing relationships between variables Such as boundary tables decision tables combination test tables Matrices A matrix is a special type of table used for data collection Such as the numeric input field matrix configuration matrices Refer to Testing Computer Software pages 217 241 For more examples see page Testing Computer Software page 218 Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 64 Traceability Matrix pe are ars er ioe rave ee rea eR ress xT eeo reste TK a ress fT PE mss TT Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 65 Traceability Matrix Loo o e The columns involve different test items A test item might be a function a variable an assertion in a specification or requirements document a device that must be tested any item that must be shown to have been tested e The rows are test cases e The cells show which test case tests which items e Ifa feature changes you can quickly see which tests must be reanalyzed probably rewritten e In general you can trace back from a given item of interest to the tests that cover it e This doesn t specify the tests it merely maps their coverage Rapid Test Planning Copyright 2001 Cem
44. hips table Many of these options have a lot of repercussions e You might analyze all of the details of all of the relationships later but for now it is challenging just to find out what all the relationships ARE e The table guides exploration and will surface a lot of bugs e PROBLEM e Works great for this release Next release what is your support for more exploration Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 84 Configuration Planning Table vert vez vws vara vars This table defines 6 standard configurations for testing In later tests the lab will set up a Config 1 system a Config 2 system etc and will do its compatibility testing on these systems The variables might be software or hardware choices For example Var 1 could be the operating system V1 1 is Win 2000 V1 2 is Win ME etc Var 2 could be how much RAM on the computer under test V2 1 is 128 meg V2 2 is 32 meg etc Var 3 could be the type of printer Var 4 the machine s language setting French German etc Config planning tables are often filled in using the All Pairs algorithm g5 Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Configuration Test Matrix O __ Config 1_ Config 2 Config 3_ Config 4 Config 5 Config 6 Test Pass Pass__ Pass__ Pass__ Pass Test2 Fail __ Pass_ Pass Pass Test3 Pass_ Pass _ Pass Pass _ Pass
45. hird Party how does that answer the prime strategic questions How will you cover the product and assess quality How is that practical and justified with respect to the specifics of this project and product e If you don t know then your real strategy is that you re trusting things to work out Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 6 1 Diverse Half Measures O e There is no single technique that finds all bugs e We cant do any technique perfectly e We can t do all conceivable techniques Use diverse half measures lots of different points of view approaches techniques even if no one strategy is performed completely Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 62 Test Plan Components s OO e The following slides give examples of several charts tables etc e You probably won t have enough time to create all the documentation that would be useful Treat these materials as optional e Use the components that you find most useful to Clarify your own thinking Communicate your thinking to others Track your work or the work of someone else Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 63 Basic Test Documentation Components Loo OO Lists Such as lists of fields error messages DLLs Outlines An outline organizes information into a
46. id Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 1 7 Potential Regression Advantages O e Dominant paradigm for automated testing e Straightforward e Same approach for all tests e Relatively fast implementation e Variations may be easy e Repeatable tests Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 l 8 GUI Regression Interesting Papers ss OO e Chris Agruss Automating Software Installation Testing e James Bach Test Automation Snake Oil e Hans Buwalda Testing Using Action Words e Hans Buwalda Automated testing with Action Words Abandoning Record amp Playback e Elisabeth Hendrickson The Difference between Test Automation Failure and Success e Cem Kaner Avoiding Shelfware A Manager s View of Automated GUI Testing e John Kent Advanced Automated Testing Architectures e Bret Pettichord Success with Test Automation e Bret Pettichord Seven Steps to Test Automation Success e Keith Zambelich Totally Data Driven Automated Testing Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 9 Domain Testing OT e AKA partitioning equivalence analysis boundary analysis e Fundamental question or goal This confronts the problem that there are too many test cases for anyone to run This is a stratified sampling strategy that provides a rationale for selecting a few test cases from a huge population e
47. igm What makes for an excellent test What is your approach best for What are some weaknesses in your approach Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved l 12 Function esting OO e Tag line Black box unit testing e Fundamental question or goal Test each function thoroughly one at a time e Paradigmatic case s Spreadsheet test each item in isolation Database test each report in isolation e Strengths Thorough analysis of each item tested e Blind spots Misses interactions misses exploration of the benefits offered by the program 113 Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Some Function Testing Tasks nn Identify the program s features commands From specifications or the draft user manual From walking through the user interface From trying commands at the command line From searching the program or resource files for command names Identify variables used by the functions and test their boundaries Identify environmental variables that may constrain the function under test Use each function in a mainstream way positive testing Push it in as many ways as possible as hard as possible Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved l 14 Regression lesting OOO e Tag line Repeat testing after changes e Fun
48. ing Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 57 General Test Techniques Loo o e We provide an appendix that describes the 10 general test techniques that we listed on the previous slide e We aren t going to work through that appendix or not in much detail in this workshop but these notes may be helpful for self study to fill in some of the details that we re skipping here Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 5 8 Test Strategy Loo o e How we plan to cover the product so as to develop an adequate assessment of quality e A good test strategy is Diversified Specific Practical Defensible Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 59 Test Strategy O Makes use of test techniques May be expressed by test procedures and cases Not to be confused with test ogistics which involve the details of bringing resources to bear on the test strategy at the right time and place You don t have to know the entire strategy in advance The strategy can change as you learn more about the product and its problems Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 60 Test Cases Procedures oo e Test cases and procedures should manifest the test strategy e If your strategy is to execute the test suite got from Joe T
49. ink binary and think digits page sizes Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 70 Variables Well Suited to Equivalence Class Analysis Doo OOO size of a document on a page in terms length of time after a timeout from of the memory requirements for the JUST before to way after what page This might just be in terms of events are important resolution x page size but it may be speed of data entry time between more complex if we have compression keystrokes menus etc equivalent output events such as e speed of input handling of concurrent printing documents events amount of available memory gt 128 e number of devices connected active i meg gt 640K etc system resources consumed available visual resolution size of screen number also handles stack space etc of colors date and time OPA UNI Sy REVER transitions between algorithms variations within a group of compatible optimizations different ways to printers sound cards modems etc compute a function equivalent event times when something most recent event first event happens timing how long between event A and event B and in which order races input or output intensity voltage e speed extent of voltage transition e g from very soft to very loud sound Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 7 l Using Test M
50. ion e By changing the value of one we determine the value of the other e For example in MS Word the extent of shading of an area depends on the object selected The shading differs depending on Table vs Paragraph Constrained to a range e For example the width of a line has to be less than the width of the page e In a date field the permissible dates are determined by the month and the year if February Selection of rules e Example hyphenation rules depend on the language you choose Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 82 Tabular Format for Data Relationships Loo OO Many relationships among data Logical selection from a list e processes the value you entered and then figures out what value to use for the next variable Example timeouts in phone dialing 0 on complete call 555 1212 but 95551212 10 on ambiguous completion 955 5121 30 seconds incomplete 555 121 Logical selection of a list e For example in printer setup choose OfficeJet get Graphics Quality Paper Type and Color Options LaserJet 4 get Economode Resolution and Half toning Look at Marick Craft of Software Testing for discussion of catalogs of tests for data relationships Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 83 Data Relationship Table OO u e Looking at the Word options you see the real value of the data relations
51. ir test cases And how well do they ensure that test changes will follow code changes e Will the test docs help us identify and revise restructure in face of a permanent shift in the risk profile of the program e Are Should docs be automatically created as a byproduct of the test automation code Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 24 Ultimately write a mission statement O OT EE e Try to describe your core documentation requirements in one sentence that doesn t have more than three components e Examples The test documentation set will primarily support our efforts to find bugs in this version to delegate work and to track status The test documentation set will support ongoing product and test maintenance over at least 10 years will provide training material for new group members and will create archives suitable for regulatory or litigation use Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 25 A Model of Software Testing Loo UU SS Project Quality Environment Criteria Test Techniques Elements Y Test Results Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 26 Project Environment Factors ss OO Stakeholders Processes Staff Schedules Equipment These aspects of the Tools amp Test Materials environment constrain and Information enable the t
52. ism to encapsulate and describe state complexity By expressing states as the cross product of operational modes and eliminating impossible states the number of distinct states can be reduced alleviating the state explosion problem Operational modes are not a new feature of software but rather a different way to view the decomposition of states All software has operational modes but the implementation of these modes has historically been left to chance When used for testing operational modes have been extracted by reverse engineering Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 5 8 Random Testing Thoughts Toward an Architecture Doo OOO e We have a population of tests which may have been sandboxed and which may carry self check info A test series involves a sample of these tests e We have a population of diagnostics probably too many to run every time we run a test In a given test series we will run a subset of these e We have a population of possible configurations some of which can be set by the software In a given test series we initialize by setting the system to a known configuration We may reset the system to new configurations during the series e g every 5th test Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved l 59 Random Testing Thoughts Toward an Architecture Doo OOO e We have an execution tool that takes as input a list
53. may be buggy Bugginess features with many known bugs may also have many unknown bugs Dependencies failures may trigger other failures Untestability risk of slow inefficient testing Little unit testing programmers find and fix most of their own bugs Shortcutting here is a risk Little system testing so far untested software may fail Previous reliance on narrow testing strategies e g regression function tests can yield a backlog of errors surviving across versions Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 45 Risks Where to look for errors Loo o e Weak testing tools if tools don t exist to help identify isolate a class of error e g wild pointers the error is more likely to survive to testing and beyond Unfixability risk of not being able to fix a bug Language typical errors such as wild pointers in C See Bruce Webster Pitfalls of Object Oriented Development Michael Daconta et al Java Pitfalls Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 46 Risks Where to look for errors _ s s o OO Criticality severity of failure of very important features Popularity likelihood or consequence if much used features fail Market severity of failure of key differentiating features Bad publicity a bug may appear in PC Week Liability being sued Rapid Test Planning Copyright 2001 Cem Kaner a
54. may be used to model many characteristics of a software program Mathematically a DFA is the quintuple M Q _ q0 F where M is the machine Q is a finite set of states is a finite set of inputs commonly called the alphabet _ is the transition function that maps Q x _ to Q q0 is one particular element of Q identified as the initial or stating state and F C Q is the set of final or terminating states Sudkamp 1988 The DFA can be viewed as a directed graph where the nodes are the states and the labeled edges are the transitions corresponding to inputs Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 57 Random Testing Model based Stochastic Tests s OO Alan Jorgensen Software Design Based on Operational Modes Ph D thesis Florida Institute of Technology When taking this state model view of software a different definition of software failure suggests itself The machine makes a transition to an unspecified state From this definition of software failure a software defect may be defined as Code that for some input causes an unspecified state transition or fails to reach a required state Recent developments in software system testing exercise state transitions and detect invalid states This work Whittaker 1997b developed the concept of an operational mode that functionally decomposes abstracts states Operational modes provide a mechan
55. n exercise in critical thinking and careful questioning A test case is a question that you ask of the program Are you broken in this way The point of a test case is to reduce uncertainty associated with the product A test is good if it will reduce uncertainty whether it finds a bug or not A test plan is a structure for asking questions of the project and the product These notes suggest strategies for asking better questions and they provide useful clusters of questions The notes also provide samples of some common test planning documents such as tables and matrices These will probably be among the building blocks of any testing program that you set up Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 3 What Makes Our Approach Rapid OO e Focus on your actual requirements not requirements dictated by standards e Focus on gathering and using information rather than writing down all possible details e Iterative development of materials adding detail and depth to test cases as you go Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Overview _ e Problems with the allegedly standard approach e Defining your documentation requirements e A model for testing and test documentation e Test documentation elements Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Problems with
56. n which the product depends External Hardware components and configurations that are not part of the shipping product but are required or optional in order for the product to work Includes CPU s memory keyboards peripheral boards etc External Software software components and configurations that are not a part of the shipping product but are required or optional in order for the product to work Includes operating systems concurrently executing applications drivers fonts etc e Operations How the product will be used Usage Profile the pattern of usage over time including patterns of data that the product will typically process in the field This varies by user and type of user Environment the physical environment in which the product will be operated including such elements as light noise and distractions Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 3 5 Product Elements Coverage O u u Product coverage is the proportion of the product that has been tested There are as many kinds of coverage as there are ways to model the product Structural Functional See Software Negligence Temporal amp Testing Coverage at Data www kaner com for 101 Platform examples of coverage Operations measures Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 3 6 Quality Criteria Capability R
57. nd James Bach All rights reserved 47 Bug Patterns as a Source of Risk ee Testing Computer Software lays out a set of 480 common defects You can use these or develop your own list Find a defect in the list Ask whether the software under test could have this defect If it is theoretically possible that the program could have the defect ask how you could find the bug if it was there Ask how plausible it is that this bug could be in the program and how serious the failure would be if it was there If appropriate design a test or series of tests for bugs of this type Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 48 Build Your Own Model of Bug Patterns Loo OO e oo many people start and end with the TCS bug list It is outdated It was outdated the day it was published And it doesnt cover the issues in your system Building a bug list is an ongoing process that constantly pays for itself Here s an example from Hung Nguyen This problem came up in a client server system The system sends the client a list of names to allow verification that a name the client enters is not new Client 1 and 2 both want to enter a name and client 1 and 2 both use the same new name Both instances of the name are new relative to their local compare list and therefore they are accepted and we now have two instances of the same name As we see these we develop a library
58. ngths Design issues are more credibly exposed Can demonstrate that some aspects of product are incomprehensible or lead to high error rates in use In house tests can be monitored with flight recorders capture replay video debuggers other tools In house tests can focus on areas tasks that you think are or should be controversial Blind spots Coverage is not assured serious misses from beta test other user tests Test cases can be poorly designed trivial unlikely to detect subtle errors Beta testing is not free beta testers are not skilled the technical results are mixed Distinguish marketing betas from technical betas Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Exploratory Testing Loo o Simultaneously Learn about the product Learn about the market Learn about the ways the product could fail Learn about the weaknesses of the product Learn about how to test the product Test the product Report the problems Advocate for repairs Develop new tests based on what you have learned so far Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 140 Exploratory Testing O e Tag line Simultaneous learning planning and testing e Fundamental question or goal Software comes to tester under documented and or late Tester must simultaneously learn about the product and about the test cases strate
59. nistic oracles see discussion of oracles below Other tests using heuristic oracles see discussion of oracles below Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 146 Statistical Reliability Estimation OO e Fundamental question or goal Use random testing possibly stochastic possibly oracle based to estimate the stability or reliability of the software Testing is being used primarily to qualify the software rather than to find defects e Paradigmatic case s Clean room based approaches Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 147 The Need for Stochastic Testing An Example Caller gt wau On Hold HN Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 148 Stochastic Tests No Model Dumb Monkeys Doo OOO e Fundamental question or goal High volume testing involving a long sequence of tests A typical objective is to evaluate program performance over time The distinguishing characteristic of this approach is that the testing software does not have a detailed model of the software under test The testing software might be able to detect failures based on crash performance lags diagnostics or improper interaction with other better understood parts of the system but it cannot detect a failure simply based on the question Is the program doin
60. o review _ helps to reveal blind spots in test by someone other than the person who design but it can also help promote wrote them The review process used dialog and peer education about test should be commensurate with the practices criticality of the document Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 09 Evaluating Your Plan Context Free Questions Loo o Based on The CIA s Phoenix Checklists Thinkertoys p 140 and Bach s Evaluation Strategies Rapid Testing Course notes Can you solve the whole problem Part of the problem What would you like the resolution to be Can you picture it How much of the unknown can you determine What reference data are you using if any What product output will you evaluate How will you do the evaluation Can you derive something useful from the information you have Have you used all the information Have you taken into account all essential notions in the problem Can you separate the steps in the problem solving process Can you determine the correctness of each step What creative thinking techniques can you use to generate ideas How many different techniques Can you see the result How many different kinds of results can you see How many different ways have you tried to solve the problem Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved l 1 0 E
61. of issues The discovery method is exploratory requires sophistication with the underlying technology Capture winning themes for testing in charts or in scripts on their way to being automated Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 49 Building Bug Patterns OOO There are plenty of sources to check for common failures in the common platforms www bugnet com Wwww cnet com links from www winfiles com various mailing lists Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 50 Risk Driven Testing Cycle Perform Appropriate Analyze Potential Risks Improve Risk Analysis Report amp Resolve Analyze Actual Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 5 Test Case Design Ooo u u elf the purpose of testing is to gain information about the product then a test case s function is to elicit information quickly and efficiently ein information theory we define information in terms of reduction of uncertainty If there is little uncertainty there is little information to be gained eA test case that promises no information is poorly designed A good test case will provide information of value whether the program passes the test or fails it Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 52 Thinking A
62. problem Weinberg s How much time is available to solve the problem Exploring Context free product questions Requirements What problems could this product create p 59 64 What kind of precision is required desired for this product Metaquestions when interviewing someone for info Am I asking too many questions Do my questions seem relevant Are you the right person to answer these questions Is there anyone else who can provide additional information Is there anything else I should be asking Is there anything you want to ask me May I return to you with more questions later Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 20 What is your group s mission Oooo e Find important problems e Advise about QA e Assess quality e Advise about testing e Certify to standard e Advise about quality e Fulfill process mandates e Maximize efficiency e Satisfy stakeholders e Minimize time e Assure accountability e Minimize cost The quality of testing depends on which of these possible missions matter and how they relate Many debates about the goodness of testing are really debates over missions and givens Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 2 l Test Docs Requirements Questions ee e Is test documentation a product or tool e Is software quality driven by legal issues or by market forces How
63. quickly is the design changing How quickly does the specification change to reflect design change Is testing approach oriented toward proving conformance to specs or nonconformance with customer expectations Does your testing style rely more on already defined tests or on exploration Should test docs focus on what to test objectives or on how to test for it procedures Should the docs ever control the testing project Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 22 Test Docs Requirements Questions ee If the docs control parts of the testing project should that control come early or late in the project Who are the primary readers of these test documents and how important are they How much traceability do you need What docs are you tracing back to and who controls them To what extent should test docs support tracking and reporting of project status and testing progress How well should docs support delegation of work to new testers What are your assumptions about the skills and knowledge of new testers Is test doc set a process model a product model or a defect finder Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 23 Test Docs Requirements Questions i e A test suite should provide prevention detection and prediction Which is the most important for this project e How maintainable are the test docs and the
64. ration with all other functions have information about product of the project especially problems or potential problems that developers technical support and can be of use to the test team Their technical writing Whenever perspective may help the testers possible testers should also make a better analysis of risk collaborate with actual customers Testers may also have information and users in order to better that is of use to them understand their requirements 9 The test project should consult The likelihood that a test strategy will with development to help them serve its purpose is profoundly build a more testable product affected by the testability of the product Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved l 04 Heuristics for Test Plan Evaluation TT 10 A test plan should highlight the Virtually every software project worth non routine project specific doing involves special technical aspects of the test strategy and test challenges that a good test effort project must take into account A completely generic test plan usually indicates a weak test planning process It could also indicate that the test plan is nothing but unchanged boilerplate Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 05 Heuristics for Test Plan Evaluation TT 11 The test project should use Many test projects suffer under the humans for what human
65. re time analyzing the code and too little time analyzing the customer and her uses of the software Potential to create an inappropriate prestige hierarchy devaluating the skills of subject matter experts who understand the product and its defects much better than the automators Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 62 Random Testing Interesting Papers ee e Larry Apfelbaum Model Based Testing Proceedings of software Quality Week 1997 not included in the course notes e Michael Deck and James Whittaker Lessons learned from fifteen years of cleanroom testing STAR 97 Proceedings e Doug Hoffman Mutating Automated Tests e Alan Jorgensen An API Testing Method e Noel Nyman GUI Application Testing with Dumb Monkeys e Harry Robinson Finite State Model Based Testing on a Shoestring e Harry Robinson Graph Theory Techniques in Model Based Testing Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 63
66. rely to cumulative effects of repetition of a single test e Therefore for each Ti we run the test on its own many times in one long series randomly switching as many other environmental or systematic variables during this random sequence as our tools allow e We call this the sandbox series Ti is forced to play in its own sandbox until it proves that it can behave properly on its own This is an 80 20 rule operation We do want to avoid creating a big random test series that crashes only because one test doesn t like being run or that fails after a few runs under low memory We want to weed out these simple causes of failure But we don t want to spend a fortune trying to control this risk Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved l 54 Random Testing Sandboxing the Regression Tests Eee Suppose that you create a random sequence of standalone tests that were not sandbox tested and these tests generate a hard to reproduce failure You can run a sandbox on each of the tests in the series to determine whether the failure is merely due to repeated use of one of them Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 5 5 Random Testing Model based Stochastic Tests Loo OO e Fundamental Question or Goal Build a state model of the software The analysis will reveal several defects in itself Generate random events inputs to
67. rials Software required or desired for testing Automation Are such tools available Do we want to use them Do we have them Do we understand them Probes or diagnostics to help observe the product under test Matrices checklists other testing documentation e Information As needed for testing about the project or product Specifications requirements documents other reference materials to help us determine pass fail or to credibly challenge odd behaviour e What is the availability of these documents e What is the volatility of these documents Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 29 Project Factors OT e Items Under Test Anything that will be tested For each product element e Is it available or when will it be e Is it volatile and what is the change process e Is it testable e Logistics Facilities and support needed for organizing and conducting the testing Do we have the supplies physical space power light security systems if needed procedures for getting more e Budget Money and other resources for testing Can we afford the staff space training tools supplies etc e Deliverables The observable products of the test project Such as bug reports summary reports test documentation master disk e What are you supposed to create and can you do it Will we archive the items under test and other products of testing
68. s do well false belief that human testers are and use automation for what effective when they use exactingly automation does well Manual specified test scripts or that test testing should allow for automation duplicates the value of improvisation and on the spot human cognition in the test execution critical thinking while automated process Manual and automated testing should be used for tests that testing are not two forms of the same require high repeatability high thing They are two entirely different speed and no judgment classes of test technique Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 06 Heuristics for Test Plan Evaluation A 12 The test schedule should be A monolithic test schedule in a test represented and justified in such a plan often indicates the false belief way as to highlight any that testing is an independent activity dependencies on the progress of The test schedule can stand alone development the testability of the only to the extent that the product the product time required to report highly testable development is problems and the project team s complete and the test process is not assessment of risk interrupted by the frequent need to report problems 13 The test process should be kept This is important in order to deflect off of the critical path to the extent pressure to truncate the testing possible This can be done by testing process in p
69. t Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 3 3 Risk Based Testing Loo o e Strengths Optimal prioritization assuming we correctly identify and prioritize the risks High power tests e Blind spots Risks that were not identified or that are surprisingly more likely Some risk driven testers seem to operate too subjectively How will I know what level of coverage that I ve reached How do I know that I haven t missed something critical Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 34 Evaluating Risk O e Several approaches that call themselves risk based testing ask which tests we should run and which we should skip if we run out of time e We think this is only half of the risk story The other half is focuses on test design It seems to us that a key purpose of testing is to find defects So a key strategy for testing should be defect based Every test should be questioned e How will this test find a defect e What kind of defect do you have in mind e What power does this test have against that kind of defect Is there a more powerful test A more powerful suite of tests Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 3 5 Evaluating Risk Loo O e Many of us who think about testing in terms of risk analogize testing of software to the testing of theories K
70. t values under test If all variables are continuous we end up with all pairs of all boundary tests of all variables We don t achieve all triples all quadruples etc elf some combinations are of independent interest add them to the list of n tuples to test With the six columns of the example we reduced 96 tests to 8 Give a few back make it 12 or 15 tests and you still get enormous reduction Examples of independent interest are known from tech support high risk cases cases that jointly stress memory configuration combinations Var 1 is operating systems Var 2 is printers etc that are prevalent in the market etc Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 99 Charts References O You can find plenty of example charts in Bill Perry s books such as Effective Methods for Software Testing 2nd Ed Wiley Several of these will probably be useful though like the charts in these notes you ll have to adapt them to your circumstances Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved l 00 Heuristics from James Bach s Test Plan Evaluation Model L Eee 1 Testing should be optimized to find important problems fast rather than attempting to find all problems with equal urgency 2 Test strategy should focus most effort on areas of potential technical risk while still putting some effort into low risk areas just in case
71. the allegedly standard approach n al OO e IEEE Standard 829 for Software Test Documentation Test plan Test design specification Test case specification e Test case specification identifier We often SCE e Test items one or more e Input specifications pages per e Output specifications e Environmental needs e Special procedural requirements e Intercase dependencies Test procedure specification test case Test item transmittal report Test log Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Problems with the allegedly standard approach _ s_ ss OOO e What is the documentation cost per test case e What is the maintenance cost of the documentation per test case e If software design changes create documentation maintenance costs how much inertia do we build into our system How much does extensive test documentation add to the cost of late improvement of the software How much should we add e What inertia is created in favor of invariant regression testing e Is this incompatible with exploratory testing Do we always want to discourage exploration Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved Problems with the allegedly standard approach Doo OOO e What is the impact on high volume test automation e How often do project teams start to follow 829 but then give it up
72. the program The program responds by moving to a new state Test whether the program has reached the expected State e Paradigmatic case s I haven t done this kind of work Here s what I understand Works poorly for a complex product like Word Likely to work well for embedded software and simple menus think of the brakes of your car or walking a control panel on a printer In general well suited to a limited functionality client that will not be powered down or rebooted very often e Maintenance is a critical issue because design changes add or subtract nodes forcing a regeneration of the model Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved l 56 Random Testing Model based Stochastic Tests s OO Alan Jorgensen Software Design Based on Operational Modes Ph D thesis Florida Institute of Technology The applicability of state machine modeling to mechanical computation dates back to the work of Mealy Mealy 1955 and Moore Moore 1956 and persists to modern software analysis techniques Mills et al 1990 Rumbaugh et al 1999 Introducing state design into software development process began in earnest in the late 1980 s with the advent of the cleanroom software engineering methodology Mills et al 1987 and the introduction of the State Transition Diagram by Yourdon Yourdon 1989 A deterministic finite automata DFA is a state machine that
73. the risk analysis is wrong 3 Test strategy should address test platform configuration how the product will be operated how the product will be observed and how senate will be used to evaluate a Test Planning The later in the project that a problem is found the greater the risk that it will not be safely fixed in time to ship The sooner a problem is found after it is created the lesser Complete testing Is sipessble and we can never know if our perception of technical risk is completely accurate Sloppiness or neglect within any of these four basic testing activities will increase the likelihood that important problems will go undetected Copyright 2001 Cem Kaner and James Bach All rights reserved 0 l Heuristics for Test Plan Evaluation Coo ee 4 Test strategy should be diversified in terms of test techniques and perspectives Methods of evaluating test coverage should take into account multiple dimensions of coverage including structural functional data platform operations and requirements 5 The test strategy should specify how test data will be designed and generated No single test technique can reveal all important problems in a linear fashion We can never know for sure if we have found all the problems that matter Diversification minimizes the risk that the test strategy will be blind to certain kinds of problems Use diverse half measures to go after low hanging fruit It is common
74. valuating Your Plan Context Free Questions Loo o What have others done Can you intuit the solution Can you check the results What should be done How should it be done Where should it be done When should it be done Who should do it What do you need to do at this time Who will be responsible for what Can you use this problem to solve some other problem What is the unique set of qualities that makes this problem what it is and none other What milestones can best mark your progress How will you know when you are successful How conclusive and specific is your answer Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 111 Appendix on General Test Techniques Loo O e The following slides review 10 general test techniques In previous talks we ve called these paradigms because many companies have organized their entire testing effort and testing thinking around one or two of them e We won t discuss many of these slides in the workshop but we hope that these will be helpful reference materials to add some detail to comments that we make about these techniques in class e There is nothing magical about these techniques They overlap They don t collectively cover everything that would be good to do e Imagine that you are one of the people who has adopted one of these techniques as your primary approach your parad
75. y e For example Who will use this feature What does this user want to do with it Who else will use it Why Who will choose not to use it What do they lose What else does this user want to do in conjunction with this feature Who is not allowed to use this product or feature why and what security is in place to prevent them e We use these in conjunction with questions that come out of the testing model see below The model gives us a starting place We expand it by asking each of these questions as a follow up to the initial question Rapid Test Planning Copyright 2001 Cem Kaner and James Bach All rights reserved 1 7 Context Free Questions Defining the Problem Loo o Based on The CIA s Phoenix Checklists Thinkertoys p 140 and Bach s Evaluation Strategies Rapid Testing Course notes Why is it necessary to solve the problem What benefits will you receive by solving the problem What is the unknown What is it that you don t yet understand What is the information that you have What is the source of this problem Specs Field experience An individual stakeholder s preference Who are the stakeholders How does it relate to which stakeholders What isn t the problem Is the information sufficient Or is it insufficient Or redundant Or contradictory Should you draw a diagram of the problem A figure Rapid Test Planning Copyright 2001 Cem Kaner

Rapid Test Planning - Cem Kaner, JD, Ph.D.

Contents

Download Pdf Manuals

Related Search

Related Contents