slide - Testing Education
Contents
1. Try clicking everywhere Bach broke into a touchscreen system once by poking every square centimeter of every screen until he found a secret button Multiple Instances Run a lot of instances of the application at the same time Open the same files Risk Based Testing QAI Copyright 2008 Cem Kaner 4 Even More QuickTests from Bach Bolton e Feature Interactions Discover where individual functions interact or share data Look for any interdependencies Tour them Stress them Bach once crashed an app by loading up all the fields in a form to their maximums and then traversing to the report generator Cheap Tools Learn how to use InCtrl5 Filemon Regmon AppVerifier Perfmon Task Manager all of which are free Have these tools on a thumb drive and carry it around Also carry a digital camera Bach carries a tiny 3 megapixel camera and a tiny video camera in his coat pockets He uses them to record screen shots and product behaviors Elisabeth Hendrickson suggests several additional tools at http www bughunting com bugtools html Resource Starvation Progressively lower memory and other resources until the product gracefully degrades or ungracefully collapses Risk Based Testing QAI Copyright 2008 Cem Kaner 42 Even More QuickTests from Bach Bolton e Play Writer Sez Look in the online help or user manual and find instructions about how to perform some interesting activity Do those actions Then impro2. e Put the printer on local e Put a database under use by a competing program lock a record so that it can t be accessed yet Risk Based Testing QAI Copyright 2008 Cem Kaner 35 Additional QuickTests Interference testing Swap out of memory e Swap the process out of memory while it s running e g change focus to another application keep loading or adding applications until the application under test is paged to disk Leave it swapped out for 0 minutes or whatever the timeout period is Does it come back What is its state What is the state of processes that are supposed to interact with it Leave it swapped out much longer than the timeout period Can you get it to the point where it is supposed to time out then send a message that is supposed to be received by the swapped out process then time out on the time allocated for the message What are the resulting state of this process and the one s that tried to communicate with it e Swap a related process out of memory while the process under test is running Risk Based Testing QAI Copyright 2008 Cem Kaner 36 Additional QuickTests Interference testing Compete e Compete for a device such as a printer put device in use then try to use it from software under test start using device then use it from other software f there is a priority system for device access use software that has higher same and lower priority access
3. or to cost too much or to dissatisfy customers or other stakeholders e Analysis of the potential costs associated with each risk e Development of plans and actions to reduce the likelihood of the risk or the magnitude of the harm e Continuous assessment or monitoring of the risks or the actions taken to manage them Useful material available free at http seir sei cmu edu http www coyotevalley com Brian Lawrence The problem for our purposes is that this level of analysis doesn t give us much guidance as to how to test Risk Based Testing QAI Copyright 2008 Cem Kaner 52 Project level risk analysis e Might not give us much guidance about how to test e But it might give us a lot of hints about where to test e f you can imagine a potential failure e In many cases that failure might be possible at many different places in the program e Which should you try first Sometimes risks associated with the project as a whole or with the staff or management of the project can guide our testing Risk Based Testing QAI Copyright 2008 Cem Kaner 53 Project risk heuristics Where to look for errors New things less likely to have revealed its bugs yet New technology same as new code plus the risks of unanticipated problems Learning curve people make more mistakes while learning Changed things same as new things but changes can also break old code Poor control without SCM files can be overr
4. redundant This test represents a larger group that address the same risk Motivating Your client will want to fix the problem exposed by this test Maintainable Easy to revise in the face of product changes Repeatable Easy and inexpensive to reuse the test Risk Based Testing QAI Copyright 2008 Performable Can do the test as designed Refutability Designed to challenge basic or critical assumptions e g your theory of the user s goals is all wrong Coverage Part of a collection of tests that together address a class of issues Easy to evaluate Supports troubleshooting Provides useful information for the debugging programmer Appropriately complex As a program gets more stable use more complex tests Accountable You can explain justify and prove you ran it Cost Includes time and effort as well as direct costs Opportunity Cost Developing and performing this test prevents you from doing other work Cem Kaner 9 Test design Techniques convey vision of a well designed test e Scenario testing e complex stories that capture how the program will be used in real life situations Good scenarios focus on validity complexity credibility motivational effect The scenario designer might care less about power maintainability coverage reusability e Risk based testing e Imagine how the program could fail and try to get it to fail that way e Good risk based tests are powerful
5. valid non redundant and aim at high stakes issues refutability e The risk based tester might not care as much about credibility representativeness performability we can work on these after if a test exposes a bug Risk Based Testing QAI Copyright 2008 Cem Kaner 0 Preliminaries on test design Tying this together Design to create fashion execute or construct according to plan to conceive and plan out in the mind Websters Designing is not scripting The representation of a plan is not the plan Usually or at least preferably within the context of a testing strategy e the test designer uses the test ideas guidance contained in a test technique to craft a specific test that helps her collect a specific type of information answer a reasonably specific question Risk Based Testing QAI Copyright 2008 Cem Kaner Risk Based Design e We often go from technique to test Find all variables domain test each Find all spec paragraphs make a relevant test for each Find all lines of code make a set of tests that collectively includes each e It is much harder to go from a failure mode to a test The program will crash The program will have a wild pointer How do we The program will have a memory leak map from The program will be hard to use a failure The program will corrupt its database mode to a test Risk Based Testing QAI C
6. very promising empirical research Risk Based Testing QAI Copyright 2008 Cem Kaner 60 Risk based testing Failure Modes Failure Mode amp Effects Analysis FMEA failure mode way that the program could fail Example Portion of analysis for an installer product e Wrong files installed temporary files not cleaned up old files not cleaned up after upgrade unneeded file installed needed file not installed correct file installed in the wrong place e Files clobbered older file replaces newer file user data file clobbered during upgrade e Other apps clobbered file shared with another product is modified file belonging to another product is deleted Risk Based Testing QAI Copyright 2008 Cem Kaner 62 Failure mode amp effects analysis Widely used for safety analysis of goods Consider the product in terms of its components For each component e Imagine the ways it could fail For each potential failure each failure mode ask questions What would that failure look like How would you detect that failure How expensive would it be to search for that failure Who would be impacted by that failure How much variation would there be in the effect of the failure How serious on average would that failure be How expensive would it be to fix the underlying cause e On the basis of the analysis decide whether it is cost effective to search
7. Cem Kaner 45 Risk The possibility of suffering harm or loss In software testing we think of risk on three dimensions e A way the program could fail technically this is the hazard or the failure mode but I ll often refer to this as the risk because that is SO common among testers e How likely it is that the program could fail in that way e What the consequences of that failure could be For testing purposes the most important Is For project management purposes How likely e What consequences A way the progi could fail Risk Based Testing QAI Copyright 2008 Cem Kaner 46 For testing A way the program could fail The essence of risk based testing is this 1 Jai Imagine how the product could these potential failures I 2 Design tests to expose Risk Based Testing QAI Copyright 2008 Cem Kaner 41 Just one little problem Imagine how the product could fail How do you do that N Risk Based Testing QAI Copyright 2008 Cem Kaner 48 Just one problem Imagine how the product could fail How do you do that We ll consider three classes of heuristics e Apply common techniques quicktests or attacks to take advantage of common errors we did that already e Recognize common project warning signs and test things associated with the risky aspects of the project e Apply failure mode and effects analysis to many or all elem
8. Improve the Power of Your Tests with Risk Based Test Design Cem Kaner J D Ph D Professor of Software Engineering Florida Institute of Technology Risk Based Testing QAI Copyright 2008 Cem Kaner Conference Abstract Risk based test management evaluates each area of a product and allocates higher testing budgets for areas of greater risk Once you have the budget how should you spend it Risk based test design on the other hand is based on the idea that every test presents the program with an opportunity to fail The first core task of risk based test design is to imagine ways the program can fail The second task is to design tests that are effective triggers for those failures The most powerful tests are the ones that maximize a programs opportunity to fail In this keynote Cem will survey techniques for stretching your failure related imagination such as using guideword heuristics as is commonly done in Failure Mode and Effects Analysis for example to quickly gain and apply knowledge about e The type of application e The environment and programming language e The projects management development and support history Cem will also examine ways of turning ideas about potential failure into tests ranging from quicktests straightforward applications of a theory of error such as Whittaker s standard attacks through tests that are more tightly customized to the specific concern Join Cem and learn how to plan to m
9. They help generate test ideas but are not suited for classifying test ideas Risk Based Testing QAI Copyright 2008 Cem Kaner 23 Building failure mode lists from product elements Shopping cart example Think in terms of the components of your product e Structures Everything that comprises the logical or physical product Database server Cache server e Functions Everything the product does Calculation Navigation Memory management Error handling e Data Everything the product processes Human error retailer Human error customer e Operations How the product will be used Upgrade Order processing e Platforms Everything on which the product depends Risk Based Testing QAI Copyright 2008 Cem Kaner 14 FMEA amp quality attributes In FMEA we list a bunch of things components of the product under test we could test and then figure out how they might fail Quality attributes cut across the components e Usability Easy to learn Reasonable number of steps Accessible to someone with a disability Auditory Visual magine evaluating every product element in terms of accessibility to someone with a visual impairment Risk Based Testing QAI Copyright 2008 Cem Kaner 15 Using a failure mode list Test idea generation Find a potential bug failure mode in the list e Ask whether the software under test could have this bug e fit is theoretica
10. ailure into a story and gradually evolve the story into something you can test from This is what we did with Joe and the Fridge A story is easier for some people to work with than a technologically equivalent but inhuman description of a failure e There are no guarantees in this but you get better at it as you practice and as you build a broader inventory of techniques Risk Based Testing QAI Copyright 2008 Cem Kaner 16 SO HOW DO WE DESIGN RISK BASED TESTS Risk Based Testing QAI Copyright 2008 Cem Kaner 7 Risk based testing OuickTests Simple Risk Derived Test Techniques QuickTests A quicktest is a cheap test that has some value but requires little preparation knowledge or time to perform e Participants at the 7th Los Altos Workshop on Software Testing Exploratory Testing 1999 pulled together a collection of these e James Whittaker published another collection in How to Break Software e Elisabeth Hendrickson teaches courses on bug hunting techniques and tools many of which are quicktests or tools that support them Risk Based Testing QAI Copyright 2008 Cem Kaner 19 A Classic QuickTest The Shoe Test Find an input field move the cursor to it put your shoe on the keyboard and go to lunch Basically you re using the auto repeat on the keyboard for a cheap stress test e Tests like this often overflow input buffers In Bach s favorite variant he finds a dialog bo
11. ake it fail Risk Based Testing QAI Copyright 2008 Cem Kaner 2 Preliminaries on Test design What s testing Software testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test We design and run tests in order to gain useful information about the products quality Risk Based Testing QAI Copyright 2008 Cem Kaner 3 Preliminaries on test design Test and test case Think of a test as a question that you ask the program e You run the test the experiment in order to answer the question A test case is a test e Usually when we just say a test we mean something we do e Usually when we say test case we mean something that we have described documented A test idea is the thought that guides our creation of a test For example what s the boundary of this variable Can we test it is a test idea For our purposes today the distinction between test and test case is irrelevant and will switch freely between the two terms Risk Based Testing QAI Copyright 2008 Cem Kaner 4 Preliminaries on test design Testing strategy Given an information objective My client wants me to find the most bugs My client wants to know if the program meets the specification The testing strategy specifies an integrated view of such things as e The techniques we ll rely on to help us generate tests t
12. ay help or hinder your test design process Iry to exploit every resource Customers Anyone who is a client of the test project Do you know who your customers are Whose opimons matter Who benetits or suffers from the work you do Do you have contact and communication with your customers Maybe they can help you test Maybe your customers have strong ideas about what tests you should create and mm Maybe they have conflicting expectations You may have to help identify and resolve those Information nformation about the product or project that is needed for testing Are there any engineering documents available User manuals Web based materials Does this product have a history Old problems that were fixed or deferred Pattern of customer complaints Do you need to familianze yourself with the product more before you will know how to test it Is your information current How are vou apprised of new or changing information Is there any complex or challenging part of the product about which there seems strangely little information a Relations How you get along with the programmers Hubris Does the development team seem overconfident about any aspect of the product Defenstveness Is there any part of the product the developers seem strangely opposed to having tested apport Have you developed a inendly working relationship with the programmers feedback loop Can you communicate quickly on demand wit
13. ents people realize what they want as the product develops Adhering to a start of the project requirements list may meet the contract but yield a failed product Buggy anything known to have lots of problems has more Recent failure anything with a recent history of problems Upstream dependency may cause problems in the rest of the system Downstream dependency sensitive to problems in the rest of the system Risk Based Testing QAI Copyright 2008 Cem Kaner 56 Project risk heuristics Where to look for errors Distributed anything spread out in time or space that must work as Unit Open ended any function or data that appears unlimited Complex what s hard to understand is hard to get right Language typical errors such as wild pointers in C Little system testing untested software will fail Little unit testing programmers normally find and fix most of their own bugs Previous reliance on narrow testing strategies can yield a many version backlog of errors not exposed by those techniques Weak test tools if tools don t exist to help identify isolate a class of error e g wild pointers the error is more likely to survive to testing and beyond Risk Based Testing QAI Copyright 2008 Cem Kaner 57 Project risk heuristics Where to look for errors Unfixable bugs that survived because when they were first reported no one knew how to fix them in the time available Untestable anything that re
14. ents of the product and to the product s key quality criteria We call these heuristics because they are fallible but useful guides You have to exercise your own judgment about which to use when Risk Based Testing QAI Copyright 2008 Cem Kaner 49 Risk based testing Project Level Risk Factors Classic project level risk analysis Acrobat Reader tutorial from sei 2 pdf File Edit View Tools Window Help 15 x E e ey all ia gt or ol GC al i Mission and goals Cost overruns Decision drivers Schedule slips ss Organization management Inadequate functionality Customer end user Canceled projects Budget cost Sudden personnel Schedule changes Project characteristics Customer dissatisfaction Loss of company image Demoralized staff Poor product performance Legal proceedings Development process Development environment Personnel Operational environment New technology EEE HPA TeraQuest b D Page 12 of 53 145 E 9 SEPG Risk Workshop 1998 TeraQuest Lu 28 in j4 Project level risk analyses usually consider risk factors that can make the project as a whole fail and how to manage those risks Risk Based Testing QAI Copyright 2008 Cem Kaner 5 Project level risk analysis Project risk management involves e Identification of the different risks to the project issues that might cause the project to fail or to fall behind schedule
15. for this potential failure Risk Based Testing QAI Copyright 2008 Cem Kaner 63 failure mode amp effects analysis FMEA Several excellent web pages introduce FMEA and SFMEA software FMEA http www fmeainfocentre com http www fmeainfocentre com presentations SFMEA IIE pdf http www fmeainfocentre com papers mackel pdf http www quality one com services fmea php e http www visitask com fmea asp http healthcare isixsigma com library content c0403 7a asp http www qualitytrainingportal com resources fmea e http citeseer ist psu edu 69 7 html FMEA As some of the papers presentations on the preceding slide note one of the key weaknesses of FMEA in practice is e It is hard to identify all the ways the product can fail e So we get a long but not necessarily thorough list of failure modes e This can be misleading In traditional industries e g automotive this type of analysis is guided by long experience with failures in the field and failures discovered in manufacturing e In the absence of strong records for a particular product how do we generate a strong failure mode list for software e The next several subsections of this presentation leading up to Bach s heuristic test strategy model provide a series of ideas Risk Based Testing QAI Copyright 2008 Cem Kaner 65 Bug catalogs Testing Computer Software included an appendix that listed almost 500 commo
16. ge data with something other than the user such as with other programs hard disk network printer etc pplication any function that defines or distinguishes the product or fulfills core requirements Calculation any anthmetic function or anthmetic operations embedded m other functions Jime related time out settings daily or month end reports nightly batch jobs time zones busimess holidays interest calculations terms and warranty periods chronograph functions Transformations functions that modify or transform something e g setting fonts mserting clip art withdrawing money from account Startup Shutdown each method and interface for invocation and imtalization as well as exiting the product Multimedia sounds bitmaps videos or any graphical display embedded m the product Error Handling any functions that detect and recover from errors mcluding all error messages Jnteractions any interactions or interfaces between functions within the product Testability any functions provided to help test the product such as diagnostics log files asserts test menus etc Structure Functions Data Platform Operations Time Project Environment Creating and executing tests is the heart of the test project However there are many factors in the project environment that are critical to your decision about what par ticular tests to create In each category below consider how that factor a m
17. h the programmers feedback What do the developers think of your test strategy Customers Information Developer relations Test team Equipment amp tools Schedule Test items Deliverables Risk Based Testing QAI Copyright 2008 Cem Kaner 10 Quality Criteria Categories quality criterion is some requirement that defines what the product should be By looking thinking about different kinds of criteria you will be better able to plan tests that discover important problems fast Each of the items on this list can be thought of as a potential risk area For each item below determine if it 1s important to your project then think how you would recognize if the product worked well or poorly in that regard Operational Criteria 1 Capability Can if perform the required functions Jq Reliability Will it work well and resist failure in all required situations Error handling the product resists failure in the case of errors 15 graceful when it fails and recovers readily Data Integritv the data in the system is protected from loss or corruption Safety the product will not fail in such a way as to harm life or property Operational criteria Capability Reliability Usability Security Scalability Performance Installability Compatibility Development criteria Supportability Testability Maintainability Portability Localizability Risk Based Testing QAI Copyright 2008 Cem Kaner II No
18. hat are best suited to giving us the type of information we need e The logistical support resources needed and available at different times in the project e The human support staff and their skills other sources of information people who will help you get the resources staff you need etc Risk Based Testing QAI Copyright 2008 Cem Kaner 5 Preliminaries on test design Information objectives Find important bugs to get them fixed Assess the quality of the product Help managers make release decisions Block premature product releases Different objectives drive Help predict and control costs of product support YOu toward different Check interoperability with other products Testing techniques Find safe scenarios for use of the product Project mgmt styles Assess conformance to specifications Results reportin Certify the product meets a particular standard methods Ensure the testing process meets accountability standards Politi cS on th Minimize the risk of safety related lawsuits project Help clients improve product quality amp testability Help clients improve their processes Evaluate the product for a third party Risk Based Testing QAI Copyright 2008 Cem Kaner Preliminaries on test design Test techniques test technique is essentially a recipe or a model that guides us in creating specific tests Examples of common test techniques e Function testing e Data flow testing e Spec
19. hat aren t directly tied to a simple fault model Many of the ideas in these notes were reviewed and extended by the other LAWST 7 participants Brian Lawrence Ill Jack Falk Drew Pritsker Jim Bampos Bob Johnson Doug Hoffman Chris Agruss Dave Gelperin Melora Svoboda Jeff Payne James Tierney Hung Nguyen Harry Robinson Elisabeth Hendrickson Noel Nyman Bret Pettichord amp Rodney Wilson We appreciate their contributions Risk Based Testing QAI Copyright 2008 Cem Kaner 30 Additional QuickTests Interference testing We look at asynchronous events here One task is underway and we do something to interfere with it In many cases the critical event is extremely time sensitive For example e An event reaches a process just as just before or just after it is timing out or just as before during after another process that communicates with it will time out listening to this process for a response Just as if special code is executed in order to accomplish the handling of the timeout just as means during execution of that code e An event reaches a process just as just before or just after it is servicing some other event e An event reaches a process just as just before or just after a resource needed to accomplish servicing the event becomes available or unavailable Risk Based Testing QAI Copyright 2008 Cem Kaner 3 Additional QuickTests Interference testing Generate interr
20. idden or lost Late change rushed decisions rushed or demoralized staff lead to mistakes Rushed work some tasks or projects are chronically underfunded and all aspects of work quality suffer Fatigue tired people make mistakes Distributed team a far flung team communicates less Risk Based Testing QAI Copyright 2008 Cem Kaner 54 Project risk heuristics Where to look for errors Other staff issues alcoholic mother died two programmers who won t talk to each other neither will their code Surprise features features not carefully planned may have unanticipated effects on other features Third party code external components may be much less well understood than local code and much harder to get fixed Unbudgeted unbudgeted tasks may be done shoddily Ambiguous ambiguous descriptions in specs or other docs lead to incorrect or conflicting implementations Conflicting requirements ambiguity often hides conflict result is loss of value for some person Risk Based Testing QAI Copyright 2008 Cem Kaner 55 Project risk heuristics Where to look for errors Mysterious silence when something interesting or important is not described or documented it may have not been thought through or the designer may be hiding its problems Unknown requirements requirements surface throughout development Failure to meet a legitimate requirement is a failure of quality for that stakeholder Evolving requirem
21. ification based testing e Build verification testing e Domain testing e State model based testing e Risk based testing e High volume automated e Scenario testing testing e Regression testing e Printer compatibility testing e Testing to maximize statement and branch coverage e Stress testing e User testing e All pairs combination testing Risk Based Testing QAI Copyright 2008 Cem Kaner l Test design Examples of test techniques e Scenario testing Tests are complex stories that capture how the program will be used in real life situations Specification based testing Check every claim made in the reference document such as a contract specification Test to the extent that you have proved the claim true or false e Risk based testing A program is a collection of opportunities for things to go wrong For each way that you can imagine the program failing design tests to determine whether the program actually will fail in that way Risk Based Testing QAI Copyright 2008 Cem Kaner 8 Test design Techniques differ in how to define a good test Power When a problem exists the test will reveal it Valid When the test reveals a problem it is a genuine problem Value Reveals things your clients want to know about the product or project Credible Client will believe that people will do the things done in this test Representative of events most likely to be encountered by the user Non
22. ion org coursenotes amland_stale cm_200212_ exploratorytesting e Carl Popper Conjectures amp Refutations e James Whittaker How to Break Software e Giri Vijayaraghavan s papers and thesis on bug taxonomies at http www testingeducation org articles Risk Based Testing QAI Copyright 2008 Cem Kaner 18 About Cem Kaner e Professor of Software Engineering Florida Tech e Research Fellow at Satisfice Inc lve worked in all areas of product development programmer tester writer teacher user interface designer software salesperson organization development consultant as a manager of user documentation software testing and software development and as an attorney focusing on the law of software quality Senior author of three books e Lessons Learned in Software Testing with James Bach amp Bret Pettichord e Bad Software with David Pels e Testing Computer Software with Jack Falk amp Hung Quoc Nguyen My doctoral research on psychophysics perceptual measurement nurtured my interests in human factors usable computer systems and measurement theory Risk Based Testing QAI Copyright 2008 Cem Kaner 19
23. isk Based Testing QAI Copyright 2008 Cem Kaner 4 Design Evolving the test case from the story Second question What makes a system crash Data overflow too much stuff in the fridge Wild pointer grunge accumulates because we ve used the fridge too long without rebooting Stack overflow what could cause a stack overflow Ask the programmers Unusual timing condition Can we create a script that lets us adjust timing of our input to the fridge Unusual collection of things in the fridge e If you had a real customer who reported this problem you MIGHT be able to get some of this information from them But in risk based testing you don t have that customer You just have to work backwards from a hypothetical failure to the conditions that might have produced it Each set of conditions defines a new test Risk Based Testing QAI Copyright 2008 Cem Kaner 5 How to map from a test idea to a test e When it is not clear how to work backwards to the relevant test four tactics sometimes help Ask someone for help Ask Google for help Look for discussions of the type of failure look for discussions of different faults and see what types of failures they yield Review your toolkit of techniques searching for a test type with relevant characteristics For example if you think it might be a timing problem what techniques help you focus on timing issues Turn the f
24. lly possible that the program could have the bug ask how you could find the bug if it was there e Ask how plausible it is that this bug could be in the program and how serious the failure would be if it was there If appropriate design a test or series of tests for bugs of this type Risk Based Testing QAI Copyright 2008 Cem Kaner 16 Using a failure mode list Test plan auditing e Pick categories to sample from e From each category find a few potential defects in the list e For each potential defect ask whether the software under test could have this defect e fit is theoretically possible that the program could have the defect ask whether the test plan could find the bug if it was there Getting unstuck e Look for classes of problem outside of your usual box Training new staff e Expose them to what can go wrong challenge them to design tests that could trigger those failures Risk Based Testing QAI Copyright 2008 Cem Kaner 11 Risk based testing Some papers of interest e Stale Amland Risk Based Testing http www amland no WordDocuments EuroS TAR99Paper doc e James Bach Reframing Requirements Analysis e James Bach Risk and Requirements Based Testing e James Bach James Bach on Risk Based Testing Stale Amland amp Hans Schaefer Risk based testing a response at http www satisfice com e Stale Amland s course notes on Risk Based Agile Testing December 2002 at http www testingeducat
25. n bugs actually failure modes The list evolved across several products and companies It was intended to be a generic list more of a starting point for failure mode planning than a complete list To be included in the list e A particular failure mode had to be possible in at least two significantly different programs e A particular failure mode had to be possible in applications running under different operating systems we occasionally relaxed this rule You can find the TCS 2 edition list appendix on Hung Neguyen s site http www logigear com resources articles_lg Common_Software_Erro rs pdf fileid 2458 Risk Based Testing QAI Copyright 2008 Cem Kaner 66 Bug catalogs Testing Computer Software included an appendix that listed almost 500 common bugs actually failure modes Some people found this appendix very useful for training staff generating test ideas and supporting auditing of test plans However it was e organized idiosyncratically e its coverage was uneven and e some people inappropriately treated it as a comprehensive list because they didn t understand it or were unable to do the independent critical analysis needed to tailor this to their application Eventually stopped recommending this list even though developed the first edition of it and had found it very useful for several years in favor of an early version of James Bach s Heuristic test strategy model latest version a
26. opyright 2008 Cem Kaner 2 Design Mapping from the failure mode to the test e Imagine that someone called your company s help desk and complained that the program had failed They were working in this part of the program And the program displayed some junk on the screen and then crashed They don t know how to recreate the bug but that s no surprise because they have no testing experience How would you troubleshoot this report In terms of the ski lls you us Working from a fa ke trying to replicate someone else Inadequate report of that failure Risk Based Testing QAI Copyright 2008 Cem Kaner 3 Design Mapping from the test idea to the test e Let s create a slightly more concrete version of this example Joe bought a smart refrigerator that tracks items stored in the fridge and prints out grocery shopping lists One day Joe asked for a shopping list for his usual meals in their usual quantities The fridge crashed with an unintelligible error message e So how to troubleshoot this problem First question What about this error message System level probably part of the crash the programmers won t have useful info for us Application level what messages are possible at this point This leads us to our first series of tests Try to recreate every error message that can come from requesting a shopping list Does this testing suggest anything R
27. quires slow difficult or inefficient testing is probably undertested Publicity anywhere failure will lead to bad publicity Liability anywhere that failure would justify a lawsuit Critical anything whose failure could cause substantial damage Precise anything that must meet its requirements exactly Risk Based Testing QAI Copyright 2008 Cem Kaner 58 Project risk heuristics Where to look for errors Easy to misuse anything that requires special care or training to use properly Popular anything that will be used a lot or by a lot of people Strategic anything that has special importance to your business VIP anything used by particularly important people Visible anywhere failure will be obvious and upset users Invisible anywhere failure will be hidden and remain undetected until a serious failure results Risk Based Testing QAI Copyright 2008 Cem Kaner 59 Project risk heuristics Where to look for errors If you have access to the source code and have programming skills take a look at work on prediction of failure prone files and modules by e Dolores and Wayne Zage Ball State University SERC e Emmet James Whitehead UC Santa Cruz for example S Kim E Whitehead Jr and Y Zhang Classifying Software Changes Clean or Buggy IEEE Transactions on Software Engineering to appear 2008 manuscript available at http www cs ucsc edu ejw papers cc pdf This is very recent and think
28. red Louis Pasteur Quicktests give us straightforward useful examples of tests that are focused on easy application of an underlying theory of error They are just what we need to learn about to start stretching our imagination Risk Based Testing QAI Copyright 2008 Cem Kaner 22 Attacks to expose common coding errors Jorgensen amp Whittaker pulled together a collection of common coding errors many of them involving insufficiently or incorrectly constrained variables They created or identified common attacks to test for these An attack is a stereotyped class of tests optimized around a specific type of error Think back to boundary testing e Boundary testing for numeric input fields is an example of an attack The error is mis specification or mis typing of the upper or lower bound of the numeric input field Risk Based Testing QAI Copyright 2008 Cem Kaner 23 Attacks to expose common coding errors In his book How to Break Software Professor rosoit gh are typi ae andin ing r room only and often sell out within e a Ss Whittaker expanded the list and for each attack discussed How to e When to apply it Break Softwar C What software errors make the attack successful 0010101011011 101 How to determine if the attack exposed a failure d ni de di e How to conduct the attack and e An example of the attack We ll list How to Break Software s attacks here bu
29. t http www satisfice com tools satisfice tsm 4p pdf Risk Based Testing QAI Copyright 2008 Cem Kaner 67 Heuristic Test Strategy Model Project Environment Test Techniques Quality Product Criteria Elements Perceived Quality http www satisfice com tools satisfice tsm 4p pdf _ EE Product Elements Ultimately a product is an experience or solution provided to a customer Products have many dimensions So to test well We must examine those dimensions Each category listed below represents an important and unique aspect of a product Testers who focus on only a few of these are likely to miss important bugs J Structure Everything that comprises the physical product 7 Code the code structures that comprise the product from executables to mdividual routmes Jnterfaces pomts of connection and communication between sub systems Hardware any hardware component that 15 mtegral to the product Non executable files any files other than multimedia or programs like text files sample data or help files Collateral anything beyond software and hardware that 15 also part of the product such as paper documents web links and content packaging license agreements etc E FUNCT Everything that the product does User Interface any functions that mediate the exchange of data with the user e 2 navigation display data entry System Interface any functions that exchan
30. t recommend the book s full discussion Risk Based Testing QAI Copyright 2008 Cem Kaner 24 Attacks to expose common coding errors User interface attacks Exploring the input domain e Attack Apply inputs that force all the error messages to occur e Attack 2 Apply inputs that force the software to establish default values e Attack 3 Explore allowable character sets and data types e Attack 4 Overflow input buffers e Attack 5 Find inputs that may interact and test combinations of their values e Attack 6 Repeat the same input or series of inputs numerous times From Whittaker How to Break Software Risk Based Testing QAI Copyright 2008 Cem Kaner 25 Attacks to expose common coding errors User interface attacks Exploring outputs e Attack 7 Force different outputs to be generated for each input e Attack 8 Force invalid outputs to be generated e Attack 9 Force properties of an output to change e Attack 10 Force the screen to refresh From Whittaker How to Break Software Risk Based Testing QAI Copyright 2008 Cem Kaner 26 Attacks to expose common coding errors Testing from the user interface Data and computation Exploring stored data e Attack Il Apply inputs using a variety of initial conditions e Attack 12 Force a data structure to store too many or too few values e Attack 13 Investigate alternate ways to modify internal data constraints From Whittaker How
31. t reset the system Leave windows and files open Let disk and memory usage mount You re hoping the system ties itself in knots over time e Adjustments Set some parameter to a certain value then at any later time reset that value to something else without resetting or recreating the containing document or data structure e Dog Piling Get more processes going at once more states existing concurrently Nested dialog boxes and non modal dialogs provide opportunities to do this e Undermining Start using a function when the system is in an appropriate state then change the state part way through for instance delete a file while it is being edited eject a disk pull net cables or power cords to an inappropriate state This is similar to interruption except you are expecting the function to interrupt itself by detecting that it no longer can proceed safely Risk Based Testing QAI Copyright 2008 Cem Kaner 40 Even More QuickTests from Bach Bolton e Error Message Hangover Make error messages happen Test hard after they are dismissed Developers often handle errors poorly Bach once broke into a public kiosk by right clicking rapidly after an error occurred It turned out the security code left a 1 5 second window of opportunity for me to access a special menu and take over the system Click Frenzy Testing is more than banging on the keyboard but that phrase wasn t coined for nothing Try banging on the keyboard
32. tes on the Heuristic Test Strategy Model e The individual elements Customers Capability etc are Guide Words e A lot of work has been done applying different types of guide words in safety critical applications HAZOPS depends fundamentally on guidewords e See United States Coast Guard Risk based Decision making Guidelines accessed 2008 March 3 Available from http www uscg mil hq g m risk e guidelines hazop htm for discussion of HAZOPS and other safety critical test analysis techniques e The model is extensive but not exhaustive Giri and Ajay see next slide both had to customize for their applications We including Bach all expected this Risk Based Testing QAI Copyright 2008 Cem Kaner 12 Building a failure mode catalog Giri Vijayaraghavan and Ajay Jha followed similar approaches in developing failure mode catalogs for their M Sc theses available in the lab publications set at www testingeducation org e Identify components They used the Heuristic Test Strategy Model as a starting point Imagine ways the program could fail in this component They used magazines web discussions some corporations bug databases interviews with people who had tested their class of products and so on to guide their imagination Imagine failures involving interactions among components e They did the same thing for quality attributes see next section These catalogs are not orthogonal
33. to Break Software Risk Based Testing QAI Copyright 2008 Cem Kaner 21 Attacks to expose common coding errors Testing from the user interface Data and computation Exploring computation and feature interaction e Attack 14 Experiment with invalid operand and operator combinations e Attack I5 Force a function to call itself recursively e Attack 16 Force computation results to be too large or too small e Attack 7 Find features that share data or interact poorly From Whittaker How to Break Software Risk Based Testing QAI Copyright 2008 Cem Kaner 28 Attacks to expose common coding errors System interface attacks Testing from the file system interface Media based attacks e Attack Fill the file system to its capacity e Attack 2 Force the media to be busy or unavailable e Attack 3 Damage the media Testing from the file system interface File based attacks e Attack 4 Assign an invalid file name e Attack 5 Vary file access permissions e Attack 6 Vary or corrupt file contents From Whittaker How to Break Software Risk Based Testing QAI Copyright 2008 Cem Kaner 29 Additional QuickTests from LAWST Several of the tests we listed at LAWST 7th Los Altos Workshop on Software Testing 1999 are equivalent to the attacks later published by Whittaker He develops the attacks well and we recommend his descriptions In addition LAWST generated several other quicktests including some t
34. to the device before and during attempted use by software under test e Compete for processor attention some other process generates an interrupt e g ring into the modem or a time alarm in your contact manager try to do something during heavy disk access by another process e Send this process another job while one is underway Risk Based Testing QAI Copyright 2008 Cem Kaner 37 Additional QuickTests Follow up recent changes Code changes cause side effects e Test the modified feature change itself e Test features that interact with this one e Test data that are related to this feature or data set e Test scenarios that use this feature in complex ways Risk Based Testing QAI Copyright 2008 Cem Kaner 38 Even More QuickTests Quick tours of the program e Variability Tour Tour a product looking for anything that is variable and vary it Vary it as far as possible in every dimension possible Exploring variations is part of the basic structure of Bach s testing when he first encounters a product e Complexity Tour Tour a product looking for the most complex features and data Create complex files Sample Data Tour Employ any sample data you can and all that you can The more complex the better e from Bach amp Bolton s Rapid Testing Course Risk Based Testing QAI Copyright 2008 Cem Kaner 39 Even More QuickTests from Bach Bolton e Continuous Use While testing do no
35. upts e From a device related to the task e g pull out a paper tray perhaps one that isn t in use while the printer is printing e From a device unrelated to the task e g move the mouse and click while the printer is printing e From a software event e g set another program s or this program s time reminder to go off during the task under test Risk Based Testing QAI Copyright 2008 Cem Kaner 32 Additional QuickTests Interference testing Change something this task depends on e swap out a floppy e change the contents of a file that this program is reading e change the printer that the program will print to without signaling a new driver e change the video resolution Risk Based Testing QAI Copyright 2008 Cem Kaner 33 Additional QuickTests Interference testing Cancel e Cancel the task at different points during its completion e Cancel some other task while this task is running a task that is in communication with this task the core task being studied a task that will eventually have to complete as a prerequisite to completion of this task a task that is totally unrelated to this task Risk Based Testing QAI Copyright 2008 Cem Kaner 34 Additional QuickTests Interference testing Pause Find some way to create a temporary interruption in the task Pause the task e for a short time e for a long time long enough for a timeout if one will arise For example
36. vise from them Often writers are hurried as they write down steps or the software changes after they write the manual Crazy Configs Modify O S configuration in non standard or non default ways either before or after installing the product Turn on high contrast accessibility mode or change the localization defaults Change the letter of the system hard drive Grokking Find some aspect of the product that produces huge amounts of data or does some operation very quickly For instance look a long log file or browse database records very quickly Let the data go by too quickly to see in detail but notice trends in length or look or shape of the patterns as you see them Risk Based Testing QAI Copyright 2008 Cem Kaner 43 Parlour tricks are not risk free These tricks can generate lots of flash in a hurry e The DOS disk I O example e The Amiga clicky click click click example As political weapons they are double edged If people realize what you re doing you lose credibility e Anyone you humiliate becomes a potential enemy Some people incorrectly characterize exploratory testing as if it were a collection of quicktests As test design tools they are like good candy e Yummy e Everyone likes them e Not very nutritious You never get to the deeper issues of the program Risk Based Testing QAI Copyright 2008 Cem Kaner 44 SO HOW DO WE DESIGN RISK BASED TESTS Risk Based Testing QAI Copyright 2008
37. x so constructed that pressing a key leads to say another dialog box perhaps an error message that also has a button connected to the same key that returns to the first dialog box This will expose some types of long sequence errors stack overflows memory leaks etc Risk Based Testing QAI Copyright 2008 Cem Kaner 20 Another Classic Example of a QuickTest Traditional boundary testing e All you need is the variable and its possible values e You need very little information about the meaning of the variable why people assign values to it what it interacts with e You test at boundaries because miscoding of boundaries is a common error Note the foundation level assumption of this test Assumption This is a programming error so common that it s worth building a test technique optimized to find errors of that type Risk Based Testing QAI Copyright 2008 Cem Kaner 2I Why do we care about quicktests Point A You imagine a way the program could fail Point B You have to figure out how to design a test that could generate that failure Getting from Point A to Point B is a creative process It depends on your ability to imagine a testing approach that could yield the test that yields the failure The more test techniques you know and the better you understand them the easier this creative task becomes e This is not some mysterious tester s intuition e Luck favors the mind that is prepa
Download Pdf Manuals
Related Search