WhenToStop Frestimate Module User`s Manual
Contents
1. p e Estimated current MTTF MTTF Start of test is the milestone in which the software has been integrated and is now ready for a system level verification and validation The start of test milestone is denoted by the o subscript By this milestone the developer unit tests and integration tests should already be complete for the particular software package If you have incremental or spiral releases this would be the point in time for which this increment or spiral is ready for a system level test You will probably need to create a new project database for each spiral or increment as well The start date will always be equal to the earliest date in the failure database If you had no defects on the earliest date s of testing you need to add records for these dates and set the defects fields to 0 This will allow the models to take into account periods of time with no failures The end date defaults to the most recent date in the failure database You can override this in this dialog as long as it is not less then the start date The end of test date affects the calculations for e End of Test MTTF MTT Fae WhenToStop Software Users Manual Copyright SoftRel LLC 2008 e End of Test Failure Rate Ager e Estimated end of test reliability The start and end date are very key pieces of information The end date must be greater then or equal to the start date This milestone is denoted by the peL subscript in this manual If you set2. 37 38 42 44 45 46 47 48 49 50 52 53 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 0 Overview WhenToStop used to be a standalone software estimation product It is now an integrated module in the Frestimate software tool set This module is purchased as part of the Frestimate Manager s Edition If you have purchased the Manager s Edition the Test data growth button shown below will be enabled Otherwise it will be dimmed Predictions ye x File Tools Help General Survey Filter Defect Failure MTTF Reliability Availability Trends Reports Compare Cost Test data Field Print inputs inputs for report profile rate profile profile profile j Results scenarios growth Data this model profile Select an Extrapolation w Select a model for predicting defects Atea Enee EE ACS E Y Bounds 80 v E Bounds on defect density prediction critical defects Bounds on defect density prediction all defect types Defect 0 Sze ost Jg os Defect 0 Size 3019 1l sors are g S0 l density density density density as This button is Results filtered for critical defects only Results for all defect types that res how you access the Upper Bound Lower Bound Defect density Nominal Upper aA Start of test defect density 2037 2 052 2 022 predictions are in 40 736 reliability End of test defect density 03 045 O15 bean ceo 604 estimation per i Sta
3. Therefore the inherent defects parameter is used by the Binomial Exponential fault count Exponential time to failure and NHPP models The Weibull model uses it s own parameter estimation to determine inherent defects For the other models the parameter is estimated by plotting cumulative defect rate vs cumulative defects the y intercept of this plot will be the estimated inherent defects Initial failure rate Ao The x intercept will be the estimated initial failure rate which is used by the Binomial Exponential fault count Exponential time to failure and NHPP models The logarithmic models use the ACTUAL observed initial failure rate which is just the cumulative defects divided by the cumulative time for the very first time period in which there were observed defects If you look at the cumulative table in the Record Failures Per Day page this will be the value for failure intensity that it is in the very first row of the table There are 4 ways to determine the y and x intercepts You select one of the below methods from the Estimation gt Detailed Results page e Best Straight LineLeast Squares EstimateWeighted Least Squares EstimateMaximum Likelihood Estimate You may do the modeling with the computed default value for inherent defects or you can modify it as discussed shortly 4 3 2 2 Logarithmic model parameters The exponential models presumes that there is a straight line between the inherent defects estimate No and the ini
4. WhenToStop Present failure Present MTTF Reliability estimation rate estimation estimation for present or end of p test milestone Binomial k No n 1 p Time to kNo e kt 1 Ap Exp A mission Failure Mission mission time Exponential 29 1 n No 1 p fault count NHPP a b exp bt 1 p Logarithmic Ao Ao 0 t 1 T 1 time based To 1 0 01 28G W LA GSS ERG aS EE HO SEM B Logarithmic o exp n T 1 defect T 1 0r 983 Y8 8640 68 TW VATE VM 031 based Weibull b a t b a 1 NoT 1 a 1 OR exp mission b exp tk T 1 a 1 if a 1 then you b 1 a have an exponential model if a 2 then you have a Raleigh model t mission time x appropriate milestone subscript depending on whether we are solving for present or end of test milestone If the milestone is the end of operational growth period then the exponential formula for reliability is used WhenToStop computes all parameters and model results automatically when you enter failure data The model options in this section allow you to e View the parameters estimated for each model e View the results for each model e Temporarily change the inputs to perform sensitivity analysis 29 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 The following functions are provided for each of the models Reset values You can modify the inputs for each model and view the results If you wan
5. an exponential model Note that failure rate and MTTF for this model change only when there is a defect detected and not directly with calendar time 32 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 6 3 Exponential Time to Failure Model aa tJ eBin FREstimate Time To Failure Model Inputs Initial failure rate 445174 Failures Per Hour Cumulative time 640 N estimated inherent defects Nominal 256 611672 LowerBound 206 46165 Upper Bound 306 761694 Predicted MTTF in hours Nominal 989 364e 2 UpperBound 108 032e1 Lower Bound 932 501e 2 Pont y nner gt A Predicted Failure Rate K Nominal f 011e 1 Lower Bound 9 257e 2 Upper Bound 1 072e 1 Failures Per Hour Reset Values Print Close Calculate Values The inputs for the exponential model are e Inherent Defects No e Initial Failure Rate o e Cumulative Test Usage Time t The exponential model assumes that the hazard rate is constant and that faults are equal in severity and probability of being detected Failure rate Ao exp t Ao NO MTTF 1 failure rate since this is an exponential model 33 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 6 4 Logarithmic Time Model Logarithmic time model inputs Decay parameter 00525 Cumulative time 640 Estimated initial failure rate Nominal 445174 Lower Bound 1 53729 Upper Bound K 36619 Failures Pe
6. check the Use this value for total inherent defects and Use this value for initial failure rate and then enter the inherent defects and the initial failure rate WhenToStop will do no checking of these parameters other than to make sure that both are positive numbers You cannot leave either field blank if you check this option 20 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 4 Summary results for all models Select the Summary Results for all models button from the WhenToStop main page shown in section 4 0 The below page is displayed Results of all models Predicted MTTF hours Predicted Failure Rate 5 95 Nominal 5 95 NHPP 125e1 e0 229e0 8 889077e 2 e0 2 364439e 1 Binomial 945e0 199e1 13420 1 117954e 1 8 342388e 2 1 401669e 1 Time To Failure 894e0 08e1 325e0 1 010749e 1 9 256527e 2 1 072383e 1 Fault Count 945e0 025e1 054e0 1 117954e 1 f3 75774le 2 1 241618e 1 Logarithmic timel 025e1 068e1 865e0 9 757826e 2 fa 363706e 2 1 013701e 1 Logarithmic count 389e0 977e0 867e0 1 065049e 1 f 002286e 1 1 127813e 1 Bayesian 085e1 1 085e1 1 085e1 9 21875e 2 9 21 875e 2 9 21875e 2 Weibull 882e2 e0 254e1 1 503e 3 e0 1 094e 1 Failures Per Hour Close This selection will display the inputs to all of the models and will generate output for all model calculations You can use this function to compare the results of each of the models in one summary report You can pri
7. exponential model is used to extrapolate past the end of test The number that is displayed here is calculated based on The model you selected The curve fitting you selected The failure data that you entered The start date you entered in the general inputs 25 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 e The end date you entered in the general inputs e The growth period and duty cycle specified in the general inputs Operational MTTF T Nei exp Q i 1 i exp Q Operational failure rate 1 Operational MTTF Where T duty cycle per month that you input in the Estimation gt General Inputs page Nae NO n number of defects estimated between now and end of test Q growth rate computed by WhenToStop and shown on the estimation parameters page i number ofmonths in growth period that you entered on the General inputs page Note The estimates for the operational milestone are denoted with no subscript 4 5 3 5 Estimated Reliability The estimated current reliability is computed by the following algorithm It is the probability that the software operates over the mission time p estimated present failure rate Adel estimated end of test failure rate operational failure rate Using an exponential formula Estimated current reliability exp Ap mission time Estimated end of test reliability exp Age mission time Estimated operational reliability exp A mission t
8. failure rate and MTTF for this model change only when there is a defect detected and not directly with calendar time You can override the values of a b and time to determine sensitivity on the MTTF and failure rate results Your inputs and outputs are not saved once you close this dialog however you can print them Section 4 8 describes the inputs for this model The NHPP model assumes that faults do not have an equal probability of being detected This model is typically used during the earlier phases of testing usage 31 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 6 2 Binomial Model Binomial Model Inputs k slope parameter 000566 n number defects detected so far 59 MN estimated inherent defects Nominal 256 611672 LowerBound 206 46165 Upper Bound 306 761694 m Predicted MTTF in hours Nominal 894 069e 2 Upper Bound 119 813e 1 Lower Bound 713 098e 2 m Predicted Failure Rate Nominal 1 118e 1 Lower Bound 8 346e 2 Upper Bound 1 402e 1 Failures Per Hour Reset Values Print Close Calculate Values The inputs for the binomial model are e Number of detected defects n e k parameter e Inherent Defects No This model assumes that all faults are equal in probability of being detected and severity and that when a fault is detected it is immediately removed corrected Estimated failure rate k No n Estimated MTTF 1 failure rate since this is
9. opposed to prediction To use WhenToStop you must have observed failure data from testing or operation Predictions on the other hand can be performed without observed failure data The prediction models in Frestimate including full scale ShortCut and Rome Labs compliment WhenToStop in that you can use WhenToStop to validate the predictions made without observed failure data You can also use the results of the predictive models as seeding values for the WhenToStop models It should WhenToStop Software Users Manual Copyright SoftRel LLC 2008 be noted that you can use the estimation models without doing a prediction and vice versa When you select the Test data growth button the below WhenToStop main page will be displayed Estimated MTTF failure rates General Inputs Estimated Inherent Defects 513 911e0 7 Defects estimated T between now and end of test End of testing 4 26 2002 Failures Per Hour Input failures by Defects found so far in testing 119 Estimated Current Failure Rate 1 110403e 1 day Ut SA Estimated Current MTTF 900 574e 2 Hours End of Test Failure Rate 1 110403e 1 Failures Per Hour Parameter estimation End of Test MT TF 900 574e 2 Hours Summary results Operational Failure Rate 3 1291 5e 3 Failures Per Hour eelnes aperin MUTE 319 573e0 Hours Model sensitivity m Other projections Compare results Estimated reliability for missio
10. reliability for mission time specified and operational failure rate Mission time period in hours over which reliability will be measured This field must be a non negative numeric value Mission time should not be confused with testing time Mission time is a finite period of time over which reliability will be measured Let s say your product is a dishwasher with an average cycle time of 60 minutes Then the mission time would be 60 minutes If your product is a medical therapy with an average therapy injection time of 8 hours then the mission time would be 8 hours The results of these outputs are directly related to what you input for mission time e Estimated current reliability e Estimated end of test reliability e Estimated operational reliability WhenToStop Software Users Manual Copyright SoftRel LLC 2008 Desired MTTF The desired MTTF is often abbreviated as MT TF and the desired failure rate as Ap Each of these fields must be a positive and finite numeric value The end of test objective is the objective with no growth at delivery day The desired MTTF after the growth period is the desired MTTF after whatever number of months you have input for the growth period If you enter 0 months of growth then these two values should be the same as there is no growth Desired MTTF is generally computed based on system reliability software reliability system availability software availability objectives or a requirement to remove som
11. the start and end dates so that no date in the failure database falls within the range between the start and end dates an error message will be displayed The Post Delivery Usage inputs directly affect the calculations that are output The prediction menu has a wizard to help you determine the below inputs Number of months in growth period This field must be a non negative integer This is the number of months after delivery in which you expect the failure rate to go down or the MTTF to go up Usually this is equal to the number of months between this release and the next major release While 0 is a valid input it is not recommended If the growth period is set to zero this is equivalent to stating that all inherent defects will be found immediately upon delivery The months in the growth period is used by the following calculations Operational MTTF Operational Failure Rate Estimated operational reliability Number of hours operating per month This field must be a positive numeric value It is the duty cycle and is used to normalize the testing time observed to the operational time that will be observed by the end users If the software will operate continuously by the end user then the default is approximately 730 times the number of fielded units that are running this software The results of these outputs are directly related to what you input for operating hours per month e Operational MTTF e Operational Failure Rate e Estimated
12. the system determines the amount of defects that would need to be discovered for that extrapolation It is calculated based on The model you selected The curve fitting you selected The failure data that you entered The start date you entered in the General Inputs page The objective MTTF you entered in the General inputs page Additional defects are denoted as An in this manual Objective MTTF or failure values are denoted with an obj subscript Binomial and An No Ao Ap Ai Exponential 1 k No Ao models NHPP model An 1 b Ap Ay Logarithmic An 1 6 INn Ap Ar models Weibull model obj a b t b exp t At b Once you solve for At then compute defects associated with that time by solving for b and keeping a the same 28 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 6 Model Sensitivity Select the Model Sensitivity button from the WhenToStop main page shown in section 4 0 The model parameters that are associated with the model that you have selected will then be displayed For example if you have chosen the Binomial model in the pull down menu then the inputs for the Binomial model are shown and you are able to do the sensitivity analysis on those parameters Each of the sensitivity pages is shown in sections 4 6 1 through 4 6 8 The following software reliability models capabilities are included in
13. 74e 2 3 129175e 3 319 573e0 Defects estimated between now and 0 end of test End of testing Failures Per Hour Hours Failures Per Hour Hours Failures Per Hour Hours Estimated reliability for mission time specified and current failure rate Estimated reliability for mission time specified and EOT failure rate Estimated reliability for mission time specified and operational failure rate Estimated availability for current MTTF and MTSWA specified in prediction Estimated availability for end of test MTTF Estimated availability for operational MTTF MTSWR Objective delivered MTTF Test hours needed to reach objective Defects to discover to meet objective The failure rate and MTTFs are for these type Only catastrophic and critical Select the model that you want to see results for Select the curve fitting method Binomial Best Straight Line Help Update results i 4 26 2002 4 113454e 1 4 113454e 1 9 752773e 1 0 8182766 0 8182766 0 9937806 2 1000 167 5e2 391 4e0 4 5 1 Select the model that you want to see results for You can toggle among any of the models selected here to view the most current estimates for each model All model results are stored in the database however only the model that you select in this dialog is displayed on this page or the reports 4 5 2 Select the curve fitting method 22 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 The preferred
14. 857628 5 206 461649948951 95 306 7616937663 Nominal 1 45174e 1 5 1 36619e 1 95 1 53729e 1 Inputs used by the Logarithmic Model m Inputs used by Binomial Model Theta 5 24981 6 3 k slope parameter 5 65733e 4 Inputs used by the NHPP Model anominal 3 92414e2 5 2 85663e2 4 99164e2 b nominal 2 69096e 4 5 2 20755e 3 2 74575e 3 Inputs used by the Weibull Model a nominal 1 25781e0 5 1 25315e0 95 1 26246e0 b nominal 1 26369e 3 5 3 51032e 2 95 3 76306e 2 weibullk parameter 6 27114e 6 BO parameter 6 81123e0 You should not override these values unless you are an advanced user Use this value for total inherent defects 56 611671 and this value for initial failure rate fi 15174e Failures Per Hour coe _ 16 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 3 1 Observed data These are observed data as opposed to estimated parameters They are shown on this page mainly because they are inputs to the estimation models as are the parameters shown below them n number of defects found so far This is the cumulative total of all defects in your database between the start and end date of systems testing If you chose to filter only the severe defects in the global preferences then this value will include only the catastrophic and critical defects in the database between the start and end of systems testing t Cumulative test time This is the total number of cumulative testing hours in your dat
15. MTTF 2 601844E 06 Estimated availability for operational MTTF 0 9994649 MTSWR 6 06253 Objective delivered MTTF 1000 Test hours needed to reach objective 701 4e1 Defects to discover to meet objective 247 5e5 The failure rate and MT TFs are for these type 4ll severity types Select the model that you want to see results for Exponential ooo Help Update results Print Each of these trends is generated from the failure data that you input for a specific project You can print your trend by pressing the Print button on the trend screen 41 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 8 1 Failure Rate MTTF Select the Failure Rate MTTF option from the pull down menu It is the very first option Then you can toggle between the failure rate growth and the MTTF growth graphs as shown below These graphs show an extrapolation of the fielded failure rates and MTTF values given your inputs for end of testing duty cycle and growth period While end of test failure rates are extrapolated by using the selected model the fielded or operational failure rates and MTTFs are extrapolated by using only the exponential model If you have selected 0 as your growth period or duty cycle then these plots will not display a result I Serious Failure Rate over Growth Period L Only Failure rate growth Show entire growth period catastrophic and critical defects C Program Files Frestimate Manag
16. Model Inputs a slope parameter 11 257909 b o01 264 k Rate parameter 000006 Cumulative time hours 640 Predicted Failure Rate m Predicted MTTF Nominal 1 504e 3 Failures Per Hour Nominal 198 131e0 Reset Values Print Close Calculate Values The inputs for the Weibull model are Number of detected defects Test usage time a Weibull parameter b Weibull parameter This model assumes that the failure rate can be increasing or decreasing MTTF b a T 1 a Failure rate a a b t b exp t b 38 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 7 Compare The purpose of the Compare functions are to allow you to compare the estimated values for each of the models for each defect occurrence in the failure databases to the actual next time to failure The actual next time to failure is equal to Time to next failure observed time between last failure and this failure You can view the following only when there is at least two defects recorded in the record failures per day database Select the Compare results button from the WhenToStop main page shown in section 4 0 The model that is selected on the main WhenToStop page will be displayed for the comparison These graphs are useful for determining if a particular model is currently estimating with the lowest relative error and or what points in the testing phase was this model estimat
17. Observed initial failure rate Ao e Theta 0 e Number of detected defects n The logarithmic model assumes that some faults are likely to be detected before others This model is typically used earlier in test usage than other models It assumes that inherent defects is infinite and therefore does not model this parameter Instead it models the initial failure rate and the change in that failure rate theta This model differs from the time based logarithmic model only to the extent that failure rate is estimated as a function of defects as opposed to a function of time Estimated failure rate observed Ao exp 0 n MTTF 0 1 6 observed Ap 0 t 1 1 35 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 6 6 Exponential fault count model z gt tl Recycle Bin FREstimate Fault Count Model Inputs A Initial failure rate ja 45174 n number defects detected so far 59 N estimated inherent defects Nominal 256 611672 Upper Bound 206 46165 Lower Bound 306 761694 m Predicted MTTF in hours Nominal 894 489e 2 Upper Bound 964 432e 2 Lower Bound 952 961e 2 roadrunner D i Predicted Failure Rate SS Nominal f1 118e 1 Lower Bound 1 037e 1 Upper Bound f1 173e 1 Failures Per Hour Reset Values Print Close Calculate Values The inputs for this model are TAAL Observed Initial failure rate o Cumulative test usage time t Nu
18. Rate and MT TPE ee cceeccecsseceeeeeceeceeneecsseeenees 4 5 3 4 Estimated Operational Failure Rate and MT TF cece ceeeceseceecesecneeeeeeneeees 45 3 5 BStimated READ ty soseen eie en e a a E E Soasebvat onto A 4 5 3 6 Test hours needed to reach objective MTTF ssessseessssessereesesrsererrsreerrsrerrsserrreresreee 4 5 3 7 Defects to discover to meet objective MTTF s sessseessereesesrsererrsreerssrerrsserrreresreee 4 6 MODEL SENSITIVITY sssssteciiscstercticciescdiuedecndiendeccsnasiersdedsdeweravednces 4 6 1 NHPP Model 4 6 2 Binomial Model 31 32 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 6 3 Exponential Time to Failure Model 4 6 4 Logarithmic Time Model 4 6 5 Logarithmic fault count Model 4 6 6 Exponential fault count model 4 6 7 Bayesian model 4 6 8 Weibull Model 4 7 COMPARE ssicccissssesstsssinentecacesssecitacesnaecerevesicacedadsdncededidaradativessass 4 8 DEFECT TRENDS iisisti renneri taranan reretia tuens nonett sne taisna 4 8 1 Failure Rate MTTF 4 8 2 All Actuals to Estimates 4 8 3 Most recent actuals to estimates 4 8 4 Defect Trend 4 8 5 Test Time Trend 4 8 6 What will MTTF be after this many more testing hours 4 8 7 Testing needed to remove defects 4 8 8 Staffing trend 4 8 9 Cumulatives Trend 4 8 10 Testing needed to reach some objective 4 9 ADDENDUM ssiiiusisniiinaiunininnanninanniniaanunana naain a mananan anaani naninita 33 34 35 36
19. WhenToStop Software Users Manual Copyright SoftRel LLC 2008 WhenToStop Frestimate Module User s Manual 4 0 OV BRVIEW A 4 1 GENERAL INPU S wstcisccesdnicscersnansnerdutnsneusuansuendeiasneudnansuanduansaeudubn 4 2 INPUT FAILURE DATA sisssiiteissatiietiiniasutiindsesisetannstdedinaiicutinnaiiaie 4 2 1 Record Failures Per Day PATA Pd ed Wal 10 11 0 RAEE REEE E E ee eee ee 4 2 2 Importing Data 4 3 PARAMETERS ESTIMATED eeeccsseseeeeeesseeeeeeeenseeeeeeeeneenes 4 3 1 Observed data 4 3 2 How parameters are estimated 4 3 2 1 Estimated inherent defects and initial failure rate eee eee ceee cee cneeeneeeneeees 4 3 2 2 Logarithmic model parameters Sisirin sia ss 4 3 2 3 Binomial model parameters erens a a 43 24 NPP parameters mer nessa sve titaiige tau E A A ga ee ee 473 25 Weibull parameters osn hist stay neers lead a eee ae eae 4 3 3 Bypassing estimates for inherent defects initial failure rate 4 4 SUMMARY RESULTS FOR ALL MODE LG cccccsssseeereeseeees 4 5 DETAILED RESULTS iscscscecctvcscicetinacttecwcdnvantvewsateanenceecasencaewsienciecs 4 5 1 Select the model that you want to see results for 4 5 2 Select the curve fitting method 4 5 3 Outputs 4 5 31 Detect and test datas vc2cces exces ivediotaceh cus a ced co ist cute seuscaes dubeadcaseheubs tabs E SA 4 5 3 2 Estimated Current Failure Rate and MT TE uu eeeeeeseceeeeecsseceseeeceaecesneecnaeeenees 4 5 3 3 Estimated End of Test Failure
20. abase between the start and end of systems testing Growth rate Qo This is the observed growth rate of the defects in your database between the start and end of systems testing If you chose to filter only the severe defects in the global preferences then this value will include only the catastrophic and critical defects in the database between the start and end of systems testing The growth rate is determined by plotting the failure intensity for each row in the cumulatives table shown in section 4 2 versus the cumulative defects The observed growth rate is the natural log of the x intercept of that graph over the y intercept of that graph 4 3 2 How parameters are estimated 4 3 2 1 Estimated inherent defects and initial failure rate The estimated number of inherent defects No in the software at the start of system test is estimated by WhenToStop based on your collected data Some models assume that inherent defects and initial failure rate is fixed and finite e Binomial e Exponential fault count e Exponential time to failure Some models assume that inherent defects and initial failure rate is finite but not fixed e NHPP e Weibull Some models assume that inherent defects and are infinite e Logarithmic time to failure e Logarithmic fault count 17 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 Some models make no assumption about inherent defects at all e Bayesian Inherent defects No
21. cts This is the estimated parameter associated with the model and the curve fitting method that you selected See section 4 3 for a description Defects found so far in testing This is the actual defects in the database See section 4 3 for a description End of testing date This is the date that you entered on the general inputs page Defects estimated between now and end of testing This is an extrapolation of the present defect count using the model that you selected The extrapolation formula depends on the model selected as shown below Note If the end of testing date has already passed then the above estimate will be 0 23 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 5 3 2 Estimated Current Failure Rate and MTTF These results are calculated based on The model you selected The curve fitting you selected The failure data that you entered The time values that you entered The start date you entered The model for determining the failure rate depends on which model is selected Refer to section 4 6 for the model formulas Please note that MTTF is not always the inverse of the failure rate They can be inverted only when there is a constant hazard rate such as the NHPP exponential and binomial models Parameters associated with the present point in time are often denoted with a subscripted in this manual 24 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 5 3 3 Estimated En
22. curve fitting is the technique used to estimate certain parameters used for some or all of the models These parameters include e k parameter e Inherent Defects No e Initial Failure Rate Ao Whenever you change the curve fitting options the system will automatically recalculate the results that are shown in this page If there is allot of data in the failure database this may take a while The curve fitting options are e Best Straight Line This best straight line can be viewed from the defect trends menu item Select inherent defects trend and then select linear e Least Squares Estimate This technique generally takes less time then the MLE estimation and often times less time then the best straight line Weighted Least Squares EstimateThis technique is just like the LSE except that most recent data points are weighted heavier Maximum Likelihood Estimate This method of estimating parameters will generally take longer to perform then the other calculations 4 5 3 Outputs All MTTF outputs are in terms of hours All failure rate units are defined based on the selection you made in the File gt Global Preferences dialog If you chose hours then all failure rate outputs will be in hours The same applies to million of hours and billion of hours You can change this preference at any time and the failure rate outputs as well as the labels will be updated accordingly 4 5 3 1 Defect and test data Estimated inherent defe
23. d of Test Failure Rate and MTTF The end of test failure rate is an extrapolation of the current failure rate up to the end date that you entered in the preferences The number that is displayed here is calculated based on The model you selected The curve fitting you selected The failure data that you entered The start date you entered in the preferences The end date you entered in the preferences The formulas for extrapolating the present failure rate MTTF to the future failure rate MTTF using At which is the testing hours remaining between now and the end of testing milestone Model Formula to extrapolate the present to end of test Binomial and Apel Ap exp kAt exponential MTTF pei 1 Ap exp kAt models NHPP model Apel Ap exp bAt MTTFpei 1 p exp bAt Logarithmic MTTFpei 0 At MTTF models Apel 1 0 At MTTF Weibull model Ape a b t b exp t At b Parameters associated with the end of test milestone are often denoted with a subscripted peL in this manual Parameters associated with the present time are often denoted with a subscripted p in this manual 4 5 3 4 Estimated Operational Failure Rate and MTTF The operational failure rate is an extrapolation of the end of test failure up to the end of the growth period that you specify in the General Inputs page The model you have selected is used to extrapolate the end of test failure rate but then the
24. dicted 0 14517364242150 Calculate The estimated MTTF at that point is 19 0995 Close Print 48 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 8 7 Testing needed to remove defects Select the Test time removal rate option from the pull down menu This trend shows you how many testing hours total are needed to reach some level of defect removal The percentage is the percentage of testing defects So if 200 defects are predicted for testing then 95 of that would be 190 defects t additional testing hours No Ao IN Ao Aobj Where No Defects predicted for start of testing obj objective failure rate 1 x Ao o failure predicted at start of testing t cumulative testing hours so far Testing time needed to reach a removal Inputs Inherent defects predicted at start of testing 256 61167185 762 Testing growth rate Q0 7 477389 Initial failure rate predicted 0 1451 736 Percentage of defects expected to be removed 95 by end of test Testing hours exhausted so far 640 Additional test hours needed to reach the above defect removal 4655 313 Failure rate predicted at the above defect removal 7 25868E 03 Predicted MTTF at the above removal rate 137 7661 Close 49 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 8 8 Staffing trend Select the Staffing trend option from the pull down menu The average repair time and number of correctiv
25. e actions made so far are user inputs which determine the other calculations shown The valid input range for average repair time is any number gt 0 The valid input range for the number of corrective actions made is gt 0 and lt number of defects detected so far The number of defects detected so far comes directly from the defect database and is the cumulative number that you have recorded in the failure database that are between the start and end of testing date If you have set the global preferences to filter for only the severe types then this will be the cumulative sum of only the defects that have a catastrophic or critical classification The average testing time per week is also determined directly from the defect database The time in hours spent so far t is also determined directly from the defect database The total time in hours required to find all defects is the y intercept of the inherent time graph discussed next The time in hours required to find the remaining defects inherent time t The total weeks to find the remaining defects inherent time average testing time per week The estimated defects to be fixed inherent defects number of defects corrected so far The total corrective action time needed to fix these defects estimated defects to be fixed average repair time 50 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 Correction action staffing required to address pred
26. e specific percentage of the defects in the software prior to shipment The results of these two outputs are directly related to the desired MTTF e Test hours needed to reach objective MTT Fob e Defects to discover to meet objective MTT Fob WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 2 Input Failure Data To use the estimation models you must input failure data from systems testing There are two ways to input this data You can enter it directly one day at a time Record failures per day or you can import several days worth of testing data at once 4 2 1 Record Failures Per Day Select the Input Failures By Day button from the WhenToStop main page shown in section 4 0 Input the daily test time and failure information Daily inputs Date ls M20 2001 Number of catastrophic defects found today f Number of serious defects found today fo Testing hours Number of moderate defects found today fo during this day Number of negligible defects found today fo Append Previous Next First Last Record eg Record Record record record Number Baus Print Close Bypass calculations Cumulatives table Table view WhenToStop Software Users Manual Copyright SoftRel LLC 2008 Input the daily test time and failure information Daily inputs Date ja 30 2001 Number of catastrophic defects found today f Testing hours 24 Number of serious defects found today fo Nu
27. ein ke ta tt ole bill lt gt Bayes bl A Achal Export image Copy image Print to file to clipboard 45 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 8 4 Defect Trend Select the Defect trend option from the pull down menu The Cumulative defects vs Failure intensity trend is significant because it is the curve that the inherent faults initial failure rate and k parameters are derived from As your software testing usage progresses you should see this curve progress towards the top left hand corner The y and x intercepts are determined by the curve fitting method that you selected in the Estimation gt Detailed Results page You can select from one of these 4 methods Best Straight Line Least Squares Estimate Weighted Least Squares Estimated Maximum Likelihood Estimate Rate vs Cumulative Defects Cumulative defects The curve fitting method that you have chosenis Best Straight Line The estimated inherent defects for this curvefitting method is 256 611671857628 Export ee Copy Print cose to file to clipboard Close 46 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 8 5 Test Time Trend Select the Test time trend option from the pull down menu The Cumulative time vs Failure intensity trend is used to determine the total amount of testing time required to find 100 of all defects Generally you can count on this estimate being much larger then
28. er s Edition demoprog mdb i Failures Per Test hours Binomial model Help Export image Copy image Print to file to clipboard Best Straight Line estimate 42 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 catastrophic and critical defects Binomial model Export image Copy image Print Help r Best Straight Line estimate bi CO 43 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 8 2 All Actuals to Estimates Select the All Actuals option from the pull down menu It is the second option This trend shows all actual time to failures versus all estimated time to failures Generally the average relative error of all models is more useful then this trend however you may want to view this trend to ensure that the model with the lowest relative error hasn t changed recently Help Export image Copy image Print to file to clipboard 44 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 8 3 Most recent actuals to estimates Select the Recent Actuals option from the pull down menu It is the third option This trend shows the last 5 computations for relative error for each of the estimation models We can use this to compare which models are the closest to the actual MTTF shown on the trend Last actual vs estimated MTTF A NHPP BB Bliom tal T Fastteout MTTF in hours estimated j A Lagarttim ie time amp Time ollie L Lojane t
29. ers Manual Copyright SoftRel LLC 2008 4 8 Defect Trends General Inputs Input failures by day Import Failure data Parameter estimation Summary results of all models Model sensitivity Compare results Test time defects Recent actuals Defect trend Test time trend Future MT TF Test time Removalrate Staffing trend Cumulative defects Test time defects Select the Select a trend pull down menu from the WhenToStop main page E Estmation results for growth model selected Select the curve fitting method Best Straight Line r Estimated MTTF failure rates Deenen Estimated Inherent Defects 877 5e 1 e E 0 Defects found so far in testing 63 end of test Estimated Current Failure Rate 6 339637e4 Bie citean oe Failures Per Hour Estimated Current MT TF 157 738e 1 Hours End of Test Failure Rate 6 339638e4 Failures Per Hour End of Test MTTF 157 738e 7 Hours Operational Failure Rate 8 830208e 5 Failures Per Hour i Operational MTTF 113 248e2 Hours L o m Other projections Estimated reliability for mission time specified and current failure rate Not Available Estimated reliability for mission time specified and EOT failure rate Not Available Estimated reliability for mission time specified and operational failure rate 9 992938e 1 Estimated availability for current MTTF and MTS WA specified in prediction 0 722365 Estimated availability for end of test
30. etely unacceptable Number of catastrophic defects You will need to determine what critical means to your project This classification is generally used for the serious defects that are show stoppers but do not have a catastrophic impact on either your company or your end users WhenToStop Software Users Manual Copyright SoftRel LLC 2008 Number of moderate defects You will need to determine what moderate means to your project This classification is generally used for defects that have a workaround Number of negligible defects You will need to determine what negligible means to your project This classification is generally used for the defects that are not very visible to your end users Record number The record number is shown on the left hand side of this dialog It is read only and is used to help you toggle through the records in your database 4 2 1 1 Buttons Print This button prints out the contents of the current failure record It is not enabled in the demonstration version There are several functions available for finding particular records in the failure database Next This button will display the next record in the database Previous This button will display the previous record in the database Top This button will display the first record in the database Bottom This button will display the last record in the database Bypass the System Calculations If you have a lot of records in your databa
31. he General Inputs page It uses whatever option you have chosen calendar CPU or operational time to accumulate the time before failure Weighted Least Squares Estimate The weighted LSE is another method for estimating inherent defects and initial failure rate It is similar to the LSE except that the most recent points are weighted more heavily then earlier data points X values cumulative detected defects cumulative test time for each time interval in which a defect was detected Y values cumulative detected defects for that time interval 55 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 56
32. hown in section 4 2 1 These are called csv files and are exported by common office applications such as spreadsheets You can output your failure logs in the csv format and import them into Frestimate with this function There is an example csv file shipped with Frestimate which you can use for reference If your file is not formatted properly Frestimate will inform you of this status Select the Import Failure Data button from the WhenToStop main page shown in section 4 0 The below dialog is displayed Import Failure Data 5 c HP_PA ILION EJEA E3 Program Files compimp cs Example csv SUME CEW 13 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 Import Failure Data Pee w 321frestimate example csv Import Cancel If you do not have the last 3 fields in the csv file they will be ignored However you must have the first 3 fields in the exact format as shown here Valid date Hours Catastrophic defects Critical defects Moderate defects Negligible defects Valid date This can be in any date format It is the date in which either testing was performed with no defects found or testing was performed and defects were found Hours The number of hours of testing performed on this date Include hours of testing by all testers and users This can exceed 24 hours This field can be a floating or an integer value Catastrophic defects Critical defects Moderate defects and negl
33. icted defects Average repair time Number of defects detected so far Number of defects corrected so far j Numbed of estimated defects total 256 612 Average testing time per week so far 12 549 Total time hours required to find all defects 2716 667 Time hours spent so far 640 Time hours needed to find the remaining defects 2076 667 Time weeks needed to find the remaining defects 165 484 Estimated number of defects to be fixed 249 612 Total corective action time needed hours 1996 893 Export image Copy image to Print Close to file clipboard workbours 100 150 Time weeks reeded D Md remaliiig d Ect All severity types Binomial model Best Straight Line estimate 51 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 8 9 Cumulatives Trend This trend shows this cumulative defects Vs cumulative time plot Typically you will see the defects detected curve level off over time when your product s testing is nearing completion This trend should not be used to make any quantitative decisions regarding When To Stop testing as it is subjective Rate vs Cumulative Defects umulative time nours vs cumulative defects Cama lathe tme li hours Esport image Copy image Print to file to clipboard 52 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 8 10 Testing needed to reach some objective Select the Test time removal rate option from the pull down menu This t
34. igible defects If you know the severity of the defects found on this day then you can enter them according Otherwise the catastrophic defects should represent the total number of defects found These values are expected to be integers Short Example 4 29 02 0 00 00 8 00 1 0 0 0 4 30 02 0 00 00 8 00 0 1 0 0 5 1 02 0 00 00 8 00 0 0 1 0 5 2 02 0 00 00 8 00 0 0 0 0 5 3 02 0 00 00 8 00 1 0 0 0 5 4 02 0 00 00 8 00 0 0 0 0 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 Once the data is imported it is merged with whatever data you have entered via the record failures per day You can use both of these options in any order at any time Duplicate entries are merged such that the defects from duplicate dates are added together 15 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 3 Parameters estimated Select the Parameters Estimation button from the WhenToStop main page shown in section 4 0 The below dialog is displayed Parameters are calculated by Frestimate directly from the defect data that you enter Every time you enter or change data these parameters can and will change also If you have a lot of data these parameters can take a while to calculate Parameters are used by the models to calculate failure rate and MTTF Parameters estimated n number defects detected so far 59 Cumulative test time 640 Growth rate Q0 7 47738875130038 N estimated inherent defects Nominal 256 611671
35. ime The reliability formulas for the models that are not exponential are shown in section 4 6 Reliability is calculated based on e The model you selected The curve fitting you selected The failure data that you entered The start date you entered general inputs The mission time that you entered in the general inputs 4 5 3 6 Test hours needed to reach objective MTTF 26 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 The current MTTF is extrapolated to the objective MTTF and the system determines the amount of test time needed for that extrapolation It is calculated based on The model you selected The curve fitting you selected The failure data that you entered The start date you entered in the General Inputs page The objective MTTF you entered in the General Inputs page Additional test hours are denoted as At in this manual Objective MTTF or failure values are denoted with an obj Subscript Model Formula to determine additional testing hours needed Binomial and At No Ao IN Ap Aotj exponential models NHPP model At 1 b IN Ap Acbj Logarithmic At 1 60 1 A 1 p models Weibull model Solve for At by optimizing this formula Aobj a b t b exp t At b 27 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 5 3 7 Defects to discover to meet objective MTTF The current MTTF is extrapolated to the objective MTTF and
36. ing with the lowest relative error NHPP Model Inputs Estmated Fallire Rate Uslig Mock lb Falleres per Hour Cumutstie Time Export image Copy image Update calculated values Close to file to clipboard Relative error is computed by the absolute value of Estimated MTTF using this model actual time to failure actual time to failure The relative error is computed for each MTTF computed at each defect occurrence as recording in the record failures per day database Estimated MTTF for each model is also computed based on the curve fitting and model selected in the Estimation gt Detailed results page The relative errors are then averaged for each model and displayed on this plot You can print this plot or close it 39 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 Each of the compare estimate plots has the ability for e Export e Copy to Clipboard e Print Export This will export the graph to a predefined file name This file name will be displayed to you and it will be saved in the same folder that Frestimate is running in You can import this exported graph into various office applications Copy to Clipboard This works just like the export function except that the graph is saved in the clipboard You can open an office application and press paste and the image will be pasted from the clipboard Print This will print the entire screen to your printer 40 WhenToStop Software Us
37. mber of detected defects n Inherent Defects No This model assumes that all faults are equal in probability of being detected and severity and that when a fault is detected it is immediately removed corrected Estimated failure rate Ao 1 n No Estimated MTTF 1 failure rate since this is an exponential model Note that failure rate and MTTF for this model change only when there is a defect detected and not directly with calendar time 36 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 6 7 Bayesian model FREstimate Bayesian Model Inputs Cumulative time 640 n number defects detected so far 59 Predicted MTTF in hours Predicted Failure Rate Nominal 10 8474576 Nominal 99921975 Failures Per Hour Reset Values Print Close Calculate Values The inputs for this model are e Number of detected defects n e Cumulative test usage time t This model is useful for measuring software projects that are not failing often The failure rate estimate is a function of time therefore if there is a period of time in which there is testing but no failures found this model will reflect it This model makes no assumptions about inherent defects or initial failure rate The formula for this model is Estimated failure rate n 1 t Estimated MTTF t n 1 37 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 6 8 Weibull Model Weibull
38. mber of moderate defects found today fo during this day Number of negligible defects found today lo Next Record First record Last record Print Close Bypass the system calculations Append Delete 4 5 4 aes ae pim Pa The following information is input for each day in the testing cycle Date Enter the date in which either testing was done and no failures occurred or testing was done and failures occurred If you do not record data for a particular date then WhenToStop will presume that no testing was done on that date Time This is the amount of time spent testing that day Do not include development time or other time that did not result in operation of the software Decide whether to use calendar time CPU time or operational time in advance and be consistent in defining time from one record to the next The default value for this field is 8 hours If you leave zero in the time field the system will assume that no testing transpired on this day and will ignore any information in the following fields For each of the following fields these are the default minimum and valid values Default values 0 Minimum values 0 Valid values Positive integers Number of catastrophic defects You will need to determine what catastrophic means to your project This classification is generally used for the show stopper defects that make the software unusable or have some effect on the end user which is compl
39. n time specified and current failure rate 4 113454e 1 Select atrend Estimated reliability for mission time specified and EOT failure rate 4 113454e 1 Estimated reliability for mission time specified and operational failure rate 9 752773e 1 Reports Estimated availability for current MTTF and MTSWA specified in prediction 0 8182766 Estimated availability for end of test MTTF 0 8182766 Estimated availability for operational MT TF 0 9937806 MTSWR 2 Objective delivered MT TF 1000 Test hours needed to reach objective 167 5e2 Defects to discover to meet objective 391 4e0 The failure rate and MTTFs are for these type Only catastrophic and critical Select the model that you want to see results for Binomial v Select the curve fitting method Best Straight Line Help Update results i WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 1 General inputs Select the General Inputs button from the WhenToStop main page shown in section 4 0 General Inputs Sel General Inputs Start of Testing End of Testing 4726 2002 Post Delivery Usage Number of Months in Growth Period 4 Number of hours operating per month 300 Mission time period in hours over which reliability will be measured E Desired MTTF after growth period f 000 Desired MTTF at end of test 200 Analyze Save Print The start of test date affects the calculations for e Estimated inherent defects No e Estimated current failure rate
40. nt this dialog or close it The dialog shows the nominal MTTF and failure estimates It also shows the 5 and 95 confidence limits on these values The confidence bounds are determined by e Establish confidence 95 e Using normal charts determine Z 95 2 e Lower interval estimated parameter Z 95 2 v Variance estimated parameter e Upper interval estimated parameter Z 95 2 v Variance estimated parameter e Use the lower and upper interval estimates in the failure rate and MTTF formulas to determine a 5 and 95 bound The formulas used to compute the above values are shown in section 4 6 of this manual 21 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 5 Detailed results The detailed results are shown on the WhenToStop main page If there are no results shown then that means that you need to enter failure data and or general inputs ata Parameter estimation Reports LS Estmation results for General Inputs Input failures by Import Failure Summary results of all models Model sensitivity LRP Compare results Select atrend Estimated MTTF failure rates Estimated Inherent Defects Defects found so far in testing Estimated Current Failure Rate Estimated Current MTTF End of Test Failure Rate End of Test MTTF Operational Failure Rate Operational MTTF Other projections 513 911e0 119 1 110403e 1 900 574e 2 1 110403e 1 900 5
41. nts are not cumulative They Cose represent the number of hours and defects for just the date oh shown in their row WhenToStop Software Users Manual Copyright SoftRel LLC 2008 Cumulative defect data Failure intensity 0 125 8 333334E 02 0 125 0 1458333 0 140625 0 1375 0 1363636 9 210526 02 0 14375 0 1428571 0 1302083 0 15 0 1538462 0 1574074 0 1607143 0 1724138 0 1708333 0 1414474 0 127907 0 125 0 1105769 0 1052632 0 1056034 0 1008065 9 960938E 02 0 1019231 0 1014493 Orr non Pa Cumulatives Table This table shows the cumulative defect data cumulative testing time and cumulative failure intensity for each point in time in which any of these values changed The failure intensity is the cumulative defects divided by the cumulative time When you select the Inherent defects trend the failure intensity shown below is plotted on the x axis and the cumulative defects shown below is plotted on the y axis If you have selected the global preferences option to filter only the serious defects then the serious defects column is plotted on the y axis When you select the inherent time trend the failure intensity shown below is plotted on the x axis and the cumulative test time shown below is plotted on the y axis 12 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 2 2 Importing Data You have the option of importing comma separated text files that contain the same data as s
42. r Hour m Predicted MTTF in hours Nominal 102 483e 1 Upper Bound 986 495e 2 Lower Bound 106 796e 1 Predicted Failure Rate Nominal 9 758e 2 Lower Bound f o 4e 1 Upper Bound 9 364e 2 Failures Per Hour Reset Yalues Print Close Calculate Values The inputs for the logarithmic model are e Observed initial failure rate Ao e Cumulative test usage time t e Theta 0 The logarithmic model assumes that some faults are likely to be detected before others This model is typically used earlier in test usage than other models It assumes that inherent defects is infinite and therefore does not model this parameter Instead it models the initial failure rate and the change in that failure rate theta Estimated failure rate observed o observed Ao t 1 34 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 6 5 Logarithmic fault count Model Logarithmic fault count model inputs Decay parameter 00525 Cumulative defects detected 59 Estimated initial failure rate Nominal K 45174 Lower Bound 1 53729 Upper Bound K 36619 Failures Per Hour Predicted MTTF in hours Nominal 938 931e 2 Upper Bound 886 68e 2 Lower Bound 997 727e 2 Predicted Failure Rate Nominal 1 06561 Lower Bound 1 128e 1 Upper Bound 1 002e 1 Failures Per Hour Reset Values Print Close Calculate Values The inputs for the logarithmic fault count model are e
43. rend allows you to see how many more testing hours will be needed at the current rate to meet the objectives that you defined in the General Inputs page of the WhenToStop module t additional testing hours No Ao IN Ao Aobj Where No Defects predicted for start of testing Aopj Objective failure rate 1 objective MTTF o failure predicted at start of testing t cumulative testing hours so far Testing time needed to reach an objective MI TF m Inputs Objective end of test MT TF 200 Objective operational MT TF 1000 Inherent defects predicted at start of testing 256 611671857628 Testing growth rate Q0 7 477388751 30038 Initial Failure rate 0 1451 73622421504 Testing hours exhausted so far 640 Additional test hours needed to reach objective end of test MT TF 5314 211 Additional test hours needed to reach operational MTTF objective 8159 085 coe r 53 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 9 Addendum Best Line Estimate The inherent defects and initial failure rate can be estimated by plotting defect rate on the x axis and cumulative defects on the y axis The best straight line through these points is the best line estimate This software allows you the option of using a best line estimate for parameter estimation There is no best line estimate if the slope of this plot is positive or if there is no positive y intercept when drawing the best straight line X values cum
44. rt of test defects 413 417 410 Object Oriented 9125 m ode sin End of test defects 6 9 3 L anguage 135 Failure rate predictions Wh enToStop End of test failure rate 8 396e 3 1 259e 2 4 198e 3 Failure rate 1 853e 1 Failure rate at next release 1 817e 4 2 559e 4 7 754e 5 predictions are in 4 01e 3 Average failure rate during release 3 192e 4 4 788e 4 1 596e 4 terms of failures 7 045e 3 MTTF predictions per Hour End of test MTTF 119 108 79 405 238 216 5 397 3 598 10 794 MTTF at next release 5503 713 3907 625 12896 875 MTTF MTBI 249 387 177 064 584 39 Average MTTF during release 3132 708 2088 472 6265 417 predictions are 141 951 94 634 283 902 MTBI predictions in terms of End of test MTBI 23 822 15 881 47 643 hours 1 079 72 2 159 MTBI at next release 1100 743 781 525 2579 375 49 877 35 413 116 878 Average MTBI during release 626 542 417 694 1253 083 28 39 18 927 56 78 Reliability predictions End of test reliability 93 50401e 2 90 41602e 2 96 69747e 2 Reliability at next release 99 85475e 2 99 79548e 2 99 939799e 2 Average reliability during release 99 74496e 2 99 61 768e 2 99 8724e 2 Availability predictions End of test availability 95 15659e 2 92 90667e 2 97 91819e 2 Availability at next release 99 88997e 2 99 84509e 2 99 95301e 2 Average availability during release 99 80685e 2 99 71055e 2 99 90333e 2 C Program Files Frestimate Manager s Edition demoprog mdb WhentToStop differs from the other modules in Frestimate because it does estimation as
45. se it may take a while for the system to update the calculations every time you enter date You can bypass the system calculations while you are entering data and then update them later by pressing the Estimation gt Recalculate menu item Append This will add a new record to your database The date will default to the latest date from your database Delete This will delete the record which is currently shown Close To close the Failure database and update all system parameters calculated by WhenToStop press the Close button at the bottom of the window If there is a lot of data in your database this can be a time consuming process Therefore there is another option for closing This option is Bypass the system calculations gt It will close the database and save it but will not update the system calculations You can later update the system with the close button 10 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 Table View This shows the defect data as a list allowing you to see all of the records at once Cumulative defect data Date Test hours Catastrophic Critical Moderate Negligible mm m oOo CB eI tt I SO 0 sooo oo oOo o o o Oo Oo Oo o lo o o Oo Oo Oo o o oo o o o Pe oa E oa B oa E m A ma S a A a A a E oa BE ma B man A ma A oa G ma E ma E a 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 The testing hours and defect cou
46. t e F bt 1 e E f i ti e ti4 Ptit y e Pt 7 ety Hint solve for a with first equation Find values for b that make equation 2 equal on both sides 4 3 2 5 Weibull parameters a Weibull parameter This is a slope parameter of the natural log of the defect detection times plotted on the x axis and In In 1 1 Fi where Fi is the ni N for each time interval in which a defect was detected You can see this parameter estimation in the Model Sensitivity gt Weibull gt Weibull Parameter estimation page b Weibull parameter bO the y intercept of the natural log of the defect detection times plotted on the x axis and In In 1 1 Fi where Fi is the ni N for each time interval i in which a defect was detected b exp b0 a Weibull Inherent Defects parameter When you have the Weibull model selected in the results page the below value will be displayed as the Inherent defects WhenToStop Software Users Manual Copyright SoftRel LLC 2008 No exp Weibull k parameter K b No Weibull curve fitting Wen CLiCI F DD Hitt Export image Copy image Print to file to clipboard 4 3 3 Bypassing estimates for inherent defects initial failure rate If you have historical data from a previous and similar software project you may choose to override the system calculated parameters This option is only for advanced users who have collected historical data To choose this option
47. t to clear your inputs and reset the values to those that were computed by WhenToStop then press this button Print This will print the form Close This will close the form but will NOT save any of your inputs The model dialog screens allow you to perform analyses by modifying the input parameters to the models They are not intended however to overwrite the system parameters computed by WhenToStop Calculate values If you change the default values in these dialogs the calculate button will recalculate the failure rate and MTTF values for you based on the new inputs It will not however overwrite the system parameters calculated by WhenToStop When you exit this dialog the changes will not be saved 30 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 6 1 NHPP Model NHPP Model Inputs a intercept parameter 392 41 3619 6 slope parameter 000269 See ee tiie feso Predicted Failure A ate Predicted MTTF in hours l Nominal 9 686e 2 penni f 12 533 e Failures Per Hour Reset Values Print Close Calculate Values The inputs for the NHPP model are e aparameter e b parameter e Test usage time The outputs are e Predicted MTTF in hours e Predicted failure rate in terms of the unit of time that you specified in the global preferences dialog Estimated failure rate a b exp bt Estimated MTTF 1 failure rate since this is an exponential model Note that
48. the time to find 95 of all of the defects The y intercept is determined by the curve fitting method that you selected in the Estimation gt Detailed Results page You can select from one of these 4 methods Best Straight Line Least Squares Estimate Weighted Least Squares Estimated Maximum Likelihood Estimate Rate vs Time Cumulative time 0 05 0 10 Defect rate The curvefitting method that you have selected is Best Straight Line The estimated total time to find all defects ts 2716 66 749215613 Export image Copy image Print cose to file to clipboard Dee 47 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 4 8 6 What will MTTF be after this many more testing hours Select the Future MTTF option from the pull down menu This trend shows what the MTTF will be if there are some additional test hours t additional testing hours No o IN Ao obj Where No Defects predicted for start of testing obj Objective failure rate 1 objective MTTF Ao failure predicted at start of testing t cumulative testing hours so far Therefore t additional testing hours Ao No IN Ao opj exp t additional testing hours Ao No Ao Aopj exp t additional testing hours A9 No Ao 1 j MTT Fopj MTTF after this many testing hours Testing growth rate Q0 TAr 308751 30038 Inherent defects predicted at start of testing 256 611671857628 Initial failure rate pre
49. tial failure rate Ao estimate when plotting the cumulative defects vs the failure intensity Logarithmic models on the other hand presume that this plot is not a straight line but rather a logarithmic curve Since slope cannot be used in this case the rate of change of the curve is used instead Theta 6 is the rate of change between the initial failure rate and the current failure rate This is also called the failure rate decay If you plot the natural log of failure intensity versus cumulative defects the slope of this plot is theta 4 3 2 3 Binomial model parameters 18 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 k parameter This parameter is calculated by WHENTOSTOP as the inverse of the slope of the line of cumulative defects vs failure intensity trend The k parameter is only used by the Binomial model 4 3 2 4 NHPP parameters The NHPP a parameter is estimated to be approximately equal to number of inherent defects Noin the software at the start of system test The NHPP model assumes that inherent defects are finite but not fixed The NHPP b parameter is estimated to be approximately equal to the initial failure rate Ao of the software at the start of system test The NHPP model assumes that inherent defects No are finite but not fixed Therefore initial failure rate is also assumed to be finite but not fixed Simultaneous equations are used to determine the NHPP a parameter EQ 1 a Z i EQ 2 a
50. ulative detected defects cumulative test time for each time interval in which a defect was detected Y values detected defects for that time interval Number of detected defects This is exactly as the name implies WHENTOSTOP calculates this default value as being equal to the number of defects in the fault log for this project If you are using the failure recording method then this will be equal to the number of records in that database If you are using the time of failure method to record failures then this will be the sum of all of the defect counts in every severity class Least Squares Estimate The least squared estimate is another option for estimating inherent defects and initial failure rate X values cumulative detected defects cumulative test time for each time interval in which a defect was detected Y values cumulative detected defects for that time interval Maximum Likelihood Estimate The Maximum Likelihood estimate is another method for estimating inherent defects and initial failure rate Operational time this is the amount of staff hours spent solely in the activity of testing or using the software If you choose this option do not count development time that is not spent directly testing or using the software Test usage time 54 WhenToStop Software Users Manual Copyright SoftRel LLC 2008 This is the calculated cumulative test usage time WHENTOSTOP calculates this based on the system start date in t
Download Pdf Manuals
Related Search