Help: Problem Solving and Troubleshooting
Contents
1. How do we get in this mess Understanding the precursors 1 Pressure Management time resourcing Rationale and the ability to reason often disappear under pressure Your focus is set on fix rather than solution There may be few incentives to step back and think before doing auc 2 How do we get in this mess Understanding the precursors 2 Limited Familiarity The technology is unknown to you or you have only basic knowledge You ve inherited a system and it s broken You re new to a role or organisation auc 2 How do we get in this mess Understanding the precursors 3 Overconfident Massive underestimation of the problem how hard can it be Nu XWII How do we get in this mess Understanding the precursors 4 Quick Fix Temptation It s tempting Quick Fix Now probably a really big It s delicious problem later You ll regret it later auc 2 Problem Solving Methods Stage 1 Problem Definition 1 Determine if there is actually a problem Gather information Understand the situation Establish a baseline where the problem is a variation on normal ie capacity performance problem Verify the problem exists XWII Stage 1 Problem Definition 2 clearly define what the problem is Scope Impact Nature ae XWII Stage 1 Problem Definition 3 what ar2. World 2011 Help Problem Solving and Troubleshooting Daniel Rodwell Australian National University auc XWII Intro Outline Today s Session Two Parts Problem Solving Concepts and Theory Methods Group Solve Troubleshooting Concepts Methods e XWII Today What s in it Professional Development workshop Toolset for you to use Lighthearted not too serious Mixture of Skills and Backgrounds hopefully theres something here for everyone auc 2 Part 1 Problem Solving Problem Solving Concepts Problem Solving The dictionary says problem noun 1 a matter or situation regarded as unwelcome or harmful and needing to be dealt with and overcome mental health problems as ad city planners consider it a problem district e a thing that is difficult to achieve or accomplish motwat on of staff can also be a problem ORIGIN late Middle English originally denoting a riddle or a question for academic discussion from Old French probleme via Latin from Greek problema trom proballein put forth from pro before ballein to throw Les auc 2 Problem Solving The thesaurus says problem 1 they ran into a problem difficulty trouble worry complication difficult situation snag hitch drawback stumbling block obstacle hurdle hiccup setback catch predicament plight misfortune misha
3. experimentation and discovery Output Evolutionary goal The best solution progressive auc 2 Bake off XWI I Funnel Method Lots of Ideas 6 TA Concept generation Modular Grouping AUC Group Solve Group Solve Solve for X Likely to encounter this scenario in your organisation Problems progressively revealed as you traverse the scenario individually pair up amp think of the problem and how you might start to solve it modules categories attributes auc 2 Scenario lt scenario removed gt Pe xwii Why Problem Solving Hurts Ouch If it was easy you d have solved it already It typically involves learning new stuff while simultaneously developing a solution Chances are you will not immediately know the answer You re under pressure AUC 2 Constraints Fixed vs imposed Constraints Some constraints will be fixed and are physically determined ie Cable breaking strain of 1200KG Other constraints are imposed or we unintentionally limit ourselves with prior convention Think outside of the problem as well is the problem part of a bigger picture Les auc 2 Consider this Tan Constraint Greenland POCCMA United Kingdom Polska Deutschland y T ykpaiHa 3 liraina Kazakhstan France f Mongolia italia United Espana T rkiye North Pacific S
4. Why Multiple wrong inputs Situation becomes progressively worse progress is unknown Each Troubleshooting stage should result in usable information Even if that is this part works as expected You now have one less module to isolate Les auc 2 Troubleshooting Methodologies Gather info and verify First Steps Gather info Verify situation against information Establish a baseline of a correctly operating system Rule out really obvious factors Storage full No IP address No AC input etc XWII Brute Force Guesswork Troubleshooting Methodologies Brute force Guesswork Belief based Evidence poor Procedurally inadequate variable highly uncertain if correct cause identified uncertain occasionally works for some experienced state techs Common cause of it must be this part b XWII AUC Brute Force Guesswork Methodology VT WI variable certain uncertain state i Nu XWII Split Half Troubleshooting Methodologies Split Half Eliminate half of the probable cause at each System VA level Software Requires understanding of common issues LX Requires understanding of core functions of Ba Function each function area or differentiating VAR isolation behaviour GRU Display highly structured complete but can be time consuming and indirect if starting point is vague Works
5. best for isolate verify function areas where there is no obvious likely cause auc 2 Split Half Methodology m Hardware x Function isolation Bol auc XWI I Power Signal Flow Troubleshooting Methodologies Power Signal Flow m signal Follow Signal sequence through system flow Highly sequential must be performed in order effective for no X or dead symptoms mis LI often places core modules early in the troubleshooting even if they may be a less likely cause Requires understanding of signal flow in system architecture Speaker Audio auc XWII XWI I signal flow n MLB I SMC N m Power Signal Flow Methodology Likely Cause Troubleshooting Methodologies Likelihood decreasing Likely Cause Identification Use known likely causes as starting point can often be reordered to promote more likely causes demote less likely cause works best where it is possible to identify all sources of possible cause there are few causes or the causes are well known less suitable for cases where there is no obvious cause Pe XWII Likely Cause Methodology Likelihood decreasing XWI I AUC Likely Cause Weighted Matrix Troubleshooting Methodologies Weighted Matrix Use to assist prioritising the Likely Cause isolation order
6. different requirements completeness individually solved collectively solved can be delegated or allocated A Piece or Chunk is likely to be an activity or task attribute or category auc 2 Top Down Method Tightly Defined Problem Top Down Analysis Start at highest level of system partial understanding of sub technologies You know what you want from a solution maybe not at module or piece level E XWII Analysis XWII Bottom Method Tightly Defined Problem Bottom Up Synthesis Start at lowest level of system Individual modules collectively build the system or solution You understand what is happening at module level unsure on individual relationship to whole AUC 2 Bottom Up gt m s a get Synthesis Finding the Pieces Order in chaos Ways pieces of the problem become obvious things to look for Natural Grouping Functional or Procedural Grouping Modular Derived from First Principles or Architecture auc 2 Funnel Method Loosely Defined Problem Recall Broad non specific goals Ideal based Experimental Trial Future Projects The problem may not be fully understood and solution options are completely unknown Au XWII Funnel Method Loosely Defined Problem Inputs new or unproven Ideas parallel prototyping project bake off
7. ok auc XWI I No Single Answer Select a method No single method works for all types of symptoms or fault complexity simple tightly correlated symptoms complex loosely correlated symptoms nature of failure electrical mechanical runtime configuration design capacity Intermittent a auc xwii Known Good Troubleshooting Methodologies Known Good modules are modules code or some other component that is known to be operating correctly Its often called KG or golden For core components you may need to use a KG module OR have a good understanding of the expected behaviour of the core modules but they really need to be good or golden or you ll prime your troubleshooting for failure AUC 2 Tools To Help You They re often right there Console logs would you believe have heaps of info Activity Monitor top amp ps fs_usage amp lostat SC_usage amp dtrace netstat wireshark rubbish webmin interface on your switch fabric CSS FC array AUC 2 Group Troubleshoot Group Troubleshoot Scenario Less likely to encounter this situation in your organisation You might not know all of the technology involved Use first principle knowledge of IT systems to identify modules individually pair up amp think of the problem and how you might start to solve it modules cat
8. to solve any difficulties you see Why gaps or gotchas will be exposed it might sound good in your head but verbalising it exposes the weaknesses auc 2 Stage 3 Peer Check Possibly the most powerful resource What if l m working alone Write it down Blog it Tweet it Even if no one reads it you have a record of your thoughts Gives you a point of return if you get lost Talk to your manager auc 2 Stage 4 Nature of Problem The nature of the problem will guide you toward a methodology Loosely Defined Problem Broad non specific goals Ideal based Experimental Trial Future Projects AUC 2 Stage 4 Nature of Problem The nature of the problem will guide you toward a methodology Tightly Defined Problem Specific goals Target based Production ready workflow style systems AUC 2 Problem understood Now how to solve it We have a big lump of a problem PROBLEM nuc XWII Problem understood Now how to solve it We could chip away at it and may get somewhere if we re lucky PROSLEM AUC 2 To effectively solve any problem Break it up ber Nu XWII Break it up Problem AUC XWI I Stage 5 Break it up A big problem is hard to solve Smaller chunks are easier to solve a piece or chunk is far more workable each piece may have specific but
9. Possible Cause A HIS D ndencies l HIGH Promotes more likely relevant isolation Possible Cause s Yes High A Dependencies tests for the scenario Possible Cause C Low Low LOW Demotes less likely causes Use to correctly weight troubleshooting priority auc 2 Likely Cause Weighted Matrix Methodology Possible Cause A Possible Cause B Possible Cause C Les auc XWI I Likely Cause Weighted Matrix Methodology Possible Cause A High Yes High Dependencies Possible Cause B Low Yes High Dependencies Possible Cause C Low No Low XW AUC Likely Cause Weighted Matrix Methodology Derived Order Possible Cause A High MES High Dependencies 77 HIGH RANE Possible Cause B Low Yes High Dependencies a MID RANK KT Possible Cause C Low No Low N Low RANK auc gt QUAL Minimal Config Troubleshooting Methodologies Minimal Config The Final Frontier Core Components Saviour when all else fails Highly time consuming Next Component Next Component but high accuracy Test Module D Test ok ok Must know what components are the absolute minimum for the system start auc 2 Minimal Config Methodology System Build Up Core Components Test auc XWI Minimal Config Methodology System Build Up Core Components Test
10. auc 2 At this point You should be thinking ALERT lt SUBJECTIVE INFORMATION SOURCES n HA Kel nai XWI I Subjective cf objective Perception based typically not driven by fact or data opinion rather than scientific observation May contain traces of Emotion Nu XWII How do we react How do we react to a problem AARGH PANIC ALARM EEEEK TERROR FLUSTER HYSTERIA GRUMBLE BLAMETHROWER SPAGHETTI CHUCKER GRIPE ow COMPLAINT DISMISSAL 5 SHELL BE RIGHT THERE IS NO PROBLEM MMMM K auc 2 2 How do we react How do we react to a problem Sometimes but rarely Analytically Pragmatically AUC movie clip Understanding the Problem Don t be mislead or confused Before you do anything 1 Determine if there is an actual problem 2 clearly define what the problem is 3 and what you are trying to solve the act of solving is sometimes the easy part AUC 2 Why We want to make the situation better not how many times have you seen the opposite happen DIY anyone AUC 2 What am trying to solve Constant Re evaluation mE xwii OBVIOUSNESS ALERT This all seems like common sense But Its easy to get lured into a big mess Often you don t know you have a big problem until you have a really big problem
11. e attempting to influence It gives us the sense of getting somewhere auc 2 Feedback Concept Managers are human too Managers are human too Uninformed managers can become a larger problem than the technical issue you are trying to resolve AUC 2 Feedback Concept Keep it in mind When determining the steps you are going to take in your troubleshooting task keep in mind the result you are looking for at each step and what result a normal correctly operating module would return If you have progressive results you can keep others informed ie we re ruled X out established Y is working need to test Z auc 2 Why Feedback Matters Consider this A theoretical moving car Input gt Process gt Output Steering Angle Wheels turn Change in Direction ae Feedback Visual Recognition Sensory Feedback g force auc 2 Feedback Delayed Feedback altered A theoretical moving car Input gt Process gt Output Steering Angle Wheels turn Change in Direction a Feedback 30sec Visual Recognition Sensory Feedback g force AUC 2 Feedback Removed Feedback altered A theoretical moving car Input gt Process gt Output Steering Angle Wheels turn Change in Direction Feedback Visual metognition Sensory 01 g force AUC 2 Oh no You crashed and burned
12. e you trying to solve Outcomes Deliverables Solution le What you want to see at the end of it AUC 2 Simple Example We have No Milk 1 Determine if there is actually a problem Look in the fridge Yes there s no milk 2 Clearly define what the problem is We need milk for breakfast in the morning and we don t have any and need a a coffee before leaving the house 3 What are you trying to solve Get enough milk for breakfast nothing more nothing less auc 2 How many systems or projects have you seen that don t solve the original problem What am trying to solve Remember this AUC 2 Stage 1 Problem Definition Stage 1 is your foundation weak problem definition will lead to weak solutions Your problem definition doesn t need to be pages and pages of blurb A concise accurate problem description is better Stage 1 is knowledge and familiarity building Knowledge Familiarity less stress AUC 2 Stage 2 Research Understanding What has been done so far The factors that have lead to this situation Research You might not be the first to encounter this problem Your research may lead you back to Stage 1 again XWII Stage 3 Peer Check Possibly the most powerful resource Describe the problem to a peer or colleague Clearly articulate what the problem is What youre trying
13. egories attributes MuR XWII Group Troubleshoot Scenario lt scenario removed gt mE xwii Workarounds Where it s not something you can fix Occasionally there will some some issues you have isolated to a cause that you cannot directly fix For Example a software bug Using your troubleshooting results you ll know where it s failing Use this information to develop a workaround until a permanent fix is available Report the bug to the product vendor or manufacturer When the fix is available you ll know how to correctly verify its operation auc 2 World 2011
14. here s something wrong with my computer cant download the internet AUC 2 A System and its parts Any System is a collection of modules It s normally a module that breaks not the entire system A web server is a system I O network authentication db content config A washing machine is a system pump motor controller valves sensor auc 2 Accurate Troubleshooting Report of System Failure Verification or Replication of fault where there is an actual verifiable fault locate the faulty module within system Fix only the faulty module or part Return Correctly functioning system to operational status auc 2 Progress through the troubleshooting process should reduce the uncertainty progressively isolate the modules increase the number of known states 2 What is Troubleshooting Sequential Fact Building AUC Fact Building Symptom Gathering Administrator asks probing questions User reports of problems and Priming Data description Normal Statistics 7 Log Files Error Reports Symptoms Symptom Verification Bogus Isolation Uncertainty Facts decreasing Increasing Module identification Solution Cause si XWI I AUC Feedback Concept We like to know whats going on Humans like feedback in the form of progress We like to know that our interactions are changing the environment we ar
15. p misadventure dilemma quandary informal headache nightmare 2 I don t want to be a problem nuisance bother pest irritant thorn in one s side flesh vexation informal drag pain pain in the neck 3 mathematical problems puzzle question poser enigma riddle conundrum informal teaser brainteaser adjective a problem child troublesome difficult unmanageable unruly disobedient uncontrollable recalcitrant delinquent ANTONYMS well behaved manageable AUC 2 Problem Solving The dictionary says solve s lv s lv verb trans find an answer to explanation for or means of effectively dealing with a problem or mystery the policy could solve the town s housing crisis a murder investigation that has never been solved auc 2 Problem Solving In Context For System Administrators or System Engineers design a new system grow an existing system transition to another system codify a process or activity solve an IT need auc 2 But Problem Solving Skills are reusable Core Skills can be applied generally to solve non IT problems anywhere design a building organise a world wide roadshow fix something D XWII How do we know How do we know we have a problem Two ways we typically discover a problem SENSE TOLD we sense something is someone tells us we have a different from normal problem
16. rmally behaves Les auc 2 Part 2 Troubleshooting Troubleshooting Concepts What is Troubleshooting Dictionary says troubleshoot trobol sh oot verb intrans usu as n troubleshooting solve serious problems for a company or other organization trace and correct faults in a mechanical or electronic system ae XWII What is troubleshooting Applied Problem Solving auc 2 Inherit Problem Solving methods It s reusable Core points retained Define what the issue is Understand what you are trying to fix Break the issue down into smaller parts AUC 2 Types of Failure 3 Common Types Technical Failures usually fall into three top level categories Bogus there is no failure Outright it s dead Intermittent the most problematic auc 2 Influences Influences on Troubleshooting accuracy Quality of Symptom description Symptoms often do not have a 1 1 correlation with failure mode Data may be incorrect auc 2 How not to fall The most important part Symptom Description An accurate and concise Symptom Description is critical to your troubleshooting success Without an accurate Symptom Description You ll be chasing the wrong thing be unclear where to start auc 2 Symptom Description It s easy to spot a bad one It s dead It doesn t work T
17. tates North rkiy B E Ocean ent iran ran Afghanistar China Oc cean Algeria Libya a Pakistan Arabia Mal Niger Chad wer Suda Nina Venezuela gers Ethiopia Colombia EA Kenya DR Congo ndonesia Papua New i Tanzania puaa Brasil Guine Angola Bolivia p Namibia Madagascar indian Botswana Ocean NT South South Australia Pacific Atlantic Ocean Ocean Argentina 2 2HI2 d YHON 92 IIe 909 0007 DOKN 6 pay un epeuesy eunueiiy ueso SUEY unos PIANO f seg equi Pjanzauay 99 20 union A PUEJU3319 Is on top BUEMS JOg TAWEN ejobuy PIUBZUR i o6u05 ud PAUTA 0 10 03 euad n png POS peys 13 ev 1 PNES el engl 36 DEJ 3101 N eueds3 18 392011914 e pue1u sinadg EXSIOd wopdu payun Consider this Down under amp NZ too eyegsny 990 31 M N EnOEqd EISPUORU puejeui e puj UEISIMEd 4 65 9 6 t E 5M220gd putita M N O MON No It s all Why N Someone decided North goes at the top auc 2 No Problems I m awesome No problems here yet Discover weaknesses in your systems use same approaches module by module analysis understand what normal is for your system Understand utilisation and capacity If you do have a problem you ll know how each module no
Download Pdf Manuals
Related Search