Developing and using program logic
Contents
1. DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT user guide DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT user guide PRINTED ON 100 RECYCLED PAPER Commonwealth of Australia 2009 Selected passages may be reproduced provided due acknowledgment is made The Australian Government acting through the Department of the Environment Water Heritage and the Arts and the Department of Agriculture Fisheries and Forestry has exercised due care and skill in the preparation and compilation of the information set out in this publication Notwithstanding the Australian Government its employees and advisers disclaim all liability including liability for negligence for any loss damage injury expense or cost incurred by any person as a result of accessing using or relying upon any of the information in this publication to the maximum extent permitted by the law Folder printed on 100 recycled plastic Pages printed on 100 recycled paper This guide was funded by the Australian Government s Caring for Our Country initiative Acknowledgments This guide was written by Dr Alice Roughley of the Australian Government Land and Coasts AGLC The assistance of colleagues in Australian Government Land and Coasts Jo Roberts James Austen and Lee Drummond colleagues of the National LandCare Facilitator Project Coral Love and Paul Carroll and Helen Watts of2. for program performance atool to inform learning and adaptation of strategies to improve program performance a communication tool particularly for complex programs to inform partners the community and investors The process of developing a program logic facilitates thinking planning and communicating about program objectives and actual accomplishments Program logic can enhance planning design implementation analysis and knowledge generation because it sets out a clear statement of anticipated outcomes the assumptions underlying those outcomes and how success will be measured It enables all those involved in a project to work from the same roadmap and timetable which helps to maintain a focus on the big picture as well as the component parts Because it is particularly suitable for visual depictions program logic modelling communicates well with diverse audiences from varying backgrounds DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT Program logic expresses how change is expected to occur The process of developing a program logic facilitates thinking planning and communicating about program objectives and actual outcomes Program logic sits within the broader context of monitoring evaluation reporting and improvement MERI It is therefore important to understand the principles that underpin the MERI Framework before attempting to develop a program logic Th
3. 0 E I a E lt YN cc IL cc TIP CAUTION KEY CONCEPT 36 It is important to be mindful of the issues that can either restrict capacity to answer the guestions or provide opportunities to minimise the extent of reporting needed Generally if the guestions are constructed to satisfy all key stakeholders they will also be able to address the key evaluation categories Stakeholder analysis As each evaluation guestion is devised it is useful to think about who will want the information the guestion generates All key program stakeholders should be consulted about their needs and where possible represented at this part of the workshop When the program logic 15 used as the basis for evaluating the program the stakeholders will be interested in a variety of information The funding body will want to know ifthe money was used as intended that was spent efficiently and that there have been positive outcomes Land managers might want to know how many sites the program covers and how much vegetation has been restored The program delivery agents could be interested in the appropriateness of their investment strategies did they align with the needs or means of the targeted stakeholders No evaluation can answer all of the questions stakeholders may ask so it is critical to prioritise using some criteria for refining and narrowing the questions Involving stakeh
4. 3 Sample evaluation guestions worksheet Evaluation guestions Aspirational To what extent has the project contributed to an increase in the extent program goal distribution and quality of native vegetation Longer term what ways has the quality of the remnant vegetation improved outcomes What is the extent of increase in native vegetation What proportion of the priority vegetation on farms is now protected What increase has there been in the number of land managers applying biodiversity conservation skills Intermediate What is the change in extent of remnant vegetation on targeted sites How outcomes many hectares are under agreements protected and enhanced To what extent have the threats been removed To what extent are people managing their native vegetation differently and how and why To what extent and in what ways has knowledge of condition and trend improved How has the organisation s ability to target investment improved Immediate To what extent has involvement in the program changed participants activities and appreciation of biodiversity values outcomes How many land managers were able to manage weeds as per the contract What is the level of compliance with the vegetation management agreements and permanent covenant conditions How many events were held through the program to inform educate land managers How successful were the strategies to promote the program How many
5. 48 Checklist 3 Articulating and documenting assumptions Progress towards articulating and documenting assumptions Yes Not yet Comments revision Assumptions about how change will occur were discussed Assumptions about how change will occur were documented Risks that could affect the achievement of outcomes were identified and documented 49 ATTACHMENTS REFERENCES Checklist 4 Evaluation questions Progress towards a set of evaluation Yes Not Comments revisions guestions yet Key program stakeholders were consulted engaged in the workshop Information requirements were identified for each stakeholder group There are a manageable number of questions at each outcome level Questions address how critical timing is for the project Questions generally relate to extent of change impact contribution to longer term impact An evaluation category was identified for each question Questions address the required evaluation purposes The questions will provide information to meet the requirements of stakeholders Participants are confident that evaluation reports will meet the multiple requirements and needs of stakeholders 50 DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT SLNAINNHOVLLV 5 ASPIRATIONAL LONGER TERM INTERMEDIATE IMMEDIATE
6. IUCN www iucn org programme eval documents2 finding_way finding_ way_eng pdf IUCN International Union for Conservation of Nature 2000 Finding the way practical guide to the organisation and execution of the self evaluation processes of projects centred in sustainability AC Imbach ed cmsdata iucn org downloads finding_way_eng_1 pdf IUCN 2001 The IUCN Evaluation Policy 2001 cmsdata iucn org downloads evaluation_policy_eng pdf IUCN nd Monitoring and evaluation methods and tools www iucn org programme eval methods htm Mathison S ed 2005 Encyclopedia of evaluation Sage Publications California Millar A Simeone R amp Carnevale J 2001 Logic models systems tool for performance management Evaluation and Program Planning vol 24 pp 73 81 DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 57 58 OECD Organisation for Economic Co operation and Development 2002 Glossary of key terms in evaluation and results based management www oecd org dataoecd 29 21 2754804 pdf Schwartz R Forss K amp Marra eds forthcoming Evaluating the complex attrition contribution and beyond Transaction Publishers Comparative Policy Analysis Series www inteval group org progress htm complex Australian Standard AS NZS 4360 2004 Risk management www saiglobal com shop script Details aasp DocN AS0733759041 AT Stanley J Clouston amp Binney J 2005 Understanding social and
7. e g drought climate change flood What social or institutional factors and events could impact on the ability to carry out planned activities and achieve results e g market trends and fluctuations staff capacity engagement of land managers cultural knowledge For example improving water quality might be based on an assumption of continued best practice water management by land managers But the dam could silt up from poor environmental management or the water could be diverted to other users By identifying assumptions and then what could go wrong program managers and staff can accept the ones that are outside their direct control and discuss what they can do to reduce external risks DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 31 0 F TL X oc 0 E l x 5 I 0 E I E 32 TIP Go to page 49 Equally important and more neglected are assumptions about the internal cause and effect logic of the project For example a program may invest in a project to recruit volunteers to plant trees as a strategy to revegetate a degraded landscape This approach assumes that enough people with skills in planting the right tree species in the right places will join and continue to work voluntarily until there are enough trees in enough places to achieve better landscape connectivity and biodiversity corridors But will there be enough volunt
8. os a ss aora ge au NSS S ee Ber i t os FOR gt Steps to build a program logic 17 STEP 1 SCOPING 19 a a aaa dej mi E oe a Sia SORTE SEE SEE h Who will use the program logic to guide program evaluation 20 What resources are available 20 DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT STEP 2 DEVELOPING AN OUTCOMES HIERARCHY 21 Resources 21 Process 22 Constructing the hierarchy 23 Identifying evidence to support the logic 24 STEP 3 ARTICULATING AND DOCUMENTING ASSUMPTIONS 27 Uncovering the assumptions 27 Identifying risks associated with the assumptions 30 STEP 4 FORMULATING EVALUATION QUESTIONS 33 Two levels of evaluation questions 34 Stakeholder analysis 36 A FINAL WORD 39 SECTION 4 TOOLS WORKSHEETS Worksheet 1 Evidence 43 Worksheet 2 Outcomes hierarchy with assumptions documented 44 Worksheet 3 Risks 45 Worksheet 4 Evaluation questions and stakeholders 46 CHECKLISTS Checklist 1 Scoping 47 Checklist 2 Outcomes hierarchy 48 Checklist 3 Articulating and documenting assumptions 49 Checklist 4 Evaluation questions 50 ATTACHMENTS Attachment 1 51 Attachment 2 53 Attachment 3 55 REFERENCES AND ADDITIONAL READING 57 DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT Introduction
9. Land managers need knowledge about sustainable native vegetation management skills Same land managers over time Increased awareness of the program through the range of communication mechanisms The most appropriate preferred communication types and modes are used to target key program audiences Incentives for enhancement of remnant vegetation Land managers are more likely to change their practices and participate in programs if incentives are offered Field days workshops for land managers Bringing land managers together is a good way to get them engaged in NRM programs Once engaged land managers will stay engaged Compliance with contracts Land managers will comply with contracts because they want the incentive payments DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 29 0 F XL X oc 0 E l x 5 I 0 E I E Identifying risks associated with the assumptions Understanding the environment and the systems the program will operate within is critical when it comes to assessing the relevance of strategies and activities anticipating operational problems and judging a projects contribution This is how risk management is integrated into program planning from the outset An organisations control over factors in the project environment that influence the achievement of outcomes decreases with each level of the outcome
10. reached participants can reflect on the logic in the hierarchy When one pathway 15 finished the others can be progressively worked through Participants can step back and as a group look at the hierarchy to ensure that there are clear links that is there are no cards in isolation they all connect common outputs duplicated across pathways are identified the language on the cards is self explanatory and meaningful the intent is well understood Attachments 1 2 and 3 see pages 51 55 at the end of this document provide examples of three different outcomes hierarchies Attachment 1 has a social longer term outcome and Attachment 2 has an environmental longer term outcome Attachment 3 see page 55 illustrates a more complex hierarchy like the NRM program logic at Figure 2 It has an integrated set of outcomes from sub catchment planning environmental economic and social outcomes For organisations designing an outcomes hierarchy with multiple outcomes at the longer term level like that in Attachment 3 it may be necessary to work through each stream separately and then integrate them afterwards The unidentified amounts of change x in the outcome statements in the example program logics represent targets targets are the desired or expected amount of change in the asset Attachments 1 2 and 3 show links using lines and arrows Some organisations may decide not to include the arrows because of the complex na
11. ACTIVITIES FOUNDATIONAL INPUTS AND ACTIVITIES PROGRAM GOAL OUTCOMES OUTCOMES Attachment 1 Outcome Hierachy Vision for the asset Improvements in the state of the asset Aggregate changes in how the asset is managed across a geographical area Practice and attitude change Outputs biophysical Outputs non biophisical Example outcomes hierarchy for a regional environmental protection and enhancement project Definition outcome The program contributes to community stewardship of landscapes through protection and enhancement Land mangers effectively integrate land use and conservation management supported by active partnerships with government community based environmental organisations scientific and research institutions Landscape scale reconstruction of significant communities contributes to the recovery of its preEuropean extent Community members both land managers and volunteers seek continuing involvement in achieving long term landscape outcomes Increase by X of land managers with property management plans and on ground works X increase in no of volunteers involved in on ground works Increase numbers range amp diversity of X Ha increased cover of native veg Animals X Ha remnant veg protected enhanced A REFERENCES Community amp land owners have increased Volunteers participate in a wider range of
12. Adaptive Environmental Management is greatly appreciated The wealth of experience and knowledge of these people in providing thoughtful and insightful comment on drafts has served to substantially improve the readability and accessibility ofthe document Working in partnership with AGLC and through a series of Australian Government sponsored consultancies Dr Jessica Dart of Clear Horizon introduced this approach to program logic into the natural resource management arena in Australia Through this partnership the broad natural resource management community has been inspired to adopt program logic and associated participatory evaluation approaches Author Dr Alice Roughley AGLC Program Performance MERI Layout and production management Noel Martin AGLC Communications Design and print management Lindsay Davidson Whitefox Communications In the spirit of continuous improvement further feedback on the guide from users will be welcomed Contents INTRODUCTION What is program logic 7 ae 4 a oe e a aa EN E mg E A v ni i aaa K e o Host eee aa eee SEE E E nao i 5 no Preparing to construct a program logic 13 es de Sys bits Vous iG E ka 3 a nn ka vija VE S S ee 5 dus cik a E TEEL E EE ni Dd eg ER
13. The simple criteria listed in the box below provide a guide to confirm sources relevance and rigour of information A guide for selecting evidence Is the evidence relevant to the issue area that is central to the program l l Is there a reliable explanation for the evidence in the area Would this evidence only be seen under particular spatial or temporal situations 0 H a5 0 x cc 0 F x I there particular constraints to using the evidence in the area Does the expected response always occur in the presence of the activity That is is there a credible link between intervention and response Would there be another plausible explanation for this response in the area The worksheet as shown at Figure 7 can be used for listing the types of existing evidence that are available to support the logic One worksheet can be used for each outcome For further information on assessing the likelihood of selecting most important evidence and managing risks in relation to the evidence selected see AS NZS 4360 2004 DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 25 Figure 7 Evidence worksheet Go to page 43 2 EB pag Write outcome statement here Importance Likelihood of of evidence being able to 1 5 access evidence 1 unimportant 1 5 1 rare 5 essential 5 almost certain Available evidence N
14. This guide outlines a step by step process for developing program logic in the context of Natural Resource Management NRM The guide is aimed at those who are developing a program logic for the first time and may also be helpful to other users in a range of contexts Developing a program logic is about establishing a framework for methodically exploring and defining the parameters of a project Because it serves as a roadmap for a project a program logic helps to give participants greater confidence that their efforts will be comprehensive and effective In this guide the term program covers all levels of intervention whether through a project program strategy or activity as well as program design and evaluation Through a series of exercises templates and checklists this guide outlines the key steps in developing a program logic Key steps to develop a program logic scoping defining the program boundaries developing an outcomes hierarchy expectations KEY CONCEPT about change articulating and documenting assumptions theory of change formulating evaluation questions program contribution and audiences The full set of templates and checklists at the end of the guide are intended to be copied for use when users are developing a program logic DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT How to use this guide This guide explains a step by step process for developing program
15. guide in a range of situations DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT SECTION 1 ee What is program logic Program logic is an approach to program planning lt captures the rationale behind a program probing and outlining the anticipated cause and effect relationships between program activities outputs intermediate outcomes and longer term desired outcomes A program logic is usually represented as a diagram or matrix that shows a series of expected conseguences not just a seguence of events adapted from Dart 2007 and OECD 2002 Program logic expresses how change is expected to occur How the program logic is translated into operational plans will vary across programs and organisations The concept of program logic has been applied since the 1970s particularly in international aid programs Since then it has been used in many different disciplines in a variety of formats More recently it has been adapted for use in natural resource management NRM programs Why develop program logic A program logic provides atoolto guide planning at the conceptual stage of the program away to share understanding and ownership among members of a team and with stakeholders atool for clarifying and evaluating the strengths and weaknesses of a program often when it is in the development or re development phase a framework from which to develop monitoring and evaluation
16. increase in no landholders applying X net gain in veg communities on farms A 3 5 9 9 biodiversity conservation skills state of the asset geographical area 7 Aggregate changes 6 in how the asset is X increase extent of native veg Improved quality of X less other threats Enhancement of vegetation Raised level of appreciation for X increase in extent of remnants a managed across a for threatened veg communities X veg communities on farm on x farms biodiversity values amp weeds appropriately managed Lu LL cc Increased understanding of the principles X land managers involved in fencing and practices for reveg enhancement linkages fencing weed control Practice and An increase in appropriate grazing reduces attitude change threats to X native veg Increased landholder knowledge of what veg occurs on the site Outputs X voluntary 10 year X permanent covenants on areas of remnant management agreements protected or enhanced veg biophysical X incentives for enhancement Increased land holders Increased awareness of the program X field days and X compliance through local grapevine amp other X incentives for fencin of remnant extend patch amp knowledge of threats that gt 9 9 workshops for landholders with contracts programs promotion leaflets Reveg or restore liking areas could be managed media amp landcare cold calls Outputs non bio
17. overnight to reduce creases Follow the instructions on the spray can and coat the curtain as instructed The curtain will then feel tacky once dry and paper will stick to it as if it was a sticky note When folding the curtain after use try not to crease it Fold the tacky side inwards to avoid picking up dust x When attaching the curtain to a wall use masking tape or similar strong tape as re usable adhesives such as Blu Tack do not work as well for this purpose Paper can be pressed to the sheet and will normally hold for several hours depending on the size of paper etc The curtain may need periodic recoating of adhesive to refresh its tackiness Alternatively cards can be written on and arranged on a table or sticky labels can be used to position them on a wall The advantage of the magic wall is that it can be removed intact at the end of the session and taken to where the information can be transcribed Participants can also use a digital camera to take a photo of it to provide a record Process Participants can work in small groups using the coloured A5 paper a different colour for each level of the hierarchy to write their own outcome statements and place them in the appropriate levels in the outcomes hierarchy It helps to take one pathway especially for more complex programs and develop the outcomes along it by working from the bottom up to the ultimate outcome The int
18. program logic workshop Program logic development is often undertaken in a workshop format As a guide approximately one to three days should be allowed There are a number of considerations to be taken into account when planning a program logic workshop and workshops may be delivered in different ways depending on the audience the program in guestion and the working environment Suggestions for the agenda resources people and workshop operations are provided below Workshop agenda The agenda at Figure 3 sets out the basic components of a program logic workshop The agenda will vary in different contexts Figure 3 Sample program logic workshop agenda PROGRAM LOGIC WORKSHOP Agenda Scene setting Scoping Outcomes Assumptions Evaluation guestions DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT Resources Some resources that are regularly reguired at program logic workshops are listed below Key reference documents Sufficient wall space for temporary adhesives Shower curtains explained later sprayed with repositionable spray available from office supply stores A whiteboard Marker pens Lots of sheets of A5 paper in an assortment of colours Pre typed cards that outline the levels of the outcomes hierarchy Cut out arrows for illustrating links between the outcomes in the hierarchy People Actively involve key participants in setting outcomes for the pr
19. than to answer several questions poorly These parameters are often negotiated among stakeholders How well the questions are answered will depend on the time money and expertise available to perform the functions required by the evaluation answering several questions poorly Answer a few important questions thoroughly rather than A TIP The checklist at Figure 13 can be used to assess whether the main considerations in constructing useful and realistic evaluation questions have been addressed DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 37 0 F TL na X oc fe 0 E l x 5 I 0 E I a E lt YN cc IL cc 38 Go to 50 Figure 13 Evaluation questions checklist Progress towards formulating Yes Not Comments evaluation questions yet revisions Key program stakeholders were consulted engaged in the workshop Information requirements were identified for each stakeholder group There are a manageable number of questions at each outcome level Questions address how critical timing is for the project Questions generally relate to extent of change impact contribution to longer term impact An evaluation category was identified for each question Questions address the required evaluation purposes Participants are confident that the questions will provide information to meet
20. the onger term outcomes and aspirational goal Having this framework helps to focus on questions that have real value for all stakeholders 2 2 6 6 ot only does the program logic assist in generating information for external audiences it enables adaptive management and internal learning The MERI Framework suggests that evaluations address five evaluation categories Evaluation categories ATTACHMENTS Impact In what ways and to what extent has the program contributed to changing asset condition and management practices and institutions KEY CONCEPT What if any unanticipated positive or negative changes or other outcomes have resulted REFERENCES To what extent were the changes directly or indirectly produced by the program interventions Appropriateness To what extent is the program aligned with the needs of the intended beneficiaries To what extent is the program compliant with recognised best practice processes in the field e g the type level and context of investment and associated activities How time critical is the program Effectiveness To what extent have the planned activities and outputs been achieved re current activities the best way to maximise impact or are there other strategies that might be more effective To what extent is the program attaining or expected to attain its objectives efficiently and in a way that is sustainable Efficiency To what
21. the program logic Significant impacts in NRM may not be seen for a long time sometimes in the order of decades What needs to change Deciding what needs to change involves answering the following questions Which assets are to be protected through the program How are those assets valued What is known about the current condition of the key assets the program is concerned with What is contributing to deterioration in asset condition and are there mitigation strategies for threats to the assets How urgent is the change what are the potential costs of not acting Consideration of assets may include social institutional economic and environmental assets Agreement about what needs to change in the broader NRM context and what the program will set out to achieve as a contribution to that change in the bigger picture aspirational longer term outcomes should be reached before the workshop as part of a problem analysis process This allows workshop participants to focus on the outcome statements at the next levels down the outcomes hierarchy DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 19 0 ER m EE ep x 20 Go to page 47 The organisation that is implementing the program will benefit from having a documented account of the program logic and associated scoping discussion The context in which decisions were made will always be relevant for future de
22. the requirements of stakeholders Participants are confident that evaluation reports will meet the multiple requirements and needs of stakeholders DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT The outcomes hierarchy will assist in assessing achievements against expected outcomes and understanding why differences occur which they inevitably will For example some regional NRM organisations have incorporated an outcomes hierarchy within their annual investment plan review Many programs focus on activities during progress reviews but this is not enough to manage for impact The outcomes hierarchy provides a valuable focus for regular semi annual and annual review and planning events What a program aims to achieve is the intermediate and longer term outcomes while outputs and foundational activities describe how it thinks it can do this A progress review needs to look at both the how and the what If an organisation looks only at the activities and outputs it could conclude these are all going as planned But it is also useful to ask Where is this leading to assess whether the program is on track with the planned outcomes This will help avoid wasting time and resources on unproductive outputs and activities Assumptions should be reviewed regularly to check that they are still valid This is the time to identify new assumptions that have emerged and delete those
23. OM 43 saae euojjepuno4 5 pue 5 206 jeuonesdsy suondunssy 5 5 suondunsse 2 199USAJ1OM SLSITMDSHS SLNAINHOVLLV EPO EL 3W3U1X3 S ALVYACGOW YONIW 7 INVOIAINDISNI 1 SII NINOISNO NIVLY3D LSOW1V 5 11815504 ATX INN C S L S L si uondwnsse uojydunsse 9169 3 195 3 S14 10 sosuonbosuo Jo pooy jay 7 SASIY uondwnssy J99UYSAJOM 45 sanianoe euonepunoy s pe 206 jeuoneudsy lt asoding suol sanb uonenjeng 5 suons nb y99YS4JOM 0 ET e SLNAINNHOVLLV CHECKLISTS Checklist 1 Scoping Progress towards defining program boundaries Yes Not yet Comments revisions Information about the assets and attem
24. arian outcomes X Sub catchment under coordinated weed control REFERENCES Stocktaking monitoring and benchmarking done periodically routinely Practice and Increased capacity to X land managers diversify Increased interest X land managers taking attitude change access support their business in biodiversity Landscape scale action Maintenance actions to Remnant veg fenced address on going threats issues on X properties Outputs Sustainably grazing on X of farms Cross boundary activities Groundcover maintained X ha of remnant Construction of watering Construction of contour and on ground works on X of properties veg fenced points on X properties banks on X properties biophysical Outputs Increase in landholder awareness ability X people inspired and motivated X landholders access technical training Information gathered and shared X approved non biophisical and motivation to manage sustainably by neighbours through training events amp field days to increase group knowledge sub catchment plans Technical and funding support Sub catchment planning Priority setting process Group established E 5 gained for innovative practices Facilitators Funding 55 REFERENCES AND ADDITIONAL READING REFERENCES References and additional reading Australian Government Natural Resource Management Monitoring Evaluation Reporting and Improvemen
25. can help to ensure that the significant parts of each step have been completed and to keep a record of activities or information that is still reguired Figure 4 Four key steps Scoping defining the program boundaries Step 1 Developing an outcomes hierarchy expectations about change Step 2 Articulating and documenting assumptions theory of change Step 3 Formulating evaluation guestions and audiences Yule DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT Step 1 Scoping When establishing a program logic a useful starting point is to clarify the boundaries of the program Ideally this is done in advance of the program logic workshop Identifying program boundaries The program boundaries can be identified by addressing four guestions KEY CONCEPT Is the program outcome realistic in the given timeframe What needs to change and which changes are most urgent Are there any rules or regulations that impact on the program Who will use the program logic to implement the program and who are the audiences for information generated by an evaluation based on the program logic What resources are available Is the program outcome realistic Many programs have lowered their chances of success by setting unrealistic and unachievable or unmeasurable program outcomes Often the program outcome really represents an intermediate outcome in
26. cision making Who will use the program logic to guide program evaluation Who are users and stakeholders for the program logic and the evaluation reports that will be based on the logic model What strategies are needed to ensure that those who are to use it for decision making understand the evaluation approach and what it will deliver to them What types of communication products will be best for each audience What resources are available Arethe available resources including funding people technical equipment data and data collectors adequate to achieve the program outcomes in the period available Are there other people or organisations that could contribute to the program and if so what are the best ways to engage them and or establish partnerships The checklist at Figure 5 guides reflection on the scoping process setting boundaries and clarifying some fundamental questions Have the key outputs from scoping been achieved What work remains to be done for a thorough scoping process Figure 5 Scoping checklist Progress towards defining program boundaries Yes Not yet Comments revisions Information about the assets and attempts to manage and or improve them has been explored including consultation with experts in the field The condition of the assets and issues with managing the assets and threats have been identified The assets for change have been specified and are consistent wi
27. dge base frameworks policies evaluations development strategies programs and plans FOUNDATIONAL ACTIVITIES DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 11 TIP This logic underpins the MERI Framework and acknowledges that NRM operates at a range of scales and over different timeframes The actual timeframe for each outcome level will depend on the length of programs and funding cycles The logic acknowledges that to achieve and adequately report on desired outcomes there must be a focus on both the means and the ends There are two important investment streams of NRM assets investment in biophysical outcomes ends investment in social institutional and economic outcomes means to achieve biophysical outcomes It will always be necessary to invest both in on ground environmental improvement activities and in supporting and building the knowledge base and skills of the people and institutions that manage the environment and the resources derived from it This dual investment helps to ensure that threats and pressures are reduced and that these assets are protected and enhanced DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT SECTION 2 2 Preparing to construct program logic deally a program logic is established during the development phase of a program and then refined as often as necessary during program implementation A p
28. e MERI Framework While program logic serves planning and management functions it is also an element of the evaluation process as described in the Australian Government s NRM MERI Framework The MERI Framework outlines key concepts and principles for understanding NRM program performance It presents the MERI steps as integrated components of program design program logic monitoring evaluation and reporting improvement and adaptive management One of the underlying principles of the MERI Framework is that the MERI process is embedded within the program planning and implementation cycle see Figure 1 Figure 1 The logic of MERI in program design Design program including MERI strategy Identify priority Implement assets desired program and outcomes and MONITOR program logic activities CONTINUOUS PARTICIPATION Adapt strategy AND COMMUNICATION EVALUATE accordingly and appropriateness apply to next impact program phase for effectiveness IMPROVEMENT efficiency and legacy Reflect on what s working what s not and why REPORT on outcomes and communicate learning DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT Components of the MERI Framework Table 1 illustrates the MERI Framework s key components and outputs The framework iterative activities recur throughout program planning design and implementation This guide is concerned with the first component establish
29. e in appropriate grazing Land managers are aware of and apply best management practices Land managers have the economic capacity to apply best management practices Increased number of land managers involved in fencing Program funding will continue Fences will manage the grazing threat Decline in guality and extent of vegetation and habitat can be stopped through management of identified threats Increased understanding of the principles and practices for revegetation and enhancement Land managers attend training Land managers find training useful and are prepared accept recommendations Increased land manager knowledge of what vegetation occurs on the site Land managers are aware of the principles and practices for revegetation and enhancement and apply that information Immediate activities and outcomes Increased number of voluntary 10 year management agreements Increased number of permanent covenants on areas of remnant protected or enhanced vegetation Compliance with covenants Funding Landholders agree to fence Fencing will stop the decline of quality and extent of vegetation and habitat Land managers participate in fencing Land managers need incentives Increased land manager knowledge of threats that could be managed Land managers engage in education and training initiatives Content and style of education and training are appropriate for land managers
30. economic influences on natural resource management decisions State level Investment Project SEO5 National Action Plan for Salinity and Water Quality Queensland Department of Natural Resources Mines and Water Taylor Powell E amp Henert E 2008 Developing a logic model teaching and training guide University of Wisconsin Extension Cooperative Extension Program Development and Evaluation Madison Wisconsin www uwex edu ces pdande evaluation pdf Imguidecomplete pdf URS Australian 2003 Defining social assets for the Salinity Investment Framework report prepared for the Western Australian Water and Rivers Commission Perth WK Kellogg Foundation 2004 Logic model development guide Michigan www wkkf org Walters CJ amp Holling CS 1990 Large scale management experiments and learning by doing Ecology vol 71 6 pp 2060 68 DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT
31. ed 26 DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT Articulating and documenting assumptions The next step in developing the program logic is to articulate and document the groups assumptions about how one program outcome or activity will lead to the next and to identify any risks associated with the assumptions assumptions are expectations based on current knowledge and experience about what is important for a project s success Importantly the program logic is a model not reality It depicts assumed DEFINITION causal connections not true cause and effect relationships Identifying the underlying assumptions provides a focus for testing and adapting the logic TIP Throughout this process it is helpful to refer back the table of evidence that is available and or needed to illustrate outcomes Uncovering the assumptions To get at the assumptions it is useful to talk in terms of the outcomes hierarchy A facilitated group discussion provides an opportunity for participants to work through the hierarchy They can articulate assumptions about the outcomes and the relationships among the different types of intervention strategies and activities and how they will lead to change This then enables participants to discuss how the series of consequences in the outcome hierarchy will occur implementation of the program can be debated and documented as in the examp
32. eers with the required capacity who stay on and will all the trees survive Will the trees reach maturity and will native fauna return to the area These are examples of internal logic assumptions that lie behind the simple statement Planting trees results in biodiversity In this exercise to identify the risks to assumptions one group of participants can look at the problems with the logic Another group can suggest ways of overcoming the problems This will lead to some refinement of the logic Making assumptions explicit helps to check where the objective hierarchy has weak spots and so reveals what needs to be adjusted The checklist at Figure 11 assists in making sure the key steps in articulating and documenting assumptions have been covered Figure 11 Articulating and documenting assumptions checklist Progress towards articulating and Yes Not Comments documenting assumptions yet revisions Assumptions about how change will occur were discussed Assumptions about how change will occur were documented Risks that could affect the achievement of outcomes were identified and documented DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT WORKSHEETS Evaluation questions provide basis for assessing the extent to which outcomes have been achieved at the different levels of the outcomes hierarchy and the extent to which the program has made a contribution towards
33. em in which the program will operate and how time critical activities and outcomes are an outcomes hierarchy with clear outcome statements documented assumptions about the cause and effect of interventions at each level ofthe outcomes hierarchy key evaluation question plus evaluation questions that sit below the key evaluation question and align to each level of the outcomes hierarchy Tips for the program logic workshop Actively involve key participants in setting outcomes for the program Welcome the opinions and ideas of all participants Keep a positive outlook by looking for positive outcomes in the first instance rather than potential problems Work in small interactive groups Use culturally appropriate examples to introduce ideas Use language that will be familiar to participants Build a shared vision through participatory decision making Create a draft model revise it and continue to refine it over time Recognise that logic model development is not quick or easy develop the draft model first before analysing it Adapted from lt www uwex edu ces pdande evaluation pdf Imguidecomplete pdf gt DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT SECTION 3 Steps to build a program logic This section describes each of the four steps in constructing a program logic A checklist is provided at the end of each step Using the checklists
34. ermediate outcomes help to identify the pathway This activity is usually both fun and productive as it generates discussion and debate The guide to writing outcome statements may be useful DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT WORKSHEETS KEY CONCEPT Y l x I Constructing the hierarchy The NRM outcomes hierarchy Figure 2 has five levels lt Go to 11 aspirational program goal longer term outcomes intermediate outcomes immediate activities and outcomes foundational activities When constructing a program logic the aspirational program goal is at the top level of the hierarchy Social environmental and industry economic outcomes can all be represented at the longer term level Through the aspirational goal and the longer term outcomes that sit beneath it the outcomes hierarchy provides a picture of how the program will contribute to the aspirational goal KEY CONCEPT In practice this 15 iterative and the statements do quite a lot of moving around amid group discussion before agreement is reached DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 23 24 Go to page 51 Go to page 55 DEFINITION Naturally participants will have some disagreements about the outcomes They should be encouraged to engage in debate and move the statements around until there is general agreement Once general agreement has been
35. ew evidence required Evidence Evidence should be considered after the assumptions are documented should be step 3 and again after the evaluation questions have been formulated and considered after agreed step 4 the assumptions are documented step 3 and consider existing local and external evidence again after the evaluation guestions have been formulated and agreed step 4 Participants should flag the kinds of new guantitative and gualitative evidence needed to address the evaluation guestions In some cases it will be necessary to seek advice from experts on the availability of evidence to support the outcomes and potential costs involved in generating new evidence This can occur during the program logic workshop and after the workshop as reguired Figure 8 Outcomes hierarchy checklist Go to page 44 Outcomes stated succinctly indicating clearly what change will look like Statements are in plain English with no ambiguity Assets for change have been specified and they are consistent with high level program priorities The type of evidence that is or is likely to be available to measure the stated outcomes have been identified Outcomes are likely to be achieved in the program timeframe within the funding and reporting periods Data sources and resources for collecting analysing and reporting that may be required to achieve outcomes have been identifi
36. extent has the program attained the highest value out of available resources How could resources be used more productively and efficiently What could be done differently to improve implementation and thereby maximise impact at an acceptable and sustainable cost Legacy Will the program s impacts continue over time and after the program ceases How and by whom should the legacy be managed 33 34 g DEFINITION Go to page 53 DEFINITION Go to page 53 Depending on the program the reporting requirements may include all of these categories but it is useful to cross check Some types of assessment will be required as part of an organisations funding agreement Others will be important to certain stakeholders including the organisation itself Some evaluation guestions that emerge from the logic will possibly address more than one category The final decision about which categories the program logic addresses may well be determined by the program budget or information and reporting requirements Two levels of evaluation questions The MERI Framework defines two types of evaluation questions the key evaluation question and the evaluation questions that break down the elements of the key evaluation question key evaluation question assesses the worth or significance of a project program or strategy in relation to its expected outcomes The purpose of the key evaluation question is
37. field days were held and how many land managers attended How many land managers said they would do something differently as a result of attendance at an event How many property visits were done What was the most significant trigger for a property visit Foundational What investment prioritisation activities were undertaken activities Was existing research and knowledge used to inform program planning Did the program development involve all key stakeholders When developing evaluation questions participants should draw on the logic model and assumptions about how change will occur It is useful to start by asking What is going to be evaluated at each level of the outcomes hierarchy Ouestions should be restricted to a minimum number of the most important aspects of the program at each level of the hierarchy The evaluation guestions should fall out of the outcome statements and TIP assumptions For example the assumption Land managers will engage in education and training initiatives gives rise to the guestion To what extent and in what ways has knowledge of condition and trend improved Asking why and why not will assist in the learning and adaptive management process The questions can be documented by small groups and then refined in the large workshop group DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 35 0 F TL uw X oc 02 E l x 5 I
38. g all the components of program logic recorded in this way enables regular review of progress towards program outcomes and retention of corporate memory When new people enter the organisation or program they will be able to see the assumptions that underpin the program strategy Attachments 1 2 and 3 provide a detailed illustration of a logic hierarchy for different types of programs The hierarchies were developed by regional NRM organisations The diagrams provide a good basis for discussion and debate about assumptions about the causes and effects in change pathways The references and readings invite the user ofthe guide to further explore the world of program logic DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 41 REFERENCES ATTACHMENTS CHECKLISTS WORKSHEETS SLASSHSMYOM SKS ED ET fo SLNAINHOVLLV WORKSHEETS SLSITHDSHO SLNAINNHOVLLV NIVL439 LSOWTV S 1VILN3SS3 5 AMAN Y INVIHOdWI 11915504 LNVLYOdWI C 1 GOOHITAMIT INVIHOdWI INVIHOdWINN L 1 4 S L 558222 0 ajqe Bulag jo pooy jay S L auapiAa Jo 5 SOUSPIAZ 199US4J
39. g the training relevant and accessible they cannot be held accountable for whether the farmers have all the necessary conditions on farm to use the new skills they have learned Itis important for the group Itis important for the group to brainstorm about what circumstances could to brainstorm present a risk to the likelinood of an assumption being correct that is about what that the interventions do not lead to the results assumed in the logic The circumstances worksheet at Figure 10 is designed to guide an interrogation of the risks to could present the logic so that risk management strategies can be put in place The exercise a risk to the can also help to prioritise the outcomes Often these will be the ones that rate Ji elihood of high in the worksheet an assumption being correct Figure 10 Risk worksheet 2 SHAB G 4 Assumption Risks Likelihood of Consequences for Risk Da assumption being longer term outcomes management wrong if assumption iswrong strategies 1 5 1 rare 1 5 1 insignificant 5 almost certain 5 extreme How likely is it that the assumption is wrong and how great will the impact be on the program s ability to achieve reguired outcomes if the assumption is wrong To complete the risk worksheet participants should begin by identifying external influences that may affect the program interventions and outcomes What environmental events or factors could affect program performance
40. gram and when reviewing an existing program logic In either case to develop a version of participatory MERI that suits an organisations situation it is important to first determine in which part of the MERI process participation is most important Who is going to use the final information from each stage of the process Those who will use the information for decision making should understand what it is based on and how it was calculated otherwise they will not understand its implications It is also important to determine what skills are required to develop a program logic The more complex the program logic the more caution should be used encouraging broad participation unless it is clear whom it will benefit and how Different people have different motivations for getting involved in MERI Support may be required to assist the different systems to work together lt is often helpful to negotiate and agree on how much participation for whom The more complex the program logic the more caution should be used in encouraging broad participation unless it is clear whom it will benefit and how Even if program and partner staff and primary stakeholders are motivated they still need to see progress and results if they to keep investing time and energy into joint learning Also depending on the organisations size and resources it may be necessary to invest in building capacity The big picture of NRM outcomes Fi
41. gure 2 represents the Australian Government NRM outcomes hierarchy It illustrates the series of changes in NRM assets that the Australian Government hopes to contribute to at the national level through investment in NRM programs The outcomes hierarchy provides an overarching framework to guide the logic of specific programs and initiatives Assets are useful things or qualities something that has a value the context assets can be human social natural physical or financial This generic NRM outcomes hierarchy is based on assumptions about the series of consequences that are likely to lead to improvements in the extent and or condition of NRM assets including biophysical social institutional and economic assets DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT Figure 2 Generic NRM outcomes hierarchy Conserve repair and replenish Australia s natural capital ASPIRATIONAL PROGRAM GOAL Resource managers institutions Biodiversity conservation and industries have capacity to manage NRM assets sustainably LONGER TERM OUTCOMES Enhanced NRM engagement Biophysical social institutional awareness capacity and projects and activities and partnerships between individuals their outputs communities and regional organisations IMMEDIATE 2 lt 9 08 ER 25 O lt Appropriate design Baseline assessments Awareness skills of NRM institutional and results of program and knowle
42. ing the program logic Table 1 Key components of the MERI Framework Component Outputs Program Desired changes and the types and extent of changes expected at different scales logic Key assumptions about how change will occur Anticipated outputs and outcomes Evidence in support of logic Key evaluation guestions and methods Specified targets for outcomes Performance indicators both gualitative and guantitative Monitoring Collation of relevant new and existing guantitative and gualitative data to address evaluation guestions Assessing relevant data against outcome targets Data and information management Evaluation and reporting Reflection on results of monitoring against the relevant evaluation guestions Assessment of the impact appropriateness effectiveness efficiency and legacy at different stages of the program to determine immediate intermediate and longer term outcomes Communication of evaluation results to internal stakeholders and key external stakeholders Improvement Reflection on what is working and what is not working based on monitoring and adaptive data and evaluation reports management Amendment of program strategies based on reflection on monitoring results and outcome reports e e Participatory MERI Developing a program logic can be a dynamic highly participatory activity The iterative nature of the MERI Framework promotes continuous partic
43. ipation communication and learning rather than viewing evaluation as a single event Such a learning environment enables people to reflect critically on what is happening A learning environment can be created by small changes as well as by more far reaching events and changes Any program can integrate multiple ways of working in order to stimulate learning from the very way in which a program is designed to how investment happens as well as annual reviews with program stakeholders Critical to this is the role of senior management in setting the example and facilitating dialogue between program partners Participatory MERI is based on organisational values that facilitate a common understanding of the problems to be addressed and the underlying assumptions about how change will occur This leads to sustained engagement and effort DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT The MERI Framework promotes continuous participation communication and learning rather than viewing evaluation as a single event CAUTION DEFINITION Participatory MERI means clarifying who manages the process and who benefits from the findings The point when the program team sits down with staff from partner organisations and with the target group is the time to talk about roles within the partnership A participatory approach is relevant both at the time of developing a program logic at the conceptual stage of a pro
44. le at Table 2 see page 28 which uses the regional biodiversity project outcomes hierarchy from Attachment 2 A template for documenting Go to page 44 assumptions in program logic is at Worksheet 2 see page 44 at the end oz of this document Assumptions about how change is expected to occur through the Gaita 2 DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 27 0 F I X oc 0 E a l x 5 T 0 E T E 28 Table 2 Example of outcomes hierarchy with assumptions documented Outcome statements Assumptions E 3 The program contributes Data is available and can be meaningfully aggregated 2 to biodiversity from local to national level 5 conservation Correct indicators a lt 8 Stable conditions Sa Resources are available for data collection and management 9 Net gain extent Stable climatic conditions 5 distribution and quality stable land use patterns 2 of all native vegetation 5 gt Stable land tenure status communities Stable availability of incentives E Net gain in vegetation Land managers use best management practices 5 communities on farms a 5 Increase in number of Best practice is known land managers applying same land managers over time biodiversity conservation 2 skills Land managers engage in education and training initiatives Content and style of education and traini
45. logic in the context of NRM Users may choose to change the seguence of the steps or to use only some of the steps at different times The guide may be used by trainers in program logic or by an NRM organisation to develop program logic for a specific program or project The guide is presented in a user friendly format and should be easily adapted to these different user applications Features that increase the usability ofthe guide include each section is represented by a distinctive colour and a tab making it easy to access particular parts of the guide as required aseries of templates at the end of the document provide checklists and worksheets that can be copied and used to create particular program logic documents important points to note are highlighted in colour on the page margins The following symbols draw the user to parts of the program logic methodology that require particular attention A e KEY CONCEPT CAUTION DEFINITION Fundamental An area where A suggestion The meaning component the rigour ofthe for making of a key term of program methodology the process used in this logic could be affected smoother guide if care is not more efficient taken or more fun The following arrows direct the user either forward or backward to the relevant section of the document Go to page 30 Go to page 13 All of these aids to navigating the guide are designed to make it easy for the user to find their way around the
46. ng are appropriate for land managers Land managers will adopt new land management practice Land managers will comply with covenants E Increase in extent of Stable climatic conditions 5 native vegetation for Stable land use patterns 2 threatened communities E Stable land tenure status 2 Improved quality of Stable availability ofincentives 5 5 threatened MIE Land managers use best management practices vegetation communities Fewer other threats Land managers use best management practices farm Enhancement of vegetation on farms Increase in extent of remnants and weeds managed appropriately Land managers are aware of and apply best management practices Increased appreciation for biodiversity values Land managers want to participate in field days People do not currently value biodiversity People will be prepared to change their values People will be prepared to forgo some profit for biodiversity Same land managers over time Land managers engage in education and training initiatives Content and style of education and training are appropriate for land managers Land managers need knowledge and financial incentives to develop sustainable native vegetation management skills Landholders will take up incentives and accept voluntary agreements DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT Outcome statements Assumptions Intermediate outcomes continued Increas
47. ogram This could include strategic thinkers and those who will make funding allocation decisions MERI implementers and key stakeholders including service users and financial contributors including potential contributors For those creating a program logic for the first time it can be helpful to engage a facilitator who is experienced in participatory planning and evaluation processes It can also be useful to bring in a professional evaluator as a mentor so that the organisation can develop its program logic skills Scene setting At the outset of the workshop a facilitator generally introduces participants and asks what their interest is the program and their prior experience with program logic Outlines the agenda for the day emphasises the participatory nature of the workshop introduces the workshop rules provides an overview of the environment and systems in which the program logic and program will operate provides an overview of the high level outcomes required by funding organisations and management boards and an outline in broad terms of what might need to change for the outcomes to be achieved DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 15 Workshop products The products from the program logic workshop include ashared understanding of the outcomes to be achieved and the assumptions about how change will occur documented analysis of the environment and syst
48. olders from the beginning helps to ensure that meaningful information is gathered in which program supporters have a real interest Criteria for refining evaluation questions Does the evaluation question address the type of evaluation required for the program Which stakeholder groups are interested in this information Would knowing the answer to this question improve the program Will this information assess the programs effectiveness How will this information inform better program management practices through internal learning Are there data available to answer the guestions Are there adeguate resources available to collect and analyse the identified data and report on the guestions Can some of the questions and supporting data satisfy multiple reporting reguirements DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT At the end of this exercise workshop participants should have enough Go to page 46 information to complete the worksheet on evaluation guestions and eS stakeholders as illustrated in Figure 12 Figure 12 Evaluation questions and stakeholders worksheet Outcome level Outcome Category Stakeholders guestions Aspirational program goal Longer term outcomes Intermediate outcomes Immediate activities and outcomes Foundational activities It is important to keep the evaluation manageable It is preferable to answer a few important questions thoroughly
49. phisical X landholders agree to fence Priority guidelines for threatened veg communities and habitat Program funds Collaboration across programs Assessment criteria Landholders ED materials 53 ASPIRATIONAL LONGER TERM INTERMEDIATE IMMEDIATE ACTIVITIES FOUNDATIONAL INPUTS AND ACTIVITIES PROGRAM GOAL OUTCOMES OUTCOMES Attachment 3 Example outcomes hierarchy for sub catchment planning Outcome Hierachy Definition of outcome The project contributes to resilient communities that proactively work together to build Vision for the asset environmentally healthy sustainable attractive and regions Improvements in the X increase in the extent of effective X of improved environmental management x of land managed for soil condition X increased profitability and state of asset management of natural resources including ground cover vegetation and riparian outcomes robustness of businesses Aggregate changes in how the asset is managed across a geographical area Increased community cohesion and capacity in region using local knowledge and support Increased landholder X reduced movement X in reduced risk of X land managed for soil 7 AR capacity to address or cope with of key people out of ne dd water erosion through water condition vegetation and market and climate variability the area Improved weed contro run off control rip
50. pts to manage and or improve them have been explored including consultation with experts in the field The condition of the assets and issues with managing the assets and threats have been identified The assets for change have been specified and are consistent with high level program priorities Required changes to the assets that are time critical have been identified The stated aspirational goal and longer term outcomes for the program are achievable within the funding periods specified Available and or required resources including funding staff access to research scientific data and opportunities for partnerships have been identified 47 ATTACHMENTS REFERENCES Checklist 2 Outcomes hierarchy Progress towards developing an outcomes hierarchy Yes Not yet Comments revisions Outcomes are stated succinctly indicating clearly what change will look like Statements are in plain English with no ambiguity Assets for change have been specified and they are consistent with high level program priorities The types of evidence that is or is likely to be available to measure the stated outcomes have been identified Outcomes are likely to be achieved in the program timeframe within the funding and reporting periods Data sources and resources for collecting analysing and reporting that may be required to achieve outcomes have been identified
51. rogram logic can also be developed for an existing program Developing a program logic begins with reflecting on the aspirational goal for the program and the immediate intermediate and longer term outputs activities and outcomes Through this process original assumptions about how change would occur not simply a plot of what did occur are captured and documented Program logic principles Applying the simple principles set out below can help to develop a program logic Program logic principles Developing a program logic is a participatory process A clear understanding and agreement is reguired among KEY CONCEPT participants about what needs to change and how the program can best contribute to that change in the context of the bigger sustainability picture Discussions of people s visions and aspirations are more useful than statements of problems A focus on assets helps to conceptualise reguired change Explicit immediate and intermediate outcomes pave the way for establishing program strategies and activities Program logic is more complete when accompanied by an analysis of conditions outside the program that are critical for the program to succeed Program logic is more complete when areas of uncertainty are explicitly stated Tracking assumptions as part of the evaluation process and updating the program logic increase its value DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT The
52. s hierarchy see Figure 9 Figure 9 The limits of control and accountability in a program Foundational Immediate Intermediate Longer term Aspirational goal Degree of control and accountability At the level of foundational activities and immediate outcomes and activities program managers and staff have much control External factors are unlikely to pose serious threats to carrying out activities at these levels But at the intermediate longer term and aspirational outcome levels many factors beyond the direct control of program managers will influence the impact of activities An NRM project is usually one of many contributing to improvements in the quality of a national asset such as water knowledge productivity of agriculture or Indigenous management skills project s accountability at the higher levels of the outcomes hierarchy decreases but never disappears entirely For example a project might include training activities for farmers The program can directly control the hiring of a training venue the preparation of materials the provision of a gualified trainer and the invitation of suitable participants DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT It has less control over whether potential participants will attend and considerably less control if any over whether the skills the participants learn will actually be used back on the farm While the trainers can be held accountable for makin
53. sense of nurturing the landscape ownership activities including seed collection propagation The role of public volunteers is accepted amp empowerment weed control monitoring Increased awareness of provenance issues leading to seed banks and informed re veg Direct seeding by farmers Plantings by farmers volunteers and landholders Remnant veg fenced Riparian off stream watering amp fencing Volunteers and landholders trained in seed collection Field days for landholders Vegetation amp fauna Questionnaires completed farmers amp volunteers surveys by volunteers Volunteer engagement sees Bus tours for volunteers and activities Maintenance days Consolidate knowledge about and organisational arrangements for volunteer recruitment training and management Consolidate technical knowledge Site information species selection Community members Program funds and relationships partnerships site characteristics soil climate etc 51 ASPIRATIONAL LONGER TERM INTERMEDIATE IMMEDIATE ACTIVITIES FOUNDATIONAL INPUTS AND ACTIVITIES PROGRAM GOAL OUTCOMES OUTCOMES Attachment 2 Example outcomes hierarchy for a regional biodiversity project Outcome Hierachy Definition of outcome Vision for the asset The program contributes to biodiversity conservation Improvements in the X net gain in the extent distribution 8 guality for all veg communities X
54. t MERI Framework 2008 www nrm gov au publications frameworks meri framework html Adrien 8 MacPherson N 2004 A guide for the planning and conduct of IUCN strategic reviews IUCN Cambridge Australasian Evaluation Society Inc 2006 Guidelines for the ethical conduct of evaluations www aes asn au about Documents 20 20ongoing guidelines_ for_the_ethical_conduct_of_evaluations pdf Board of Regents of the University of Wisconsin System 2002 Enhancing program performance with logic models www uwex edu ces Imcourse Carroll J amp McKenna J 2001 Theory to practice using the logic model to organize and report research results in a collaborative project Journal of Family and Consumer Science vol 93 no 4 pp 63 65 Dart 2007 unpublished Training notes monitoring evaluation reporting and improvement MERI for natural resource management in Australia Earl S Carden F amp Smutylo T 2001 Outcome mapping building learning and reflection into development programs International Development Research Centre Ottawa www idrc ca IFAD International Federation of Agricultural Development nd Managing for impact in rural development a guide for project M amp E Annex A Glossary of M amp E concepts and terms www ifad org evaluation guide index htm Imbach AC ed 2000 Finding the way a practical guide to the organisation and execution of the self evaluation processes of projects centred in sustainability
55. th high level program priorities Required changes to the assets that are time critical have been identified The stated aspirational goal and longer term outcomes for the program are achievable within the funding periods specified Available and or required resources including funding staff access to research scientific data and opportunities for partnerships have been identified DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT Developing outcomes hierarchy NRM outcomes are generally long term and involve a wide range of often complex activities These features require a program logic that strikes a balance between including enough detail and oversimplification Specific timeframes can be allocated at each level of the hierarchy especially at the intermediate and longer term levels However the program logic may be limited to the life of the program This is often determined by funding agreements Framed in the bigger picture longer term biophysical program outcomes may take decades Specific program guidelines and MERI strategies will often guide timelines An outcomes hierarchy can provide a succinct diagram that summarises the series of planned changes the program will contribute to the program at a glance This can be used to provide an overview of the program for external communications and to inform program funding decision makers who often do not have time
56. that are no longer relevant Based on the assessment of problems successes and assumptions each level of the hierarchy should be checked for relevance and completeness New activities or outputs should be added and irrelevant ones deleted in line with the assessment DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT The outcomes hierarchy provides a valuable focus for regular semi annual and annual review and planning events Assumptions should be reviewed regularly to check that they are still valid This is the time to identify new assumptions that have emerged and delete those that are no longer relevant 39 0 F TL na X oc fe 02 E a x 5 I 0 E I lt E lt W cc IL cc SLASSHSMYOM SLSITHDSHO SENEL SIN3NHOV ILV TOOLS SECTION 4 Tools The tools included in this section are designed to assist trainers and developers of program logic They include worksheets to provide templates for users to fill in their own details as they work through the program logic process checklists to ensure that all components of each key step in developing a program logic have been covered The worksheets and checklists have been designed to be copied and used in training or to form the basis of a documented program logic for a particular program Havin
57. to absorb long documents For those using the program logic as a management tool more detail will be needed Think positively about how the program can contribute to improving the assets under consideration 17 F Lu I x oc 17 F 77 l x I Only include activities after the first level of the hierarchy i foundational level Rather than creating a flow chart or action plan focus on consequences TIP Resources Magic walls which are shower curtains sprayed with repositionable spray are an excellent way to display the outcomes hierarchy and to engage Go to page 11 participants in building it Once the shower curtain is affixed to the wall pre written cards that outline the levels of the outcomes hierarchy as guided by the outcomes hierarchy in the NRM MERI Framework at Figure 2 page 11 are placed down the left hand side of the curtain Figure 6 overleaf explains in more detail how to build a magic wall DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT 21 22 Figure 6 Building the magic wall How to build magic wall Materials Aplastic shower curtain in a soft solid colour Acan of repositionable spray available from office supply stores Method Open out the curtain and lay it flat on some suitable surface not carpet Try to smooth any creases it may be necessary to leave the curtain out
58. to generate information that can be used to demonstrate the results ofthe program to funding bodies and the community lt should focus most on the effectiveness ofthe program in achieving intermediate and longer term outcomes and help to demonstrate the contribution the program has made For example taking the program logic in the outcomes hierarchy for the biodiversity project Attachment 2 the key evaluation would be To what extent has the project contributed to an increase in the extent distribution and quality of native vegetation This overarching question frames the evaluation A number of more specific evaluation questions will sit below the key evaluation question 4 evaluation questions represent a breakdown of the key evaluation question In the context of MERI for NRM the evaluation questions link to the outcomes in the different levels of the program logic particularly the foundational immediate and some extent intermediate outcomes Both kinds of evaluation questions generate information that shows the extent to which the program has had the expected impacts and provides a basis for sharing with others the successes and lessons learned through implementing the program Using the biodiversity project example from Attachment 2 again Table 3 sets out an illustrative set of evaluation questions that correspond to the program logic DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT
59. ture of their programs and the lines can make it even more complex Identifying evidence to support the logic How logical is the logic What evidence is known available or needed in order to monitor progress toward the outcomes Once the outcomes have been identified the next task is to consider whether there is evidence available to answer the guestion To what extent has the program contributed to Reviewing and considering existing evidence in support of the outcome hierarchy is a critical step This is generally a brainstorming process to allow people to list the information and data they are aware of and where more evidence is needed Additional evidence can be accessed or generated outside of the program logic workshop The logic can be refined and supported with evidence that indicates that the outcomes can be measured Usually the analysis would be applied to only the most important outcomes at each level of the hierarchy DEVELOPING AND USING PROGRAM LOGIC IN NATURAL RESOURCE MANAGEMENT Types of evidence could include research reports national state regional or local surveys studies and reports published material on the subject target monitoring benchmarking studies investment financial output reports photo point monitoring expert panels Evidence selected to support the program logic should be appropriate and affordable for the particular logic model and program arrangements
Download Pdf Manuals
Related Search