The PD monitoring process in depth

PD staff and implementing partners should complete the monitoring process for a subset of PD section activities (see What should I monitor). The PD monitoring process includes three parts:

1. Logic model
2. Monitoring plan
3. After-action review

Logic model

What is a logic model?

Logic models are visual guides that outline how section activities will lead to expected changes in priority audience groups, ultimately achieving a defined objective or goal. Similar to a math equation or recipe, a logic model ensures balance between what goes in on one side and what comes out on the other. In the context of public diplomacy, the logic model combines inputs (resources), audiences, and actions to produce outputs and short-term and long-term outcomes. The logic model serves as a template for "if-then" thinking, helping to articulate the relationship between actions and desired results. This relationship is typically presented in a table format, like the one below:

Initiative objective

If successful, what will the initiative achieve?

Initiative hypothesis

What is the logic that underpins the initiative? Why do we think it will work?

Inputs

What resources are available?

Audience

Who do you most need to reach or persuade?

Actions / activities

What PD activities will you implement to reach that audience?

Outputs

What are the direct results (typically a count of what has happened)? E.g., number of social media engagements.

Short-term outcomes

What changes in audience attitude, skills, beliefs, or behaviors do you want to see immediately?

Long-term outcomes

What long-term, lasting changes do you want to see in audience attitudes, beliefs, and behaviors?

Assumptions

What do we believe to be true about our operating context and audience(s)?

Why complete a logic model?

Logic models are useful tools for several reasons:

• Logic models establish a common reference point for PD practitioners on how section activities will work, including important “if-then” relationships between section activity elements. They are a useful, high-level guide to ensure everyone on a team understands the big picture and keeps objectives front and center during the project’s roll-out.
• Logic models underpin key elements of monitoring plans. The content outlined in logic models informs what monitoring data to collect about section activities. In situations where an evaluation is appropriate, a logic model’s articulation of an initiative/section activity’s “how and why” helps evaluation teams to identify areas to investigate.
• Logic models help ensure the alignment of resources, actions and results and add clarity to planning by asking teams to refine their objectives, hypotheses, and assumptions.
• Logic models promote accountability by providing a checklist during implementation to ensure major parts of a section activity are being implemented as intended.
• Logic models enable clear communication with stakeholders by allowing you to tell the story of how a section activity will advance policy goals.
• Logic models can guide assessments and encourage learning. Outlining the major components of an intervention makes it easier to ask and answer questions like “Did we achieve the outcomes we anticipated?” “Did we reach our priority audience?” and “Were our planning assumptions correct?”

When to complete a logic model?

Completing a logic model is part of the detailed planning process.

The logic model is a useful tool to facilitate the detailed planning process, described in greater detail in PD in Practice. PD sections should complete a logic model for each new or ongoing activity or initiative the section decides to monitor. To begin this process, PD sections already should have conducted situation analysis, which includes desk research on the relevant topic and local conditions and identifying a priority audience (REFTEL 22 STATE 115522). Additionally, sections already should have created a problem statement that articulates the gap between or challenges faced in moving from the current (observed) environment and the future (desired) environment. With these elements in place, sections are ready to begin detailed planning and complete a logic model.

Who should complete a logic model?

The team directly responsible for implementing the section activity should complete the logic model, whether that is an implementing partner or a project lead within the PD section.

The section activity lead should consult with a diverse group of stakeholders to draft the logic model to:

(1) establish strong logical relationships (i.e., between inputs, audiences, activities and outputs, and between outputs and outcomes), and

(2) identify an initiative/activity’s relevant assumptions and external factors.

How to complete a logic model

When creating a logic model, start by discussing and agreeing upon the expected outcomes, so you can plan for the other components to lead to the desired long-term results.

Then, use the logic model template to record your team’s thinking in a systematic way.

Step 1: Develop short-term and long-term outcomes
Regardless of who directly implements the section activity, PD section leads should identify the expected outcomes and how they align with foreign policy priorities during the detailed planning process.

• Outcomes: Outcomes are the effects your efforts should have on your priority audience and align with the initiative or section activity goals.

  1. Short -term outcomes: Short- term outcomes are those that happen immediately after activities or actions.

     For example: A short-term outcome of an entrepreneurship workshop could be that participants develop a marketing strategy. A short-term outcome of a social media campaign may be increased awareness of post-implemented events and workshops among the social media audience.

  2. Long-term outcomes: Long-term outcomes are also effects or results, but they happen in sequence after and often build upon short-term outcomes. A section activity may build awareness in the short term, and this awareness could lead to a long-term outcome of an attitude change. This shift could take three weeks, or it could take three months, or even three years; the sequence is more important than the timeline.

     For example: A long-term outcome of the entrepreneurship workshop could be an increase in clients as a result of implementing a marketing strategy. A long-term outcome of the social media campaign could be that followers take action, such as participating in post events and workshops.

  Additional guidance on developing outcomes can be found in Appendix E: Developing Outcomes.

Step 2: Develop outputs

• Outputs: Outputs are the direct products or actions that result from the section activity, such as the number of workshops or participants. They are typically tangible and countable. Some common PD outputs are number of participants, number of likes on a social media post, or number of workshops held.

Step 3: Brainstorm activities

• Activities: Activities are the tangible actions you will take (e.g., design and release a social media campaign raising awareness on a potential trade agreement with the United States) or things you will do to reach your priority audiences and achieve your desired outputs and outcomes. List the activities you are planning to do. If you are writing a logic model for a section activity, you may break the section activity into discrete tasks or steps (e.g., create video for social media, create survey, book venue, book speaker, register participants). If you are writing a logic model for an initiative, you may list the various section activities you will be completing (e.g., social media campaign, curriculum development, workshop series, and a speaker series).

Step 4: Identify your audiences

• Audiences: The section activity audience is the group(s) of people you want to reach and affect through your policy-centered efforts. You should prioritize and segment your audiences before planning specific efforts, so you should have a clear idea of your audience before developing a logic model, and then record that audience in the model. For more guidance on how to identify and segment your audiences, see: Audience Analysis on the PD@State site. While many sectors and industries use logic models, the “Audience” column is unique to the PD logic model, and it highlights the fact that audiences are central to PD work.

Step 5: Develop a list of inputs

• Inputs: Write the resources invested to start and maintain the section activity or initiative. This column’s entries typically include staff time, funding, and facilities. Other inputs may include venue space, curriculum materials, literature, organizational partnerships, equipment, computer software, expertise, a survey, images, graphics, etc.

Step 6: Document assumptions

• Assumptions: Assumptions are things you believe to be true that inform why you believe a section activity will work. Planning assumptions are things that you assume will hold true for your plans to stay on track. For example: if you plan to do a radio campaign, you assume that your target audience owns a radio and listens to the radio. Or you could plan for a speakers’ series in February, based on the speaker’s availability and the country’s major holiday, but plan to shift the event to May if funding is not available in time.

Step 7: Review and refine your logic model

• Review: Finally, look over your entire logic model, and work to refine it to ensure that the connections from one component to the next are simple and clear. Section activities are built on underlying assumptions of cause and effect. Use the logic model to check these assumptions, by focusing on the “if-then” relationships within the logic model. Are there any gaps? Are there any areas where you have to rely on a lot of assumptions to proceed from the “if” to the “then”?

Resources for completing logic models

For more information on completing a logic model, please consult the following:

• Appendix C: Logic model template

• Appendix D: Completed logic model example

• Appendix E: Developing outcomes

Monitoring Plan

What is a monitoring plan?

A monitoring plan is a document that outlines how you plan to measure a section activity’s outputs and outcomes. It defines what progress and success look like by identifying indicators and tools for data collection. The monitoring plan includes a list of output and outcome indicators, the tools used to collect indicator data, performance targets for each indicator, and actual values for each indicator, obtained through data collection.

A monitoring plan is typically represented in a table like the one below:

What is the difference between a section activity or initiative monitoring plan and a grantee monitoring plan?

The monitoring plan included as part of the PD monitoring process should not be confused with a grantee monitoring plan. As described in the Federal Assistance Directive (FAD), a grantee monitoring plan asks PD sections to draft a plan that describes the proposed monitoring mechanisms (e.g., regular phone calls, site visits, expense sampling, reviewing reports, ) that they will take to ensure the grant is implemented as agreed. For more information on the developing grantee monitoring plans, please consult A/OPE/FA’s Sharepoint site.

Why complete a monitoring plan?

A monitoring plan is a document that outlines how you plan to measure a section activity’s outputs and outcomes. It defines what progress and success look like by identifying indicators and tools for data collection. The monitoring plan includes a list of output and outcome indicators, the tools used to collect indicator data, performance targets for each indicator, and actual values for each indicator, obtained through data collection.

A monitoring plan is typically represented in a table like the one below:

Step 1: Transfer Monitoring plan.

Step 1			Step 2	Step 3	Step 4
Activity	Category	Output/Outcome from logic model	Indicator	Data collection tool	Target	Actual

1. Copy and paste the activity from the logic model to the first column.

2. Write whether the category is an output, short- term outcome, or long- term outcome in the second column.

3. Copy and paste the text of the output or outcome into the third column.

Step 2: Draft Indicators.

[The State edition carries a graphic here, "undefined", that did not come through the public conversion.]

What are indicators?

We cannot measure performance without indicators. Indicators measure the progress or change of an output or outcome. Indicators describe what we can count or observe to represent the outputs and outcomes from your logic model, measuring actual results compared to the expected outcomes. Indicators are checkpoints or signals that tell us if we're on the right track and getting closer to (or further away from) our goals.

Indicators are measurements that help us understand how well a section activity's expected output or outcome is progressing. For example, if a school wants to improve student capability in math, then one indicator might be student test scores on a standardized test. Another indicator could be the number of students who request additional tutoring services.

Indicators describe both outputs and outcomes.

Output indicators measure the direct, concrete results of activities. Output indicators measure the outputs from the logic model and show how much and what kind of things are being produced or completed in an activity. Often these indicators are simple accountings of what has happened.

Output indicators are commonly written as:

• # of [audience] who [participated in/completed activity] within [time]; OR
• # of [activities] held/completed/organized within [time].

Examples of output indicators include:

• Number of participants completing a workshop by the end of the period of performance.
• Number of alumni network events held in one year.
• Number of likes or shares on a social media post within one month of posting.

Outcome indicators show change or progress towards outcomes from your logic model. These indicators typically include percentage change (either increase or decrease) in awareness, understanding, knowledge, skill, behavior, or attitude.

Outcome indicators are commonly written as:

• # or % of [audience] who [how output/outcome will be measured]

Examples of outcome indicators include:

• Quantitative: The percent of participants who report an increased awareness of sources of foreign malign influence in their country by the end of the workshop.
• Quantitative: The percent of participants who report that they plan on applying what they learned in the workshop when they write essays for study abroad in the United States.
• Qualitative: The type of professional resilience tactics participants learned by the end of the program.

Because of the long-term nature of outcomes, posts can choose from multiple types of outcome indicators.

• Short-term outcome indicators capture and measure changes that occur immediately following an activity, often focusing on changes in the knowledge, awareness, or understanding of the activity’s participants.
  • Examples of short-term outcome indicators include:
    • percent of training participants who report increased awareness of credible, fact-based news sources.
    • percent of network members who report they built new connections at a conference.
• Long-term outcome indicators describe the ultimate results of the activity you hope to achieve, often focusing on changes in skills, behaviors, or attitudes. They can take weeks, months, or even years to realize and may be observed outside the time frame of your program, but the sequence is most important. Long-term outcomes simply happen after short-term outcomes.
  • Examples of long-term outcome indicators include:
    • percent of training participants who increase consumption of credible, fact-based news sources.
    • percent of network members who collaborate on a science or entrepreneurship project with fellow STEM network members.

Second, differentiate between quantitative and qualitative indicators, acknowledging the importance of both numerical data and narrative perspectives.

• Quantitative indicators help us understand something using numbers and numerical values to describe things. For example, instead of saying, “Some people liked the event.” (which is qualitative), a quantitative indicator might tell us that “50% of the people who attended the event signed up to receive more information about the topic,” or “The program resulted in a 15% decrease in attendees who believed U.S. sanctions caused food shortages from the baseline survey.” Quantitative indicators might include things like the number of attendees, percent change (positive or negative), amount spent or received, or the frequency of an event or occurrence.
• Qualitative indicators help us understand something without using numbers. These focus on the qualities or characteristics of a situation, not the counting. For example, instead of saying "there are 10 people," or “positive perceptions increased by 15%” (which are quantitative), a qualitative indicator might describe what attendees said they learned from an event. Qualitative indicators might include narratives or stories, perceptions, descriptions, or ratings such as great, good, fair, poor.

How do I create indicators?

To create indicators, follow the these steps:

1. Select an output or outcome from the logic model and brainstorm what you want to measure.
2. Determine what you will count or observe and draft the indicator to describe that.
3. Use the SMART criteria to refine the indicators.
4. Enter the indicators into the rows of the monitoring plan template in the “Indicator” column.

Step A: Select an output or outcome from the logic model and brainstorm what you want to measure. Choose the output or outcome you want to measure. Brainstorm a list of things you could observe or count that would show that your selected outcome or output is (or is not) happening. If it is an output, what kinds of measurement or observations would capture a change in the audience?

Step B: Determine what you will count or observe, and draft the indicator to describe that. Draft an indicator that describes what you will count or observe.

Step C: Use the SMART criteria to refine the indicators. SMART is a useful acronym that can help you double check that your indicators are a good fit for your activity and can actually be collected.

SMART indicators

• SSpecific
• MMeasurable
• AAchievable
• RRelevant
• TTimebound

The letters in SMART stand for the following words and concepts:

• Specific. The target should cover only one concept; don’t use indicators that have two purposes or two components because doing so will make it impossible to disentangle the two.
  • Measurable. You must have the ability to collect useful data in an effort to monitor the indicator.
  • Achievable. Targets should be achievable, but ambitious. Finding the balance is something of an art. Targets that are both ambitious and achievable elicit excellent performance.
  • Relevant. The indicator should be related to the concrete change in conditions you hope to achieve.
  • Timebound. There should be a due date. A deadline ensures everyone is aware of what is required and we are able to hold key stakeholders accountable.

Review the following not-so-SMART and SMART-er indicator examples:

Step D: Enter the indicators into the monitoring plan template rows in the “Indicator” column. Once you have double-checked that the indicators will accurately help your team learn about whether the outputs and outcomes have been achieved, add it in the “Indicator” column in the monitoring plan template.

Step 3: Select a data collection tool

[The State edition carries a graphic here, "undefined", that did not come through the public conversion.]

Each indicator requires a data collection tool to track progress. Consider where and how you will collect data for each indicator. Potential data collection tools include surveys, interviews, focus groups, direct observation, and document review.

What are surveys?
Surveys help you learn about the knowledge, attitudes, beliefs, behaviors, or attributes of a (sometimes large) group of people.

Why use a survey?

• You want to know who your audience is (sex, age, etc.)
• You want to understand the knowledge, attitudes, beliefs, and behaviors of your audience.
• You want to track how these things change over time.
• You want qualitative feedback on a program.

Resources:

• The Basics of Consuming and Using Survey Data
• Survey Design Best Practices

What are interviews?
Interviews are one-on-one conversations between an interviewer and a respondent that primarily ask about firsthand knowledge, helping you learn more about an individual’s motivations, context, or perceptions, or language use without influence from others. Interviews give time for an individual to discuss topics, attitudes, perspectives, and experiences in depth. “Expert” or “key informant” interviews are those conducted with people who have specialized knowledge or subject matter expertise about a topic.

Why use an interview?

• You want to obtain information on a topic directly from people with relevant knowledge, expertise, or experience.
• You want to gain depth of understanding beyond what is possible in a survey.
• You want to understand motivations or reasons for attitudes and behaviors.
• You want to collect recommendations about program implementation.

Interview best practices

• Create an interview guide with a set of questions that matches research needs and the interview setting.
• Take advantage of moderator’s role to:
  • Probe in depth on individual questions;
  • Create a structured and planned out conversation.
• Take thorough notes, identify key moments or statements, review transcripts for further analysis.

What is a focus group?
Focus groups are semi-structured discussions with a small group of participants (usually 4-10), and led by a moderator to explore participants’ feelings, experiences, and preferences about a topic or issue. Semi-structured discussions have pre-planned questions and themes to broadly structure the conversation, but allow for unplanned questions based on what is heard. Focus groups differ from individual interviews because they encourage participants in the group to react/respond to each other.

Why use a focus group?

• You want to gain a deeper understanding of motivations, context, perceptions, or language from a small group of participants (usually 4-10).
• You want to understand how individuals discuss a topic with other(s).
• You want to gather a large amount of information in a short period of time.
• You want to hear about shared and differing perceptions and attitudes about a topic, initiative, program, or activity.
• You want to explore language use and how participants talk about a topic.

Focus group discussion best practices

• Create a discussion guide or protocol with:
  • open-ended questions on key themes;
  • probing questions and follow-ups.
• Take advantage of the moderator's role to encourage interaction.
• Take thorough notes, identify key moments or statements including points of agreement or disagreement, review transcripts for further analysis.

Resource: The Basics of Qualitative Research

What is observation?
Observations are a “watch and listen” exercise where observers record processes, events, and actions that they see and hear. There are different types of observation, including:

• Structured observation, which records information about a predetermined checklist;
• Participant observation, which entails engaging in project activities and recording what occurred, impressions, and reflections.

Why use observation?

• You want to study processes, actions, and events in their natural setting.
• You want to uncover patterns, conditions, or problems that may not be revealed through other methods of obtaining information from individuals.
• You want to understand the results of performance monitoring and why.

Observation Best Practices

• Identify in advance what you need to observe in order to obtain the information that answers the questions you have about your section activity:
  • Consider what people do and say, what they know, and what they make and use; and
  • How will you observe and record those things?
• Take detailed notes during and immediately after the observation; these “field notes” are raw data for analysis.

What is document review?

Document review involves collecting and reviewing program documents and administrative information to gain insights into your activity. Examples of documents you may review include: program applications, past evaluation reports, grant applications, alumni records, and documents produced by participants. For example, you might ask participants in an entrepreneurship program to complete business plans as part of their participation. Reviewing those plans may help you gain insights into how many people actually completed them or the quality and utility of those plans.

Why use a document review?

• You want to gather background information.
• You want to gain additional insight into a section activity or process.
• You want to uncover additional insights that may be valuable.
• You want to compare two or more sources of data in the same category to determine their similarities and differences.

Document review best practices

• Define the reason for the document review and outline your goals in advance.
• Establishing clear criteria for the review, including the review’s purpose, target audience, and any specific guidelines or standards that must be followed.
• Establish a list of relevant documents to review.
• Extract key information and organize it in notes.

Consider what tools for data collection would be most feasible to use to collect information from the participant or beneficiary, as well as who will analyze the data and how it will be analyzed and presented. Once you identify a data collection tool, add it in the “Data Collection Tool” column in the monitoring plan template.

Step 4: Identify a target

[The State edition carries a graphic here, "undefined", that did not come through the public conversion.]

Each indicator should have a target, which describes the expected value either at regular intervals during section activity implementation (quarterly targets for example) or at the end of the activity. After you decide how you will collect data about your indicator, you will set a target or goal for that indicator. How well do you think the measure or observation will do by the end of the activity? The target sets the level of performance you expect for a specific measure during a certain period of time. When the activity is complete, or at important milestones along the way, you will compare the targets to the actual results to see how well the activity is doing. Targets should be doable but still challenging to achieve. To set targets, it helps to look at the starting point, called baseline data. For activities that have been going on for a while, the baseline could be formed from past activity performance. How many people did you train or how much did it cost last time? For larger projects, you might collect new data to determine a baseline measurement. You might, for example, send a survey to participants before the program begins to measure their awareness on a topic. For new activities, the baseline for number of participants or activities is usually zero since they haven’t happened yet.

• Once you identify a target, add it to the “Target” column in the monitoring plan template.

Resources for completing monitoring plans

For more information on completing a monitoring plan, please consult the following resources:

• Appendix F: Monitoring plan template
• Appendix G: Completed monitoring plan example