BEHIND THE DATA
BEHIND THE DATABEHIND THE DATA | FP2020 THE WAY AHEAD 2016-2017 /en/measurement-section/behind-the-data
The data presented in this report reflect methodological choices which we believe yield the most accurate and relevant information for tracking progress toward FP2020 goals. As a time-bound initiative with an urgent goal, we measure progress from the 2012 London Summit until now, taking into account all available and serviceable data. Using modeling, we produce annual estimates of critical indicators and we re-estimate the trend of additional contraceptive users on an ongoing basis. This section provides more detail on the methodology behind the data in an effort to increase understanding, promote transparency, and support mutual accountability.
TIME PERIODS COVERED IN THIS REPORT
The estimates presented in this report measure annual progress, and for Indicators 1–8, represent the value as of the mid-point of each year (e.g., the 2017 estimates for Indicators 1 and 2 show additional users and MCPR as of July 2017). The baseline year of 2012 is presented as the mid-point of 2012, or July 2012, when the London Summit took place. This 2017 Progress Report marks year five of the FP2020 initiative.
FAMILY PLANNING ESTIMATION TOOL (FPET)
The Family Planning Estimation Tool (FPET) is a Bayesian hierarchical statistical model that produces annual estimates of MCPR, unmet need, and demand satisfied. Traditionally countries have relied on estimates for MCPR and unmet need that are taken from population-based surveys, such as the Demographic and Health Survey (DHS). However, most countries do not conduct such surveys annually. In addition, although routine family planning service statistics and/or data on contraceptive commodities distributed are available in most countries through Health Management Information Systems (HMIS), they tend to not be used to monitor progress or make decisions at a program level.
FPET incorporates all available historical survey data for a country as well as service statistics (where determined to be of sufficient quality) to produce estimates of contraceptive prevalence and unmet need, which are in turn used to calculate demand satisfied. By using all available data, and regional and global patterns of change, FPET is producing a better estimate of current levels of MCPR, unmet need, and demand satisfied for each FP2020 country than has been traditionally available for assessing changes in family planning.
THE ROLLING BASELINE AND RE-ESTIMATING THE ENTIRE TREND
The methodology we use to estimate the number of additional users of modern methods of contraception has two important components, both of which confer advantages related to data quality and accuracy. The first is the designation of 2012 as the baseline year or starting point for our calculation—the point at which we set the number of additional users at zero. For each reporting period, we compare the total number of users in the current year to the total number of users in the baseline year (2012). The difference between the two totals is the number of additional users.
The second component is the use of a “rolling” baseline, meaning we recalculate annual estimates (starting with 2012) on an ongoing basis as new data become available. Continuously incorporating new data improves our ability to monitor progress, so that by 2020 our estimates for all years (2012 to 2020) will represent the most comprehensive and accurate data available. Calculations of the number of additional users depend on MCPR and the population of women of reproductive age (WRA). There is often a lag time of a year, and sometimes longer, before the surveys used to calculate MCPR are released. In addition, updated population estimates (including WRA) often include retrospective modifications of past estimates based on newly released census data and other sources.
CHANGE TO RATE MODEL
Estimates for previous years were based on the ‘level’ model which estimates historical trends by fitting curves to data on the level of MCPR. This approach smooths fluctuations from survey to survey and identifies the long-term trend, but it is relatively insensitive to short-term changes. Since we want to know whether trends have changed for the better since the start of FP2020, the ‘level’ model may under-estimate true progress. This year we changed to a ‘rate’ model, which fits trend curves to rates of change in MCPR rather than levels of MCPR. While this approach risks giving too much weight to spurious changes, it should be better at identifying recent changes that are the result of real progress. The effect of this change on current year estimates is small, but projections to 2020 should be more sensitive to recent progress.
Consequently, as new data become available, they affect not only current year estimates but those calculated in previous years as well. The advantage of using rolling estimates is seen by comparing the estimate of the number of users of modern contraception that was calculated for the London Summit on Family Planning in 2012 (258 million) to the updated estimate for 2012 that we use now (271 million).
Our revised baseline calculation incorporates new surveys that give us a better sense of the current MCPR in a country as well as what the MCPR was in 2012. These MCPR estimates are combined with UN Population Division population estimates to produce estimates of the number of users of modern contraceptives for the current year and for previous years. As a result, we now consider the total number of contraceptive users in 2012 to be 13 million more than originally estimated in 2012. Were we to use the old estimate for 2012, this discrepancy could be misconstrued as 13 million additional users on top of the actual 38.8 million additional users.
Not only is our 2012 estimate updated, but so are our 2013–2016 estimates. This means that the number of additional users that we estimated for these years in our last report has also been re-estimated. Because of these changes, it is important not to compare numbers in this report to numbers in previous reports. Instead, this report publishes the entire 2012 to 2017 trend based on the most recent data, enabling comparison of changes over time.
Similarly, next year the 2018 Progress Report will feature recalculated estimates for the entire 2012–2018 period. These estimates will incorporate new survey data, as well as revised population estimates from the UN Population Division, which were published in June 2017 but could not be incorporated into this year’s estimates due to FP2020’s annual measurement and reporting timeline (see Figure 6: FP2020 annual measurement and reporting process).
More information on the methodology for the rolling baseline can be found on the Track20 website. 18
New data from surveys and service statistics become available over the course of the year, and 18 countries have new data available since last year’s report. Due to variations in data sources, the strength and “recency” (how old the data are) of the estimates differ from indicator to indicator and country to country. The most recent data for each country ranges from 2002 to 2017 and is classified accordingly in the estimate tables: “old” (before 2012), “recent” (2012–2015.5) and “new” (2016 to the present).
USING SERVICE STATISTICS TO IMPROVE ESTIMATES
Track20 uses service statistics to inform MCPR trend estimates for countries where these data meet the following criteria:
- consistent levels of reporting over time (with at least 60% of facilities reporting data), so that changes in the volume of service statistics do not represent more facilities reporting, rather than an increase in services delivered;
- at least three years of consistent data, with at least one year overlapping with a survey so that the model can calibrate the two trends; and
- at least one year of service statistics reported after the most recent survey; if a survey is the most recent data point, the survey will be used to inform the MCPR trend.
In 2017, MCPR estimates were informed by service statistics for 13 countries.
Technical Brief: Rolling Baselines track20.org/download/pdf/Track20%20Technical%20Briefs/english/Technical%20Brief_Rolling%20Baseline%20(2015.03.13).pdf