Abstract
Population projections are increasingly utilised as tools for understanding and modelling the economic, social, and environmental futures of sparsely populated areas. This study examines issues around “fit for purpose” for the application of projections to remote contexts. We focus on projections made for the Northern Territory of Australia, a jurisdiction in the north of the country, to assess the relative accuracy of projections over time. The results conclusively demonstrate the reduced accuracy of remote population projections. Nevertheless, the exercise of comparing and contrasting the accuracy of projections provides a useful lens for understanding demographic and other issues which necessitate that approaches for developing and utilising projections can and should be different in sparsely populated areas. We provide examples of alternative approaches to projections and the analysis of errors which researchers and analysts in sparsely populated areas might apply to other jurisdictions.
1. Introduction
Population projections are becoming increasingly important as a resource for policy and program formulation and as tools for public and private decision making for investment and infrastructure creation [1]. In sparsely populated areas (SPAs or remote areas) of developed nations the demand for systematic understanding about likely future population growth and change is no less than for other jurisdictions, despite their share of national populations invariably being small [2, 3]. This reflects contemporary interest in Indigenous affairs, resource based industries for national economic production, and perceived opportunities for future national economic prosperity from “northern development” agendas; all are common themes in the political agenda for “northern” parts of developed nations [4].
The characteristics of populations residing in SPAs across the globe underscore that there are substantial difficulties in compiling projections [5, 6]. Specifically, a small stock of knowledge on the demographic composition of SPA populations in Canada, Australia, Sweden, Norway, Russia, Lapland, Finland, and Alaska has emerged to document relationships between historical demography, contemporary population characteristics, and the economies of SPAs. An interleaved theme is that the application of standard tools and techniques for understanding past, current, and future demographic change may provide limited perspectives [6, 7]. The reasons span the gamut of social, economic, and political influences but have recently been summarised in a conceptual framework titled the “8D”s of remote demography [6]. This lays out the key features of populations in SPAs which differentiate them from rural, regional, or periurban populations. An example is the “dynamic” nature of SPA populations which are subject on occasion to fundamental structural change (e.g., large swings in sex ratios) over a very short time period. These sometimes result from “black swan” events (e.g., extreme weather events), but sometimes major demographic change cannot be sourced to immediately identifiable causes [6].
Demographers and analysts who are charged with developing population projections aim to compile a series of numbers at one point in time that are the best approximation of the actual future population size and composition (usually according to age and sex distributions) for a given geographic area. Projections are made for several decades out from the initial year (called the jump-off year), for usually somewhere between 30 years and 100 years. The most commonly applied methods come under the label of “deterministic projections” whereby a set of assumptions about the starting population, migration, fertility, and mortality are parametised and the model determines the number of births, deaths, and additions to the population during each year based on predetermined algorithms. The most commonly used deterministic method is the cohort component model [8] which is widely used by governments for planning purposes.
All population projections are subject to a degree of inaccuracy from a combination of imperfect understanding about recent and future fertility, mortality, and migration behaviors in tandem with imperfect data for generating projection assumptions [9]. In demographic terms, errors in projections are measured as differences between the projected and eventual populations. For these reasons, demographers often provide information to help users determine the applicability and likely accuracy of projections—or “fit for purpose.” It is also common to produce a range of series of scenarios to demonstrate how variations in assumed fertility, mortality, or migration settings might affect population size and composition into the future (e.g., [10]). In the case of probabilistic projections, the likelihood of the eventual population falling within a (projected) range is provided in recognition that projections are an imperfect science, making the inclination for users to see them as “predictions” tenuous [11].
By contrast, users of projections continue to seek calculated and expert advice on what will actually happen to the population at a future point in time [8]. Targeted policy interventions for SPAs, particularly in relation to Indigenous populations, require intense processes of data collection, analysis, and projection. By en large, these processes aim to support politically entrenched ideologies around “evidence based” policy and program formulation as well as the evaluation of the effectiveness or otherwise of these. “Growing the North” is likewise a transnational ideology for many developed nations like Australia, in spite of some remote areas experiencing population decline in recent decades [12].
Research to date on the accuracy of projections for developed nations has almost exclusively focused on national or supranational regions where populations are relatively large (e.g., [9]). These and other empirical studies are conclusive insofar as three general principles have emerged: (1) accuracy diminishes according to the length of the projection period (or “horizon”) [13]; (2) accuracy is inversely related to population size [14]; and (3) the rate of growth during the base period on which projection assumptions are based influences the level of accuracy into the future [1, 15].
But only isolated examples of research into the accuracy of projections for populations at lower levels exist, and there are currently no studies to be found which are dedicated to the question of accuracy for SPAs, although some, like Smith’s [15] analysis of the accuracy of projections for around 3,000 USA counties, have incorporated SPAs into their overall findings. Despite a tacit awareness by governments and researchers about issues of accuracy in relation to population estimates and projections for SPAs, many examples of projections are available. The Alaska Department of Labour and Workforce Development [16] is archetypal in illustrating the tensions between the demand for and difficulties in producing accurate projections for SPAs. Their introduction to the projections warns that any modelling for populations of less than half a million is fraught with error, yet the same publication provides projections for populations of less than half this size.
In compiling projections for SPAs demographers face many additional challenges. Not least, the collection and production of demographic data for SPAs are more costly, time consuming, and difficult than for elsewhere resulting in reduced confidence in the level of accuracy and completeness of data [3, 17]. Even the register based systems used for population accounting in the Scandinavian nations are affected by remoteness [18, 19] and in the case of population estimates, standard errors for SPA populations can be exceedingly large [2]. On a more basic level, small populations in SPAs mean that small absolute differences between the projected and actual populations deliver high proportional errors. Tayman and colleagues [20] demonstrated this inverse relationship between accuracy and population size in their analysis of subcounty level projections for the USA.
Despite this context, projections for SPAs may still provide a rational basis for decision making [21] and, in the absence of other ways to assess future population trends, are certainly better than nothing. They are, for example, a resource for assessing the scale and direction of impacts from policy on the likely future behaviours of populations in relation to fertility and migration. Scenario modelling can build on this to help inform decision makers on investments for infrastructure and services. In the context of SPAs, projections can also provide preemptive warnings about environmental and other impending issues. They can be a basis for other projections and can contribute to improvements in the quality and accuracy of demographic data used as inputs to model their parameters and settings [21].
This paper addresses the gap in the contemporary literature on the relative accuracy of population projections for SPAs from which assessments about their applicability and “fit for purpose” can be made. Questions arise, in particular, about the relative scale and the timing of errors (e.g., how far into the projections do errors become noticeable for SPAs compared to elsewhere?) and whether systematic improvements in accuracy are observed over time. In this study I develop a range of indicators on the relative accuracy of population projections for an SPA in the north of Australia as a means of both addressing the absence of any such analysis for SPAs and demonstrating the value of alternative approaches to projections for SPAs. Other jurisdictions internationally will benefit from adopting and modifying this research to improve their understanding of projections accuracy and ways to improve it over time.
2. The Northern Territory of Australia
The Northern Territory (NT) of Australia (shown in Figure 1) is the least populous of all Australian states and territories with just two percent of the national population (233,000) but comprising 18% of the landmass [22]. Half the population lives in the capital city Darwin with the remainder residing in remote settlements. The NT has a significant Indigenous population constituting a third of the total. However, outside of Darwin over half the population is Indigenous and in many settlements the vast majority is so. The NTs population is Australia’s youngest, with a median age of just 31, some six years lower than the national median [10].
For as far back as records go the population of the NT has grown in a volatile fashion. Prior to the late 1970s, when it first maintained a population above 100,000, volatility was attributable to its small size. During WWII populations around the north of the NT grew and then declined rapidly from troop movements. Postwar policies to populate the north saw a doubling of the population in a short time after the war. Then in 1974 Darwin was effectively wiped from the map by Cyclone Tracy. Rebuilding and the establishment and growth in the NT public sector (after self-government in 1978) combined to generate population growth of around five percent per annum during the late 1970s and the early 1980s. A reduction in jobs in the late 1980s lowered growth but this turned around in the early 1990s from the bolstering of defence force capabilities in the north of Australia. Growth remained relatively low from the late 1990s to 2005 but after this and up until 2009 population change in the NT was consistently at or above two percent per annum. From 2010 to 2012 a hiatus in “mega projects,” requiring of large construction workforces, led the NT to record the lowest growth rate of all jurisdictions in Australia.
This simplified synopsis emphasises that a range of known and unknown factors impact on demographic research for SPA jurisdictions like the NT. Nevertheless, accurate assessments on current and future population compositions and numbers are critical since the NTs population size determines the number of political seats it holds in the House of Representatives chamber of the national parliament [2] as well as the allocation of national Goods and Services Tax transfers to the NT under national revenue transfers [24], upon which its economy relies heavily.
Given this context, understanding the size and scope of errors around population projections presents as important but unfulfilled research task. For places like the NT critical questions include the following.(i)How does the accuracy of official (in this case ABS) projections compare to the rest of Australia?(ii)What is the extent of bias towards underestimation or overestimation in projections?(iii)Does evidence exist for systematic improvements in the accuracy of official projections for NT over time?(iv)Do alternative approaches, like basic naïve modelling, locally developed projections, or other alternatives provide more accurate results?(v)What might the answers to these questions tell us about developing better projections for SPAs as a collective?
A number of practical outcomes may result from evaluations of projections for SPAs. Firstly, systematic errors in modelling processes or their underlying assumptions may be identified. As well as applying corrections to future forecast techniques, users can be advised of systematic errors and their impacts prevalent in the current set of projections. Secondly, assessments on projections errors assist users to understand the likely accuracy of new forecasts as they come to hand. Codified knowledge of past errors which are nonsystematic can help users determine the best application of such data to various tasks and scenarios, including which series (e.g., high, low, or middle series) to apply [3]. Thirdly, knowledge about errors in projections can assist modellers to calibrate time series models which rely on projections for predicting other demographic trends.
3. Materials and Methods
At any point in time there may exist around 40 “live” series of projections for the NT which are produced by a number of agencies and organisations. The majority are compiled by the Australian Bureau of Statistics (ABS), Australia’s national statistical organisation. In the analysis of the accuracy of these projections, I extend Wilson’s [9] work on the accuracy of population projections at the national level to compare and contrast the accuracy of ABS projections made for the NT. I also compare levels of accuracy for projections including a naïve model, the NT Department of Treasury and Finances NTPOP model (described in full in [25]) and projections compiled by the Australian Government Department of Health and Ageing [26]. Other than the naïve projection (which uses an exponential model holding the average annual growth rate experienced in the previous five years constant), all of these are cohort component models, which utilise the population accounting framework: where is population at time and is the population in a subsequent year, is births, is deaths, is internal in migration, is internal out migration, is international immigration, and is international emigration.
The NTPOP model is distinctive because it separately projects Indigenous and non-Indigenous populations as well as allowing for interactions between these cohorts (see [25]). The Health and Ageing projections were compiled for all Statistical Local Areas in Australia (the commonly used base unit for local demographic analysis) by the ABS on behalf of the Department of Health and Ageing [26].
ABS projections are sourced from their publication series 3222.0 (e.g., [27]) with each of the A, B, and C (and occasionally produced D) Series of projections (sometimes labeled “Series 1,” “Series 2,” or “Series 3”) from this collection for the base years (the starting year for the projections) of 1970, 1978, 1981, 1984, 1987, 1989, 1993, 1995, 1997, 1999, 2002, 2004, 2006, and 2012 incorporated in the analysis of errors. The middle series (generally the Series B) were used in the analysis of percentage errors. Accuracy was examined by comparing projected populations at the total NT level to the published estimated resident population (ERP—sourced from the ABS publication code 3101.0) at 30th June. The NTPOP and Health and Ageing projections base year was 2006, thus permitting only a preliminary assessment (seven years) of their relative accuracy.
A number of measures of the extent and direction of projection errors were employed. The degree of error is expressed as a percentage difference between projected and actual populations to give a measure of relative error. Several measures of relative error are available. Percentage error (PE) is most commonly used to evaluate forecasts of the total population. The PE is expressed as where denotes the year, population, forecast, and actual.
But several sets of forecasts may be in existence for any given year for which actual population estimates are available. In 2006, for example, there were 40 different projections of the NT population available. The mean percentage error (MPE) measurement is an indicator of the type (positive or negative) and extent of bias in a group of projections and is expressed as where is the number of forecasts.
The MPE is a measure in which the positive and negative values of errors may be offset. To examine absolute errors associated with a set of forecasts for the total population (ignoring the sign), mean absolute percentage error (MAPE) is used. It is expressed as where is the number of forecasts.
Assessments on whether the accuracy of projections for the NT is improving over time were made by comparing the accuracy of projections made within the same decade (1970s, 1980s, 1990s, and 2000s). To determine whether differences might represent systematic improvements or declines in projections accuracy or simply instability in the ERP figures, I conducted temporal autocorrelation analysis with lags of 1 to 8 years for each series.
This study is limited in scope to assessing errors against the total NT population as a baseline exploration of projections accuracy in the remote context. Wilson’s work in 2009 assessed errors for the components of change which, while outside the scope for this study, is a piece of research that should be tackled in the future. In terms of the validity of the measures applied to this study, some demographers have argued for better measures of accuracy. Tayman et al. [7], for example, proposed that MAPE tends to overestimate the extent of projection errors and have argued for more widespread use of other approaches. Nevertheless, the MAPE measure was considered as appropriate for this study primarily as a comparator to Wilson’s [25] findings on the accuracy of projections for Australia as a whole.
4. Results
4.1. Measures of Absolute Accuracy in the ABS Projections for the NT
4.1.1. Total Population
Figure 2 shows the projected total NT population figures from forty-six sets of ABS projections from the year 1976 to 2012 in comparison to the actual ERP figures. The data show ABS projections have tended towards a larger population than has eventuated in the population estimates and that scenarios which have incorporated high or low fertility assumptions (i.e., generally the A and C or D Series) have tended towards relatively large over and under projections. There are nevertheless many instances of the middle (B—blue in Figure 2) series exhibiting high absolute errors. For example, the projected 2010 population from the 1987 B Series was 289,400 when the ERP for 2010 was 229,700 (an absolute error of a quarter).
4.1.2. Accuracy by Years into the Projection
Results confirm there is a positive relationship between projections accuracy and years into projections. Figure 3 shows the percentage error (PE) for each year in the B Series ABS projections for the NT. Each column represents the percentage difference (±) between the projected population and the ERP for that year. For example, while the PE for the 1981 B Series underestimated the population by around 2.5% for the first few years, it rose to over 36% at the end of 30 years. PEs become relatively large within a short timeframe; for example, the middle series of ABS projections made during the 1980s tended to overestimate the eventual population by around ten percent within the first ten years.
4.2. The Relative Accuracy of NT and Australian Projections
Comparing the mean percentage error (MPE) for the NT with the MPE of Australia reveals the NT errors were substantially higher across all years of projections (Figure 4). The NT mean absolute percentage error (MAPE) was more than ten times the Australian MAPE by the first year into the forecast (at 1.69% compared to 0.16%) and then rapidly increased. After two years the NT MAPE is close to 2.29%, climbing to 6.46% after 10 years into the forecast and 8.72% after 15 years. The MPE for the NT shows the tendency for minor overestimations up to three years into the projections but relatively large overestimations by ten years into the forecast, while the MPE for Australia shows a small and consistent underestimation. Where it had occurred, the average level of overestimation twenty years into the projections was close to a quarter of the actual NT population.
4.3. Evidence for Improvements in Projections Accuracy
To assess whether over time population projections are becoming more or less accurate we compare and contrast the MAPE for the combined projections of each decade. Results show that 1970s and 1980s based projections exhibit similar, relatively high, and increasing MAPEs (Figure 5). However, MAPE errors for projections made in the 1990s appear to reduce further away from the base year. Nevertheless, the MAPEs for projections made during the 2000s are comparatively high and appear to be rising as subsequent years of ERPs become available. This is highlighted in Figure 6 which shows the first ten years of MAPE errors for respective decades of projections. From this it can be seen that 2000s based projections are showing similarly high levels of error to 1980s based projections.
Results from the temporal autocorrelation analysis suggest instability in the ERP figure for the NT may be evident (Figure 7). Specifically, this is seen in the “u” shaped (red) series in the left part of the chart showing the temporal correlation of ERP to the level of error (MAPE). The correlations suggest that temporal autocorrelation is variable according to the lag which is applied.
4.4. Comparisons of NT Projections Accuracy from Alternative Sources
The evaluation of ABS projections is now compared to alternative sources, including a simplified (naïve) model, NTPOP, and the Department of Health and Ageing projections. Figure 8 shows that there are significantly lower errors associated with naïve projections in comparison to the ABS MAPE. Unlike the ABS forecasts, for example, there is only a relatively small increase in the naïve model MAPE after ten years into the forecast and the 20-year MAPE is not substantially higher than the very early years of the forecast. Meanwhile the NTPOP and Health and Ageing projections track at relatively low error rates during the initial seven years.
Figure 9 looks at the first ten years of MAPEs for ABS alternative projections. Both the NTPOP and Department of Health and Ageing models exhibit relatively low errors, even when compared to the naïve projection. Encouragingly the NTPOP model appears to be quite accurate or at least is not subject to the upward trajectory of the ABS MAPE.
5. Discussion and Conclusions
In light of the demographic characteristics and demands for population projections for SPAs in developed nations, this study sought to assess the accuracy of population projections made for the Northern Territory of Australia. Several measures of accuracy were applied to ABS projections and these were compared and contrasted to projections from other sources. The research questions included the following.(i)How accurate have ABS projections for the NT been in comparison to Australia?(ii)Is there bias in the errors towards underestimation or overestimation?(iii)Has the relative accuracy of projections for the NT improved over time?(iv)Would a simplified projections technique (a naïve model) or projections from other sources produce more accurate results?(v)What might the answers to these questions tell us about developing better projections for SPAs more broadly?
The analysis shows high errors for NT ABS projections compared to Australia occurring across the lifespan of projections. In line with international findings on projections errors the magnitude of error for the NT increases during the progression into the forecast years, such that the MAPE error increases substantially after only a few years. In terms of bias, the middle series of ABS projections over four decades show a significant overestimation of the eventual NT population in projections modelled up to the 1989 edition when the NT population was under-projected right up to the most current ABS release for 2006 (with the exclusion of the 1997 middle series). In general, just 20 years into the projections the magnitude of overestimation for the total NT population was around a quarter of the eventual population.
Interdecade comparisons of errors delivered mixed results about whether projections accuracies are improving. MAPE and MPE errors for projections based in the 1990s were relatively low by historical comparison for the NT but indications for 2000s based projections are of a return to higher error rates. This raises the question of whether and to what extent temporal instability in the ERP (and perhaps specifically for the 1990s) might exist as an influence on interdecade shifts in accuracy rates. Part of the issue may lay in changing methods over time upon which the ABS bases its calculations of ERP, although these are generally accounted for by backcasting of the ERP according to the current method and through the release of historical ERPs. Empirical testing of temporal autocorrelations between the ERP and error levels conducted in this study suggests that there may be indeed some temporal ERP instability such that, even if improvements in error rates had occurred over successive decades, this may not, in the case of the NT, indicate systematic improvements in projections. While such instability may be smoothed out in large population estimates, in small populations this underscores the issues of applying demographic techniques focused on accurate national and supranational population estimation and projections processes. Equally, this supports that the development of alternative approaches to projections for SPAs is defensible and provides a forewarning to users of projections for SPAs about acceptance of projections from national statistical organisations.
The extrapolation of growth rates using a naïve projections method produced much lower errors across all years of forecasts and for all base years. The relatively low MAPEs for the alternative models (NTPOP and Health and Ageing), and particularly for the NTPOP model, are encouraging, although the measure of MAPE is comprised of just a few data points (seven years into the projections). Consequently, this study confirms ABS population projections for the NT have been subject to significantly higher errors than for Australia and that other forms of modelling would have provided more accurate numbers. Nevertheless, the aim is not to lambast the ABS which, despite persistent budget cutbacks in recent decades, maintains their national statistical agency role with dedication, expertise, and professionalism. Rather, the results here demonstrate SPAs inevitably will face higher levels of errors in projections with this study being the first to document the scale and changing directions of these.
Meanwhile the scale and changing direction of PEs demonstrate one of the fundamental weaknesses of deterministic projections, such as the cohort component model, in the context of SPAs. These essentially apply averages from several years of historical data (e.g., averages of net interstate migration) as the basis for assumptions on future directions for each component of change. Hence, projections compiled during periods of population decline, stagnation, or low growth have tended to underestimate future population numbers while the reverse is the case for projections compiled during periods of relatively high growth. Projections for the NT made during the early 1980s are a prime example, inherently assuming very high rates of population growth experienced in the late 1970s would continue, while the actual growth rate plummeted during the 1980s. This is entirely consistent with international literature on projections showing that errors are closely associated with population trends in the base years [15]. It is feasible that the population size of SPA jurisdictions like the NT can reach a point where swings in the parameters for projections as well as temporal variations in the ERP are smoothed out through scaled effects, although assessing when this might occur is not a simple task.
Results in this study provide a sense of the issues confronting the ABS and others in delivering accurate assessments on the size and composition of the Territory’s population and consequently in “doing” projections for SPAs. Such challenges are by no means unique to Australia, with national statistical organisations in Canada, Alaska, and the Nordic Circle facing the same issues. This study’s findings also challenge policy makers, demographers, planners, and users of population information in SPAs across the globe to work collaboratively in trying and testing alternative methods for projections. NTPOP is a prime example insofar as, although it is a cohort component model like those of the ABS, it separately projects Indigenous and non-Indigenous people and allows for demographic interactions to occur between these cohorts. There are other modelling methods that are gaining traction also. In the NT a project has just commenced which examines the capacity for agent based modelling to shed light on the demographic futures of small settlements [28]. Microsimulation is being used by Statistics Canada for projections of remote area populations, and Wilson and others have done much work on probabilistic projections as an alternative to cohort component modelling (e.g., [29]).
The inaccuracy of complex projections, into which a lot of work and money are invested, might suggest that these are not worthwhile or that somehow the whole exercise is fundamentally flawed. Instead, it must be recognised that the ABS works in conjunction with State and Territory governments, as well as the research community, to perennially improve the assumptions and parameters for projections which, by necessity, are part of a hierarchical process from the national level down. Nevertheless, in spite of these efforts, there is no clear evidence from the research in this study to indicate systematic improvements in the accuracy of projections for the NT over time.
In light of this study, the advice to users of population projections for sparsely populated areas is to not take the projected numbers as given or at face value. Instead it is important that user communities work with the custodians of projections and invest time into developing an understanding of the assumptions behind the numbers to build knowledge about their limitations and fit for purpose. In turn, remote communities should be actively engaged in helping to identify, from their perspective, the factors likely to influence demographic change into the future. Too often demographers work within the confines of codified data sets and focus on intricacies and shapes for statistically generated curves as likely representations of reality. Given the results here there is clearly room for considering whether and how information “from within” might be incorporated into the development of projections for SPAs. Hence, while the study of errors in projections is an endorsement of the inestimable benefits of hindsight, it is also a reminder that population projections for SPAs areas must not be taken as predictions. Their real value lies in the role which they can play in stimulating knowledge, improving the range and quality of data stocks, and generating dialogue and networks, all of which are likely to deliver long term improvements in the accuracy of the projections themselves.
Conflict of Interests
The author declares that there is no conflict of interests regarding the publication of this paper.