Abstract
This study aimed at analyzing the performance of four forecasting models in predicting the demand for medical care in terms of daily visits in an emergency department (ED) that handles high complexity cases, testing the influence of climatic and calendrical factors on demand behavior. We tested different mathematical models to forecast ED daily visits at Hospital de Clínicas de Porto Alegre (HCPA), which is a tertiary care teaching hospital located in Southern Brazil. Model accuracy was evaluated using mean absolute percentage error (MAPE), considering forecasting horizons of 1, 7, 14, 21, and 30 days. The demand time series was stratified according to patient classification using the Manchester Triage System’s (MTS) criteria. Models tested were the simple seasonal exponential smoothing (SS), seasonal multiplicative Holt-Winters (SMHW), seasonal autoregressive integrated moving average (SARIMA), and multivariate autoregressive integrated moving average (MSARIMA). Performance of models varied according to patient classification, such that SS was the best choice when all types of patients were jointly considered, and SARIMA was the most accurate for modeling demands of very urgent (VU) and urgent (U) patients. The MSARIMA models taking into account climatic factors did not improve the performance of the SARIMA models, independent of patient classification.
1. Introduction
Emergency department (ED) crowding results from mismatch between existing capacity and various input, throughput, and output factors. On such context, forecasting the demand may provide useful inputs for planning available resources [1]. ED demand prediction, expressed in terms of daily visits, has been assessed by different time-series approaches employed to develop forecasts models, with no supremacy of one method over others [2]. Studies compared several different methods for forecasting ED daily visits including ARIMA, SARIMA, multiple linear regression, time-series regression, and exponential smoothing [2–4].
Published literature assessing the impact of climate variables on ED demand (i.e., ED daily visits) provides conflicting results. While some studies indicate that climate variables such as mean daytime air temperature positively correlate with demand for ED services [4, 5], others show that climate variables add little predictive value to models of daily patient volumes [2, 6]. Temperature and calendar variables may provide different forecasting abilities, although in general forecasting accuracy of models including those variables tends to be better for the short-term horizon (7 days in advance) than for the longer term (30 days in advance) [2]. Several studies demonstrate that daily demand for ED is characterized by seasonal and weekly patterns [2, 4, 6, 7]. Daily patient’s volumes show a weekly seasonal distribution, especially on Mondays [4, 6], although published studies demonstrate little variation in daily visits by month [2, 7].
In such context, accurate prediction of ED services’ demands emerges as a powerful tool to support resources planning decisions [8, 9]. For example, long-term demand forecasts may be used to analyze infrastructure and personnel expansion plans [6], while short-term forecasts may support the operational planning of available resources on a daily basis [10]. The use of forecasting models to estimate the demand for ED services may improve the quality of the care offered to patients [9, 11]. ED managers may, for instance, identify the day in the week with the heaviest demand and align materials and personnel resources to attain better patient service levels [12]. The higher forecasting accuracy the better; therefore, factors that may influence demand in EDs should be considered as potential explanatory variables in the prediction models [13].
As reported above, studies have shown that demand forecasts in EDs may be improved by considering in the models factors such as day of the week and month and proximity to holidays [11, 12], in addition to climatic characteristics [2, 14, 15]; other studies, however, show the opposite [4, 16]. One explanation for that is the fact that each ED analyzed has its own characteristics. The effect of climate variables, for example, is dependent on the geographic location of the ED from which data were collected. In any case it is always advisable to test the influence of potential explanatory variables when developing an ED demand forecasting model [2]. Accurate forecasting according to categories of patient acuity (which indicates the degree in which healthcare services will be demanded to provided appropriate care to patients) may also provide a valuable tool for resources planning. In a study held in an ED in Singapore, attendances of lower acuity patients were significantly correlated with day of the week, month of the year, public holiday, and ambient air quality [14].
In this paper we analyze the performance of four forecasting models in predicting the demand for medical care according to patient classification using the Manchester Triage System (MTS) in the ED of a Brazilian University hospital.
2. Methods
2.1. Study Design and Setting
This is a retrospective study that uses historical data on medical care demands from an ED to develop and compare forecasting models able to predict future demands. The study was conducted at Hospital de Clínicas de Porto Alegre (HCPA), a 842 bed, tertiary care teaching public hospital in Southern Brazil. The HCPA’s Ethical Committee has approved the study and authors have complied with the recommendations of the Declaration of Helsinki. Data were extracted from the hospital’s management system. Patients’ personal data were preserved.
2.2. Selection of Participants
The study was carried out in the HCPA’s ED. The department operates 24 hours a day, 7 days a week, throughout the year. It receives patients from the city of Porto Alegre and its metropolitan region, corresponding to a population of approximately 2 million. Daily data on the number of patients who were admitted to the ED comprised the period starting on January 1, 2013, until May 31, 2015, including all patients processed between 00:00 and 24:00 hours. Medical care was provided to 57,128 individuals during that period.
Historical weather data were obtained from the Instituto Nacional de Meteorologia (INMET) [17], which monitors weather conditions 24 hours a day throughout the year in Brazil. We used weather information aggregated on a daily basis, to match with the dependent variable in the prediction models (i.e., ED daily demand). INMET methods comply with international standards defined by the World Meteorology Organization.
2.3. Study Protocol
Input data were the daily sum of patients that received care in the ED. Classification of patients that seek medical assistance at the ED follows the MTS criteria [18]. This system organizes care delivery into five levels of priority. Categories are identified by colors, case description, and the estimated time to service (ETS); they are (i) red: emergency (EM), needing immediate assistance in emergency service, ETS = 00 minutes; (ii) orange: very urgent (VU), need for care in emergency service, ETS ≤ 10 minutes; (iii) yellow: urgent (U), patient displaying clinical condition that enables waiting for care in emergency service, ETS ≤ 60 minutes; (iv) green: standard (ST), patient who may be directed to outpatient service such as ambulatory consultation, ETS ≤ 120 minutes; and (v) blue: nonurgent (NU), patient who should be directed to outpatient service such as ambulatory consultation, ETS ≤ 240 minutes. The color white is also used to indicate patients that were not classified and for which no case description or ETS is available.
The database of ED visits was divided into two periods, one for training, and the other for test. The first period, from January 1, 2013, to March 2, 2015, was used to analyze data and test forecasting models (training set). The second period, from March 3, 2015, to May 31, 2015, was used in the validation of models and for accuracy checking (postsample forecasting set). Approximately 94% of the patients triaged in the ED in the two periods were classified in categories VU or U. We analyzed series comprised of patients of all categories, in addition to individual series containing only patients in categories VU or U.
The postsample forecasting set was divided into three test intervals of 30 days: (i) 03/03/15 to 04/01/15, (ii) 04/02/15 to 05/01/15, and (iii) 05/02/15 to 05/31/15. Model accuracy was evaluated considering forecasting horizons of 1, 7, 14, 21, and 30 days. Forecasting models had their accuracy initially verified for the first interval of 30 days. Then, the first test interval was added to the training set, models were recalibrated considering the new data, and accuracy was verified for the second test interval. The same procedure was repeated when testing the third interval. The idea here was to continuously update the training set with more recent demand values.
2.4. Data Analysis
Considering that the superiority of a given forecasting model over others depends on the data under analysis [19, 20], we chose to evaluate the performance of several models and to search for explanatory variables that could influence the demand for care in the ED. We evaluated the performance of four forecasting models based on the analysis of time series of demand data. They comprised (a) simple seasonal exponential smoothing (SS), (b) seasonal multiplicative Holt-Winters (SMHW), (c) seasonal autoregressive integrated moving average (SARIMA), and (d) multivariate autoregressive integrated moving average (MSARIMA). The MSARIMA model was tested incorporating climatic and calendrical influencing factors as independent variables, as shown in Table 1. Models (a) to (c) use as input data only the time series of past demands, while model (d) tests the benefits of including climatic and calendrical factors on the prediction accuracy. All analyses were performed using PASW Statistics 18 and Minitab 14.
The first and second models tested are based on exponential smoothing coefficients. Such models are able to capture systematic variation in the time series due to seasonality and/or trend, as well as sudden changes in the demand pattern, which are common in ED data [4]. On that study, models that best fitted the data were the simple seasonal exponential smoothing (SS), and the seasonal multiplicative Holt-Winters (SMHW) [4]. SMHW presents two variations: the additive model form is preferred when seasonal variations are constant through the series, while the multiplicative model form is preferred when seasonal variations are changing proportional to the series’ level [7, 21, 22]. In our work, multiplicative form displayed a better fit to the data, with smaller associated prediction errors.
SARIMA, an extension of the traditional ARIMA (autoregressive integrated moving average) model that accounts for seasonal components, was the third method analyzed. The model captures the behavior of demand variables using historical data from the time series, being the one most widely used in healthcare-related forecasts [2, 15, 23]. The SARIMA model is described by seven parameters: , where represents the order of the autoregressive factor (AR), represents the order of differentiation required to reduce nonstationarity in the data (I), and represents the order of the moving-average model (MA). Parameters , , and are analogous to , , and , however describing the seasonal portion of the model; finally, represents the seasonal lag [24]. Definition of the best values for the model parameters may be carried out analyzing the autocorrelation function (ACF) and the partial autocorrelation function (PACF) plots. The models that offered the best fit to the training time series of historical medical care demands in the ED were when all patients were considered, considering only VU category patients, and considering only U category patients.
An extension of SARIMA, the MSARIMA model, was also tested to predict ED patient visits. MSARIMA incorporates independent (explanatory) variables to SARIMA in search of a better characterization of the demand time series [14, 15]. In our study ED demand was used as dependent variable and climatic and calendrical factors as independent variables. In addition to the climate variables presented in Table 1, we tested the influence of those same variables in a 1- to 7-day lag. Thus, a total of 64 climate variables and 2 calendrical variables were tested as independent variables in the MSARIMA model; only those displaying values ≤ 0.05 were kept.
Accuracy of the prediction models was measured comparing actual and predicted values during the test period. For that, we calculated the mean absolute prediction error (MAPE) for each forecasting horizon (1, 7, 14, 21, and 30 days). Being a scale-independent measure, MAPE allows comparison of results from different models applied to different time series. The statistic represents the average of absolute differences between predicted and observed values, given as a percentage; a small MAPE value indicates a model well fitted to data. Consider a series of predicted values () and the corresponding observed values (); the MAPE statistic is calculated as follows:
3. Results
During the interval considered in this study 57,128 patients sought medical care in the ED, being 51,046 during the 755 days that comprise the training period and 6,082 during the 90 days that comprise the test period. On average, 68 patients sought medical care daily, ranging from 26 to 119. During the same interval the average temperature was 21.08°C, varying from 0°C to 40.6°C, with an average daily range of 9.68°C.
The scatter plot of the daily demand for medical care in the ED (Figure 1) displays a change in the series level around day 165 (July 14, 2013). That corresponds to the beginning of a new mechanism, entitled “right patient in the right place,” for referral of patients with minor acute illnesses (low acuity) to retail clinics and urgent care centers. The protocol was implemented as a partnership between the hospital and Porto Alegre’s Health Department. After the new protocol was adopted the demand time series remained stationary, with no visible trends.
Figure 2 presents demand variations within the week (Figure 2(a)) and the year (Figure 2(b)) in the form of box-plots. One-way ANOVAs were used to detect differences in weekdays and months, yielding the results in Figure 2. Demand is at its peak on Monday, followed by decrease until Wednesday; on Thursday it rises again, decreasing steadily until Sunday. However, statistically significant differences could only be found between weekdays and weekends.
(a)
(b)
To analyze variations in ED demands throughout the year (Figure 2(b)) we used the data series starting in August 2013, when the series mean stabilized after introducing the new mechanism for referral of patients with minor acute illnesses to retail clinics and urgent care centers. We also removed from the analysis the month of October 2014, when maintenance carried out in the department’s air conditioning system led to an atypical reduction in demand during a period of 15 consecutive days. There were no differences in demand throughout the months, except for October, assigning to it the fact that only 2013 data were used to analyze this month. We also checked for differences in demand in holidays and regular weekdays; no differences were found.
Table 2 gives each model’s MAPE value, considering all patients (TOTAL), and those in the very urgent (VU) and urgent (U) categories, and 5 different forecasting horizons (1, 7, 14, 21, and 30 days). In general, the shorter the horizon, the more accurate the forecast. Models’ performances varied according to MTS patient classification, such that SS showed the best performance when all patients were considered, and SARIMA was the most accurate for modeling demands of high acuity (VU and U patients). Parameters of models presenting best fit to each patient category are given in Table 3. The MSARIMA models taking into account climatic factors did not improve the performance of the SARIMA models, independent of patient classification. In other words, the best MSARIMA models for every combination of patient category and horizon did not include climatic variables. When that is the case, MSARIMA reduces to SARIMA yielding identical MAPE values, and we chose SARIMA as best models. Table 4 gives the values for all climatic factors included in the models, for different time lags.
4. Discussion
In this study we analyzed the performance of four forecasting models in predicting the demand for medical care in an ED, testing the influence of climatic and calendrical factors on demand behavior. To the best of our knowledge, our study innovates by stratifying the demand time series according to patient acuity, according to the Manchester Triage System’s (MTS) classification. In opposition to other studies that investigate the use of forecasting techniques to predict demand for medical care in EDs [4, 15, 23], we analyzed data from an ED that only provides care to high complexity cases.
A study held in an ED in Singapore assessed forecasting models where patients where stratified into three acuity categories (i.e., P1, P2, and P3), with P1 being the most acute and P3 being the least acute. P1 attendances did not show any weekly or yearly periodicity and were only predicted by ambient air quality. P2 and total attendances showed weekly variation and were predicted by holidays. P3 attendances correlated with day of the week, month, holiday, and ambient air quality. Higher total attendances on Monday were contributed mainly by P2 and P3 cases, while higher attendances on Sunday were essentially P3 cases [14]. Although resembling our study in that acuity of patients was taken into account when building forecasting models, the use of a local acuity scale by Sun et al. [14] does not allow us to directly compare their results with ours. Moreover, it is clear, considering the number of patients belonging to each category, that cases in the study of Sun et al. were less complex than those consulting our ED.
Regarding model accuracy, SS yielded the smallest MAPE values, independent of the forecasting horizon chosen, when all MTS classes of patients were considered in the demand series. For a 1-day horizon, SS yielded a MAPE of 2.91%, deemed excellent if compared to results obtained by Jones et al. [4]; for all remaining horizons investigated, the model yielded an average MAPE of 11%, which is aligned with results reported in similar works [2, 4]. Despite their complexity, the ARIMA models displayed the poorest performance in modeling ED demands, when all classes of patients were considered. As reported by Marcilio et al. [2], Tandbgerg and Qualls [3], and Jones et al. [25], depending on the characteristics of the demand time series analyzed simpler models, may lead to more accurate results.
As expected, models that forecasted the ED demand from patients classified as VU and U displayed larger MAPE values, in view of the smaller number of patients in the series and the larger variability. For those classes of patients, the model with the best performance was the SARIMA, except when VU patient demands where predicted for a 7-day horizon; in that case, SHWM yielded the smallest error (see Table 2). For a 1-day forecasting horizon it was possible to obtain a MAPE value of 7.19% for VU patients, and 3.98% for U patients, which are deemed acceptable and useful to support next day’s managerial decisions in the ED. For larger forecasting horizons (7 to 30 days) MAPE values were larger, varying from 16.82% to 17.21% for VU patients and 12.79% to 15.21% for U patients. Considering that those patients are the ones that demand ED resources more intensely and frequently require inpatient beds, focusing the analysis on those classes of patients, as done here, is extremely relevant in practice. It is important to note that the results above were similar to those reported in other forecasting studies carried out using data from EDs that provide specialized and complex care [2, 14].
Regarding external information analyzed and incorporated to the models, only the day of the week showed a systematic effect on ED demand, independently of the patient classification considered. Day of the week has an effect on demand, while month of the year has not, as displayed in Figure 2. The effect of holidays and days following holidays on ED demand was also not significant. Demand peaks verified on Mondays, and demand valleys on weekend were aligned with results reported by several studies on ED demand available in the literature [2, 6, 14, 25]. There is a strong decrease in demand on weekends that could contribute to the optimization of ED resources and workforce management.
Some climatic factors displayed significant correlation with demand series from different classes of patients (see Table 4). However, when comparing the MAPE performance of MSARIMA models (that take such factors into account) and SARIMA models (that do not take them into account), the latter systematically outperformed the former. That may be due to the fact that the ED under analysis provides medical care to critical emergency situations in a teaching hospital, which seem to be less affected by climatic conditions.
Table 4 shows that VU and U classes displayed significant time lag terms for all lag instances. However, the same was not true for models considering the total number of visits. That is justified by the lack of identifiable pattern in the arrival of cases whose acuity classification leads to redirecting of patients to other types of care (i.e., all patient classes, except VU and U). Those are highly variable cases which, although representing a small percentage of the total number of visits, interfere in the prediction models’ structure.
It is known that using forecasting techniques to predict demand in EDs may contribute directly to the reduction both in the number of patients that leave the unit without being served and in the number of complaints made by patients [6]. In the present study it was observed that models fitted to the demand series that considers all classes of patients displayed smaller prediction errors than those fitted to classes of patients. In addition, it was verified that the best forecasting model when demand from all classes of patients is considered (SS) is different from the model type that best fitted demands from individual classes of patients (SARIMA). It is important to remark that SS models are easier to implement in practice and that its superior performance in modeling ED demand has been reported in the literature [7].
The comparison between forecasting methods presented here indicates that, in the high complexity ED investigated, SARIMA models provide the most accurate prediction of demand for medical care for patients classified as VU and U. In other words, the inclusion of climatic factors as independent variables in the prediction model does not increase its accuracy. In short, results in this study do not support the common belief that daily demand for ED medical care is influenced by climatic factors [2, 14, 15].
Finally, results of the present study shall be viewed in the perspective of tertiary teaching hospitals, where usually overcrowded EDs are substantially affected by frequent bed shortages and seasonal variation of the number of patients referred by medical specialties. On such context, short-term horizon forecasting models may provide valuable information not only to optimize patient flow, but also to establish interventions in order to maintain the smooth and timely flow of patients throughout hospital facilities.
5. Limitations
We investigated data from a single hospital located in a specific region of Brazil. The test period comprised 25 days of summer and the first period of winter and therefore may not reflect the full variations in climate. Thus, our results are valid for highest level EDs located in the same region or in regions with similar climatic characteristics. For more generalizable results, the study should be extended to regions with different climatic characteristics. Also, the study did not assess the frequency of patients left without seeing. The same recommendation applies to the sample of patients.
6. Conclusions
Our study confirms that daily demand for ED services is characterized by seasonal and weekly patterns. Furthermore, it indicates that time-series models can be used to provide accurate forecasts of high complexity ED patient visits, especially short-range forecasts horizons. Forecasting accuracy depends on the model employed, length of the time horizon, and classification of the patient at MTS. In our analyses, SS yielded the smallest MAPE values, independent of the forecasting horizon chosen, when all classes of patients were considered in the demand series. For VU and U classes of patients, the model with the best performance was the SARIMA, except when VU patient demands where predicted for a 7-day horizon; in that case, SHWM yielded the smallest error. Some climatic factors displayed significant correlation with demand series from different classes of patients but do not increase the accuracy of prediction when incorporated into the model.
Competing Interests
The authors declare that they have no competing interests.