Abstract
The stated preference experimental design can affect the reliability of the parameters estimation in discrete choice model. Some scholars have proposed some new experimental designs, such as D-efficient, Bayesian D-efficient. But insufficient empirical research has been conducted on the effectiveness of these new designs and there has been little comparative analysis of the new designs against the traditional designs. In this paper, a new metro connecting Chengdu and its satellite cities is taken as the research subject to demonstrate the validity of the D-efficient and Bayesian D-efficient design. Comparisons between these new designs and orthogonal design were made by the fit of model and standard deviation of parameters estimation; then the best model result is obtained to analyze the travel choice behavior. The results indicate that Bayesian D-efficient design works better than D-efficient design. Some of the variables can affect significantly the choice behavior of people, including the waiting time and arrival time. The D-efficient and Bayesian D-efficient design for MNL can acquire reliability result in ML model, but the ML model cannot develop the theory advantages of these two designs. Finally, the metro can handle over 40% passengers flow if the metro will be operated in the future.
1. Introduction
The stated preference has become the primary way of acquiring travelers’ preference data regarding different transport services. In the survey process, respondents choose their preferred travel mode in some hypothetical scenarios. The designs of these hypothetical scenarios depend on the experimental design. Thus, the stated preference questionnaire design is based on the experimental design method [1]. Currently, the widely used experimental designs for stated preference questionnaire are the random fractional factorial design and orthogonal design [2–4]. Although random fractional factorial design can effectively reduce the number of combinations, the method is likely to produce statistically inefficient or suboptimal designs. To overcome the deficiencies associated with the random fractional factorial design, the orthogonal design was introduced into the stated preference questionnaire design. The orthogonal design was aimed at ensuring independence among the attributes to avoid parameter estimation bias, which tends to result from multicollinearity [5]. However, a major shortcoming of the orthogonal design exists: due to the restriction of orthogonality among these attributes, the orthogonal design cannot determine the number of experimental combinations according to the research need. If too many alternative attributes and attribute levels are selected, the number of experimental combinations obtained by the orthogonal design becomes very large. Moreover, some scholars have questioned the effect of the reliability of estimating the model parameters based on the orthogonal design [6]. Some researchers have recognized that the experimental design would significantly impact the parameter estimation. They construct theoretical basis for some experimental designs in order to improve the reliably of parameter estimation results [7–9]. The most common ones are the D-efficient and Bayesian D-efficient design. Both designs have the same theoretical basis. The only difference between the two designs is determined by whether the values of the prior parameters are fixed. The effectiveness of these new experimental designs was demonstrated by simulations data, but little empirical research has been performed in actual environments. Bliemer validated the theoretical advantages of the Bayesian D-efficient design in an actual environment, showing that this design can be effectively applied in practice [10]. Louviere believed that the D-efficient design may cause significant errors in the estimation of parameter variance [11]. This notion is pending validation in follow-up studies. Although there has been few comparatives analyses of the traditional designs [4, 12], there has been little comparative analysis of the new designs against the traditional designs. Although insufficient empirical research has been conducted on the effectiveness of these new designs, a few researchers have started using these new experimental designs in relevant studies [13, 14].
There are some limitations in the existing papers on experimental investigations of the stated preference experimental design. In these works, the predominantly used model is the multinomial logit (MNL) model [8, 10]. However, the MNL model exists with a significant defect. That is, the traveler preferences heterogeneity and the correlation in attributes are not considered. However, in the experimental combinations obtained by the D-efficient and Bayesian D-efficient design, there exist certain correlations in the alternative attributes, and the Bayesian D-efficient design even considers the differences in individual preferences. There lacked adequate studies about whether these designs which were generated for the MNL model could work well in the advanced models, for example, the mixed logit (ML) model. The ML model takes into account the traveler preference heterogeneity and the correlation in attributes. Some researchers illustrated this problem through simulation data. They noted that the design generated specifically for the MNL model tends to perform well when more advanced models are used [15]. However, whether this conclusion holds with actual data still requires further validation.
Therefore, in this paper, the new metro between the city of Chengdu and one of its satellite cities is taken as the research subject. From an empirical point of view, comparative analysis of the new designs against the orthogonal design was made for an estimation of the model parameters; the validities of the D-efficient and Bayesian D-efficient design are empirically verified under different sample sizes. We would demonstrate whether these designs generated specially for the MNL model could work well in ML model based on the actual data.
2. The Experimental Design for the Stated Preference Questionnaire
2.1. Orthogonal Design
As a traditional stated preference experimental design, orthogonal design has been widely used in discrete choice model. The orthogonal design can reduce the number of combinations. By selecting mutually orthogonal combinations, the orthogonal design avoids the model parameter estimation errors that are caused by the correlation among the attributes [5]. The combinations that are independent, nonrepeatable, and representative were taken as the questionnaire. For example, suppose there are two available alternatives, A and B, for the travelers and that each alternative contains the attributes and L (e.g., waiting time and cost). Although the attributes of each alternative are the same, the level of each corresponding attribute is different. According to the principle of orthogonal design, all of the orthogonal and nonrepeated combinations are selected as the questionnaire content. Each column is orthogonal with respect to the other columns in the combination that is ultimately obtained; that is, , , , and However, a major shortcoming of orthogonal design exists: because the orthogonal design is restricted by the condition of orthogonality, it cannot select the number of combinations based on the research need. The number of combinations obtained by the orthogonal design is related to the number of alternatives and attribute levels. Hence, orthogonal design lacks flexibility when choosing the number of combinations.
2.2. D-Efficient Design
The theoretical basis of the D-efficient design is to obtain the minimum value of the determinant of the asymptotic-covariance matrix (AVC) of the model. The essence of this design resides in the goal of obtaining minimum values of the estimated standard deviations of the model parameters, thereby obtaining more reliable parameters estimation results. This design explicitly considers the importance of alternative attributes to ensure that the combinations can present more trade-off information in the selection process, with the goal of maximizing the respondents’ preference information [16]. Typically, the determinant of the AVC is expressed using D-error. The AVC represents the reliability of the model parameter estimation results. Therefore, the D-error value reflects the efficiency of the experimental design. Smaller values of D-error indicate higher reliability of the estimated parameter results.
According to the theoretical basis of the D-efficient design, when using this method, first, a prior AVC or prior parameter estimation results must be obtained to determine the experimental combination iteratively, using the minimum D-error value. Usually, we identify the previously obtained model AVC or model parameter estimate value as the prior information. How this prior information is obtained before the stated preference questionnaire is designed remains an issue that must be overcome when adopting this method. To overcome this issue, researchers collect data using a pilot survey. The pilot survey data are used to estimate the model parameter. The parameter estimation values are used as the prior information for questionnaire design. The mathematical derivation of the D-efficient design, based on related theory, is as follows: where is the AVC matrix for the discrete choice model and is the number of model parameters.
The AVC matrix is the inverse matrix of the Hessian matrix of the maximum likelihood function for the discrete choice model, as shown below:
The specific form of the Hessian matrix is given as follows: where each element in the Hessian matrix is calculated by the following equation: where is the maximum likelihood function for the discrete choice model and is the parameter value of the th attribute for the th alternative.
The maximum likelihood function for the discrete choice model is shown below: where represents the choice result of the respondents, and represents the choice probability of alternative . The formula used to calculate is as follows: where is the alternative set.
represents the utility function of alternative for traveler . It is calculated as follows: where represents the th attribute for the th alternative and is the unknown parameter in the utility function for the th attribute for the th alternative.
It can be observed that the D-efficient design is actually the inverse of the process that is used for parameter estimation in the discrete choice model. The theoretical bases for Bayesian D-efficient and D-efficient are identical. The only difference is that, in the Bayesian D-efficient design process, the prior values of the attribute parameters obey a certain distribution [17]. This type of experimental design takes into account the differences in travelers’ preference.
3. Method Validation and Empirical Analysis: A Case Study
In this paper, the new metro between Chengdu and Longquan (a satellite city) is used as the research subject. Based on the MNL model, the influence of different experimental design on model parameter estimation is verified. Additionally, whether the design generated specifically for the MNL model is able to obtain reliable parameter results in a more complex ML model is verified. At present, the existing modes of transportation between the cities include bus, taxis, private cars, and coach. With the continued development of the two cities, Chengdu is ready to introduce the metro to meet the diverse travel needs of its residents.
3.1. Design of the Stated Preference Questionnaire
The intention of this stated preference questionnaire is to investigate the travelers’ choice preference when introducing a subway system. Therefore, the alternative set in the stated preference questionnaire design process contains five alternatives, namely, the metro, bus, taxis, private cars, and coach. The alternative attributes include “arrival time (the time of arriving at the station),” “waiting time,” “cost,” “in-vehicle time,” and “off-vehicle time (travel time from getting off the vehicle to the destination).” Finally, the alternative attributes and attribute levels are formed as presented in Table 1.
3.1.1. Orthogonal Stated Preference Questionnaire
Orthogonal design was used to design the stated preference questionnaire. According to the alternative attributes and the number of attributes shown in Table 1, there are a total of 32 mutually orthogonal combinations which were obtained by Negene (a kind of experimental design software). The number of combinations obtained from the orthogonal design is related to the alternative attributes and the number of attribute levels. Therefore, this design cannot select the number of experimental combinations according to the research need. If we only used a part of combinations that maybe leads to a lack of orthogonality, the lack of orthogonality will cause the multicollinearity among attributes, thereby affecting the accuracy of the model results. Therefore, all 32 combinations were brought into the stated preference questionnaire. Using a block experimental design, under the premise of ensuring orthogonality, the 32 combinations were divided into 8 blocks. Each respondent attended to one block. Finally, the investigation instructions, filling instructions, attribute combinations, and personal attributes should be added; the complete orthogonal stated preference questionnaire was formed.
3.1.2. D-Efficient and Bayesian D-Efficient Stated Preference Questionnaires
The theoretical bases for the D-efficient and Bayesian D-efficient design are the same. The only difference lies in the form of prior parameter. The value of prior parameter is fixed in the D-efficient design. The prior values in the Bayesian D-efficient design are subject to normal distribution. As stated earlier, before the D-efficient design can be applied, the researcher must obtain prior information. The MNL model parameter estimation values obtained from the pilot survey data are set as the prior information for the D-efficient and Bayesian D-efficient design. The stated preference questionnaire used for the pilot survey also is designed by the orthogonal design. Using the pilot survey, 192 observations were obtained. The MNL model parameter values obtained from the pilot survey data are shown in Table 2. The prior values of the attribute parameters for D-efficient design are equal to the parameter estimation results in the MNL model. However, some differences exist in the process of setting the prior values of attribute for the Bayesian D-efficient design. The prior values for the significant attribute parameters are all subject to the normal distribution, and the nonattribute parameters are all subject to the normal distribution; the standard deviation is equal to the standard error of the parameter estimations.
According to the prior values for the parameters shown in Table 2, the stated preference questionnaire based on the D-efficient design was completed. For comparison with the orthogonal design, the number of combinations also was determined to be 32. Ultimately, the obtained D-error was 0.227. Similarly, according to the blocking experimental design, the 32 combinations were divided into eight blocks, and each respondent was assigned to one block. Finally, after incorporating the additional investigation instructions, filling instructions, attribute combinations, and personal attributes, a complete D-efficient stated preference questionnaire was formed.
The prior values of attribute parameter for the Bayesian D-efficient design are subject to normal distribution, but the values of the mean and variance are different. According to the prior values provided in Table 2, the number of combinations was also defined as 32. The corresponding D-error was 0.225. Similarly, using the blocking method, the 32 combinations were divided into eight blocks. The Bayesian D-efficient stated preference questionnaire was formed.
3.2. Data Analysis
In this paper, three types of questionnaires were utilized. Face-to-face survey was conducted. The preference data for travelers in different scenarios were obtained. Meanwhile, the respondents also were asked to provide personal information regarding their ages, income, and so forth. The overall survey sample size was 960. The gender rate between male and female is 1.03. The average age of overall samples is 36. The monthly average income of overall samples is 3556 (¥). By establishing four age intervals, each questionnaire was able to collect the age information from the surveyed sample. Of four groups, the 25–50 age range occupied the highest proportion. Similarly, five income ranges were established to evaluate the income levels of the respondents. The corresponding survey sample attributes of each questionnaire are shown in Table 3. According to Table 3, there are no significant differences between the general characteristics of the three survey samples, indicating that the general characteristics of the survey sample should not significantly influence the model parameter estimation results. Among the three groups, the rate of car ownership is relatively high and is a result of the fact that the rate of vehicle possession in Chengdu ranks first in the whole nation.
4. Analysis of the Results of Different Models
4.1. Error Term Variance of the Different Experimental Design Methods
First, we analyze whether a difference exists in the variance of the error term for the three types of experimental designs. The model result is shown in Table 4. In the model results, represents the ratio of the variance of the error term based on the orthogonal design over the variance of the error term based on the D-efficient design. represents the ratio of the variance of the error term based on the Bayesian design over the variance of the error term based on the D-efficient design. Similarly, represents the ratio of the variance of the error term based on the orthogonal design over the variance of the error term based on the Bayesian method. All three values are equal to 1; no significant difference is found between the three experimental designs with respect to the variance of error term. This conclusion also supports the conclusion made by Bliemer and Rose.
4.2. MNL Model Results Based on the Different Experimental Design Methods
Table 5 shows the MNL model parameter estimation results for different experimental designs. According to the results of the model parameters shown in Table 5, the goodness of fit for the model based on the Bayesian D-efficient design is 0.189, which is superior to that of the D-efficient design. The goodness of fit for the model based on the D-efficient design is 0.145, which is superior to the value of 0.14 obtained for the orthogonal design. Comparing the parameter estimation standard deviation (σ) values of the three experimental designs, it was found that the standard deviations of the model parameters based on the Bayesian D-efficient design were the smallest, followed by that of the D-efficient design. The estimated standard deviation of the model parameters based on the orthogonal design was the biggest. This suggests that both the D-efficient and Bayesian D-efficient design can achieve the theoretical goals in practice. According to the parameter estimation results, the D-error values of the three types of experimental designs were obtained, for which the orthogonal design gave a value of 1.06 × 10−4, the D-efficient design was 9.56 × 10−5, and the Bayesian D-efficient design was 8.22 × 10−5. By comparison with the D-error value, it can be observed that the reliability of the parameter estimates based on the Bayesian D-efficient design is higher than the D-efficient design, while the D-efficient design was higher than the orthogonal design.
The D-efficient and Bayesian D-efficient design can realize the theoretical goals in the real situation, namely, minimizing the standard deviations of the estimated model parameters. To further validate the use of these two experimental designs in actual environments, this paper compared the MNL model estimated parameter standard deviations based on the three experimental designs under different sample sizes, as shown in Figure 1.
(a)
(b)
(c)
(d)
(e)
From Figure 1, it can be observed that when using different sample sizes, the standard deviation of the estimated model parameters based on the Bayesian D-efficient design is the smallest, followed by the D-efficient design. The standard deviation of the parameters based on the orthogonal design is the biggest. These findings validate the use of the D-efficient and Bayesian D-efficient design in practice, which further verifies the conclusions made by Bliemer and Rose. The Bayesian D-efficient design can produce smaller standard deviations of parameter than D-efficient because Bayesian prior parameter is subject to certain distribution which means uncertainty by the analyst as to what the true population parameters will be empirically; the prior parameter is fixed which assumes the analyst has exact knowledge of the true population parameters when using the D-efficient design.
4.3. ML Model Results Based on the Different Experimental Design Methods
In this paper, the data are collected by the three stated preference questionnaires, which generated specifically for MNL model. The survey data was used to estimate the ML model parameters. The ML model results are shown in Table 6.
In the ML model “cost,” “in-vehicle time,” and “arrival time” are set as the random variables in the utility functions and are subject to the distribution. The model results obtained based on the three experimental designs represent the travelers’ preference differences in only one variable, as shown in the ML model results given in Table 6. The symbol of estimated parameter for the attribute “off-vehicle time” in the ML model results based on the orthogonal design is wrong, but this parameter was both correct and logical in the results given by the MNL model. The ML parameter estimation symbols based on the D-efficient and Bayesian design were correct. The chi-square test between the MNL and ML model based on the two experimental designs was obtained. The chi-square test values were 10.796 and 30.253, respectively (e.g., ). Both values were greater than the chi-square distribution critical value of 7.815 under the 5% confidence interval, indicating that the ML model results based on the D-efficient and Bayesian D-efficient design were better than MNL model. This indicates that these designs which were generated specifically for the MNL model may be not applicable to the ML model, for example, the orthogonal design. This conclusion is inconsistent with the conclusions made by Bliemer and Rose based on model simulation. They both concluded that the design generated specifically for MNL model tends to perform reasonably well when more advanced models are used. However, in this paper, it was verified based on the use of actual data that the orthogonal design may be not applicable to the complex ML model, while the D-efficient and Bayesian D-efficient design achieved accurate parameter estimation results in more complex ML model.
Based on the above analysis, the D-efficient and Bayesian D-efficient design also can achieve good parameter estimation results in the ML model. Comparing the standard deviations based on the two experimental methods in ML model, the conclusions drawn for the MNL model are not reflected. A degree of bias was present in some parameters, for example, the waiting time. To further investigate the validity of the theoretical advantages of these two experimental designs in different models under different sample sizes. The standard deviations of the estimated parameters between the MNL model and the ML model were compared based on the D-efficient design. As shown in Figure 2, it can be observed that the standard deviations of the ML model parameters were significantly higher than the MNL model. This shows that when using the D-efficient design, specifically for the MNL model, parameter estimation results also can be obtained accurately in the ML model; however, the theoretical advantages of the D-efficient design were not fully shown in the ML model.
(a)
(b)
(c)
(d)
(e)
4.4. Travel Choice Behaviors Analysis from Chengdu to Longquan
Through the above analysis, the reliability of the parameter estimation results obtained from the Bayesian D-efficient design was better. Therefore, the survey data obtained from the Bayesian D-efficient design could be chosen to calibrate the MNL model, in order to analyze the traveler behavior in the Chengdu-Longquan corridor. The model results are shown in Table 7.
The key factors affecting the traveler choice behavior in Chengdu-Longquan corridor include “cost,” “in-vehicle time,” and “arrival time.” Currently, the distance between Chengdu and Longquan is 23 kilometers, but the average in-vehicle time for the bus is relatively long (approximately 1 hours), and the in-vehicle time for the coach is 40 minutes. Therefore, neither of these travel modes is time efficient, making “in-vehicle time” a significant factor. In the current corridor status, the number of bus lines is limited. The distribution of the bus lines is not even. Additionally, the coach is not convenient because a traveler must go to certain stations to ride the coach. However, the Chengdu subway system has not been formed into a network, so there remain significant limitations associated with riding the metro. Therefore, the variable “arrival time” became a key factor on traveler choice behavior. In the model results, the alternative constant was very significant, showing that some factors that cannot be quantified also impact the traveler choice behavior.
The personal attributes of travelers also significantly impact their choice behavior. Men exhibit a weak preference for public transport, such as the bus and metro. Compared to public transport, men prefer the use of private cars. High-income groups tend to choose private cars and taxis. In particular, private car owners tend to travel with private cars, while low-income groups are more inclined to use public transport, which has relatively low costs.
Ultimately, according to the model results, it was predicted that the rate of sharing of the five alternatives in the corridor after the opening of the metro would be 0.48% for coach, 24.47% for bus, 1.64% for taxis, 28.69% for private cars, and 44.72% for the metro. The metro is expected to take on a significant amount of traffic demand, demonstrating the strong appeal of metro for travelers in this corridor.
5. Conclusion
The experimental design is a key factor affecting the reliability of the parameters estimation in discrete choice model. There are some new experimental designs, for example, D-efficient design, but insufficient empirical research has been conducted on the effectiveness of the new designs and a little comparative analysis of the new designs against the traditional design. In the paper three kinds of stated preference questionnaires can be designed based on the three types of experimental designs, respectively. The preference data of travelers can be achieved by these stated preference questionnaires. Based on the preference data we analyze whether a difference exists in the variance of the error term for the three types of experimental designs through the application of a NL model hierarchical division. We found there is no significant difference between the three experimental designs with respect to the variance of error term. This conclusion also supports the conclusion made by Bliemer and Rose.
The MNL model parameter estimation results based on the preference data were compared. According to the results, we can demonstrate the validity of D-efficient and Bayesian D-efficient design. Bayesian prior parameter is subject to certain distribution which means the analyst is uncertain about what the true parameters will be. The prior parameter is fixed which assumes the analyst has exact knowledge of the true parameters when using the D-efficient design, but the true population parameters are not known accurately. So the Bayesian design may contain the true population parameters which performs better than the D-efficient design.
According to ML model results, these designs which were generated specifically for the MNL model may be not applicable to the advanced ML model, for example, the orthogonal design. This conclusion is inconsistent with the conclusions made by Rose and Bliemer based on model simulation. Compared to the parameter estimation results of the MNL model based on the D-efficient design with the ML model under different sample sizes, we find the theoretical advantages of the D-efficient design were not fully shown in the ML model. But in this paper, we only used ML model; the other advanced model is not considered.
The preference data obtained from the Bayesian D-efficient design can be chosen to estimate the MNL model parameters. The traveler choice behavior in Chengdu-Longquan corridor was analyzed. These attributes, such as “cost,” “arrival time,” “in-vehicle time,” and “income,” can impact significantly the traveler choice behavior. These results reflect that there are some shortcomings of bus on convenience and timeliness. Finally, we predict the metro can handle over 44% passenger flow if the metro will be operated in the future.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This work was supported by National Science Foundation of China under Grant nos. 50908195 and 51178403, Specialized Research Fund for the Doctoral Program of Higher Education (no. 20130184110020) and the Fundamental Research Funds for the Central Universities (no. SWJTU11CX080 and no. 2682014CX130), Key Laboratory of Road and Traffic Engineering of the Ministry of Education, Tongji University (no. K201207), Program for New Century Excellent Talents in University (NCET-13-0977), The National Basic Research Program of China (973 Program no. 2012CB725405), and the Science and Technology Innovation Practice Program for Graduate Student, Southwest Jiaotong University (no. YC201407119), and Science & Technology Department of Sichuan Province (no. 2014RZ0037).