Abstract
The evaluation of BIM capabilities and repeatability enables a company or project to identify its current status and how to improve continuously; this evaluation can be performed with BIM maturity models. However, these maturity models can measure the BIM state but not specifically the application of BIM uses. Likewise, in interorganizational project teams with a diversity of factors from various companies, it is possible to evaluate the capacity at a specified time with specified factors, but it is not possible to evaluate the repeatability unless the client always works with the same project teams. Therefore, despite the existence of various BIM uses in the literature, there is no instrument to evaluate the level of implementation of them in construction projects. This research proposes a BIM Use Assessment (BUA) tool for characterizing the levels of application of the BIM uses in the planning and design phases of building projects. The research methodology was organized into three stages: (1) identification, selection, and definition of BIM uses; (2) proposal of the BUA tool for characterizing the level of BIM use application; and (3) validation of the BUA tool. The tool was validated using 25 construction projects, where high reliability and concordance were observed; hence, the BUA tool complies with the consistency and concordance analysis for assessing uses in the design and planning phases of construction projects. The assessment will enable self-diagnosis, stakeholder qualification/selection, and industry benchmarking.
1. Introduction
Building information modeling (BIM) is becoming an essential methodology in the architecture, engineering, and construction (AEC) industry. Indeed, public agencies from several countries are encouraging the use of this methodology by issuing requirements, user guides, and manuals regarding its use [1]. There are several ways of using BIM (BIM uses), which lead to various benefits; as stated by Kreider and Messner [2], applications of BIM during the infrastructure lifecycle can enable the realization of one or more specific objectives. Therefore, it is important to understand the way that organizations apply BIM uses.
BIM uses are defined in various ways: the Penn State guide defines BIM use as “a method of applying Building Information Modeling during a facility’s life cycle to achieve one or more specific objectives” [2]. According to Succar, BIM uses “identify and collate the information, requirements that need to be delivered as—or embedded within—3D digital models” [3]. Another definition comes from the New York guide, which defines BIM uses as “the most common applications of BIM on the Department of Design and Construction Projects; BIM uses shall be considered and aligned with project goals” [4]. Thus, while there is a consensus on the relationship of uses to project objectives, there is no agreement on whether uses are methods, applications, or actions; therefore, there is no universal definition for BIM uses.
Although there is no universal definition, the definitions are aligned in that the BIM uses are present throughout the project lifecycle, as in each phase, it is possible to realize a specific benefit. The infrastructure lifecycle is defined as consisting of four phases: planning, design, construction, and operation [5, 6]. During all these phases, it is possible to apply various BIM uses according to the objectives that are established in the project, which increases the number of possible uses of this methodology [7]. The initial stages, namely, planning and design, are considered instrumental in the development of the project. According to MacLeamy’s curve [8], the planning and design phases have a significant impact on the entire project; early efforts in these phases can help to prevent cost overruns and time delays at the construction site [9].
To realize the optimal applications of BIM, it is important to be aware of the BIM use level; in this way, it is possible to identify the possibility of improving the BIM in the project. However, it is possible to use a tool that enables the characterization of the BIM level but, not each BIM use in a project. Moreover, measuring the BIM level is not the same as measuring the BIM uses. These tools are BIM maturity models because several BIM maturity models facilitate understanding how companies apply the BIM methodology, where the term “BIM maturity” refers to the quality, repeatability, and degree of excellence within the BIM capability [10]. These maturity models can measure the BIM state but not specifically the application of BIM uses. Likewise, it is possible to evaluate the capacity at a specific time [11] in interorganizational project teams (specialists coming from different companies) [12], but it is not possible to evaluate the repeatability unless the client always works with the same project teams. Therefore, despite the existence of various BIM uses in the literature, there is no instrument to evaluate their level of implementation in construction projects.
Thus, addressing the lack of a tool to evaluate how BIM uses are applied in construction projects, this research proposes and validates a BIM Use Assessment (BUA) tool for characterizing the application levels of BIM uses in the planning and design phases of building projects. The objective assessment enabled by the BUA tool yields benefits for both the industry and the academia. Organizations can use this assessment to perform a self-diagnosis that supports strategic implementation of decisions and to qualify/select other organizations in the context of future projects or joint ventures. The academia, on the other hand, can use this assessment tool for industry benchmarking and diagnosis.
2. Research Methodology
The overall research methodology is organized into three stages: (1) identification, selection, and definition of BIM uses; (2) proposal of the BUA tool for characterizing the level of BIM use application; and (3) validation of the BUA tool. Figure 1 specifies the activities, research tools, and deliverables for each stage.
In the first stage, the user guides and manuals regarding BIM uses were identified to select the BIM uses to be evaluated in this research. The guides were classified according to three criteria: (1) definition of uses that are associated with an objective/application for the project, (2) classification of the uses according to the phases of the project life cycle, and (3) definition of uses that are supported by the scientific literature.
Subsequently, each use was defined based on the selected guide and a review of the literature from the last ten years. The search was carried out in the following libraries: Engineering Village, Web of Science, and Scopus. In this review, 64 references regarding the application of the BIM uses were identified in the design and planning phases of construction projects.
Next, a panel of experts validated the selection and definitions of the BIM uses. Table 1 lists the experts who participated in the three sessions. Finally, we created a definitive list of BIM uses and their definitions for the design and planning phases of construction projects.
In the second stage, the BUA tool is proposed for characterizing the levels of application of BIM uses in construction projects in the planning and design phases. A BIM use has various levels of complexity that are associated with its own characteristics, for example, the dimensionality of the model, the level of automation, and the number of associated systems. To establish each level, the researchers defined two or more characteristics for each use; those characteristics are elements that are associated with the objective or application of each BIM use. These characteristics were defined by analyzing the necessary factors for implementing each use successfully. Then, for each characteristic, a state was defined, which represents the level of complexity with which this characteristic is employed. Each characteristic and its state were defined in work sessions with the panel of experts.
Thereafter, combinations of states were defined according to characteristics that place a project in each level. Finally, the expert panel validated the combination of states that defined each level for each use. Therefore, in the BUA, evaluating each use on a scale from one to five is proposed, where the minimum level (1) does not use the BIM model and the maximum level (5) uses it in a way that realizes all its applications. The proposal of classification by levels allows having a structured and consistent tool. Table 2 presents a general description for each level; however, a detailed description of each level of each use is provided in the BUA (see Supplementary Data available here).
In the third stage, the BUA was validated by applying the proposed instrument to 25 construction projects in the planning and design phases. The only requirement for the evaluation is to declare that the company is using BIM. The evaluation process consisted of the following steps: First, a researcher conducted an interview with each BIM manager of each project. The interviews were conducted in a conversation-like manner to ensure that the interviewees would respond transparently [13]; this avoided the bias of previously having examined the answer for each level. Then, based on a recording of the interviews, two researchers independently defined the level of the project for each BIM use. Next, in a collaborative session, the two researchers who qualified the projects were asked to discuss the final qualification of each use.
To validate the BUA tool, we conducted a concordance analysis of the evaluations of the two researchers and between each researcher and the final decision; we used the Kappa Cohen test to measure the level of concordance. Additionally, we conducted a reliability analysis of the BUA using Cronbach’s alpha test to measure the level of internal consistency of the BUA tool.
Finally, a descriptive analysis of the results that were obtained for the 25 projects was conducted. Two measures, namely, central tendency and variability, for each of the uses were analysed. Via this approach, the uses with highest and lowest levels of implementation in the sample of this study were identified. The interviews were carried out by project and not by company because even if a company has several ongoing projects, the demands of the client or the nature of the project can affect the specifications of the application of BIM uses.
3. Identification, Selection, and Definition of BIM Uses
A BIM use is a set of actions and conditions that are associated with BIM, which together have a defined objective or application for the construction project. There are various guides or manuals that define BIM uses without specifying an objective/application or only as actions that are associated with the modeling process. Two examples are “existing condition modeling” and “record modeling,” which are defined in the “BIM uses of Penn State” [2], because these uses are utilized for modeling without a defined application for the project. In applying a BIM use to a project, it is assumed that a BIM model exists. Other guides, such as “211 in model uses BIM” by Succar [3], consider additional modeling uses. In this guide, several modeling actions are specified, such as “fire modeling” and “foundation modeling.” However, actions of this type are not considered BIM uses in this research.
To select a guide on which to base the definition of BIM uses, five guides or manuals were characterized based on three criteria: definition of uses, classification of uses, and literature background of uses (Table 3). The selected guide defined uses more in accordance with the definition that is proposed in this article, with a classification that is associated with the phases of the project’s lifecycle, and with a more extensive literature background.
The BIM uses of Penn State were selected as the baseline guide in this research. This guide was selected for three main reasons: First, most BIM uses are aligned with an objective/application for the project, not just the modeling action. Other guides have BIM uses that are directly related to the modeling and not to the specified application; however, these are BIM tools, not BIM uses. Second, the classification of lifecycle phases is better suited to the requirements of this research. Third, every definition of BIM is strongly supported by the scientific literature.
Using this guide, an analysis was conducted on each of the uses that were specified in the planning and design phases of a construction project to determine which uses are considered in the evaluation tool. Fifteen uses are proposed in the planning and design phases by the Penn State guide; however, this research did not consider all the uses that are defined in the guide. The use “existing condition modeling” was deleted because it only considers the modeling and does not explain the associated benefit. In fact, the resulting model could be used, e.g., for cost estimation, 3D coordination, and site analysis. Moreover, “lighting analysis” and “energy analysis” are too specific in comparison with other uses; therefore, they were included in the use “sustainability analysis.” Likewise, the uses “structural analysis” and “mechanical analysis” were included into the “engineering analysis” use, along with other uses such as “hydraulic analysis” and “fire protection system.” Both changes were made in order to simplify the evaluation of BIM uses; however, it is possible that these simplifications are too general for specific projects where illumination, sound, or any specific analysis are carried out. Finally, ten of the fifteen uses were selected for the planning and design phases (Table 4).
Once the uses to be evaluated had been selected, the next step was to define each of them based on the Penn State guide and the analysed literature of 64 papers from the last ten years. Once the uses were defined, working sessions were held with the expert panel to provide feedback, implement the recommended corrections, and validate each of the definitions of the BIM uses.
4. Proposal of a BUA Tool for Measuring the Level of BIM Application
Once the BIM uses were defined, the characteristics, which are the actions or conditions that are necessary for applying these uses, are identified. Table 5 lists the characteristics that are used to assess each use. Additionally, each characteristic use is defined in annexed Supplementary Data available here (BUA).
Each characteristic is evaluated in various states; for example, in the use “space programming,” the characteristic “distribution analysis” has the following states: manuals, consults, report, and automatic. A characteristic can be evaluated in one or more states since the states are not necessarily mutually exclusive. For example, in the use “site analysis” and the characteristic “type of model,” the states are BIM and GIS; therefore, a project can have one or both. In the annexed Supplementary Data available here, all the states of each characteristic are presented.
To clearly illustrate the characteristics of the uses, the example below presents the selected features for the use “cost estimation.” The “cost estimation” use has three main characteristics: the type of model, the origin of the quantities, and the number of systems on which the use is applied (Figure 2). The characteristics were defined by answering the following question: What yields a higher benefit on the application of this use? Via this approach, it will be possible to analyse the current state of a project and to identify the next steps for improvement. In this example, the use of BIM to support the cost estimation is at its maximum level of application when the extraction of quantities is bidirectional between the cost’s software and the model, and based on a BIM model, without distinguishing if this is applied to a large or small number of specialties.
The characteristic “applied systems” is used to analyse the number of systems in which a BIM use is applied. The feature is divided into two options: “<50%” and “≥50%.” In Figure 2, the feature “applied systems” corresponds to the two last columns; however, the number of systems depends on the characteristics of the project; hence, the tool cannot use a fixed number for each of the options. Therefore, it is necessary to calculate the number of systems to which a BIM use can be applied.
Once the levels have been defined, a BUA is constructed as a template for each use, which includes the use name, use definition, use levels (1 to 5), and use characteristics (Figure 2). This BUA tool serves as a rubric and should be read horizontally. The light-grey squares indicate the conditions that must be satisfied to belong to a level. Likewise, the dark-grey squares indicate the possibilities within a level. The dark-grey colour is applied with the objective of avoiding subjectivity to enable the analysis of various combinations at the same level. For example, a project is assigned a score of one for the “cost estimation” use if it uses a 2D model to calculate manually the quantities for over or under the 50% of the systems.
5. Validation of the BUA Tool via the Evaluation of Projects in the Planning and Design Phases
To validate the consistency resulting from the application of the proposed BUA tool, 25 civil infrastructure and building construction projects were considered, which were all in the first phases of the lifecycle (planning and design) (Table 6).
To evaluate each project, interviews were conducted with the BIM manager and/or project manager. All the interviews were recorded with the interviewees’ consent so that their answers could be later analysed by two BIM researchers, who independently evaluated each use of the BUA.
Finally, the evaluators held a meeting to decide on the final score of the project for each use; they focused on the categories in which the scores differed between the two evaluators. In conducting this evaluation, the evaluators followed the format that is shown in Figure 3. The template must be filled by a BIM specialist who has been selected through an interview with the project leaders.
To validate the BUA tool, three concordance analyses were performed: (i) between the answers of the two researchers, (ii) between the answers of Researcher 1 and the final decision for each use, and (iii) between the answers of Researcher 2 and the final decision for each use. Cohen’s Kappa was applied to each of these scenarios (Table 7), and an almost perfect level of agreement of over 93% was reached in all three cases [15]; hence, the tool is unbiased and precise.
Then, as the next step in validating the BUA tool, the internal consistency of the measuring scale was analysed using Cronbach’s alpha coefficient. For the BUAs of 10 items and 25 test projects, a Cronbach’s alpha coefficient of 0.8617 was calculated. Hence, the elements of the BUA tool assess the same characteristic for a project for each BIM use. From the high level of internal consistency (reliability) and the high level of agreement between the interviewers, it is concluded that the BUA is an objective and consistent tool for evaluating the levels of BIM uses in the design and planning phases of construction projects.
Based on the high measurement performance of the tool in the validation process, the BUA tool can systematically characterize the levels of application of the BIM uses in the planning and design phases of building projects. Thus, the proved trustiness of this tool to assess BIM uses in the early stages of the project enables self-diagnosis of an organization’s practices regarding how BIM is used in their projects. For example, an organization can understand that its cost estimation process (BIM use) consists of consulting quantities from 3D models in less than half of their main cost items, while its design review process (BIM use) is based on a nonimmersive visualization of BIM models with an informal identification of design issues. This understanding can allow a company to formulate an improvement plan to take full advantage of BIM for its projects. Additionally, a company can use the BUA tool to assess the BIM use of potential partners or design consultants in future projects where certain BIM uses are required. Additionally, the results of the assessment done with the BUA tool in a particular project can be compared to successful cases of BIM application within the company or the industry (if available) to identify the most efficient practices for BIM uses.
Using the 25 evaluated test projects, a descriptive analysis was conducted to determine the distribution of the levels for each BIM use. Figure 4 shows a box plot for each BIM use. According to this figure, the BIM use with the highest level of application is “3D Coordination” and the BIM use with the lowest level of application is “Phase Planning,” which is associated with Level 1 in almost all cases (three projects showed an application, which were represented as atypical dots and asterisks). This low application of “phase planning” is due, according to the interviewees, to the lack of a specific requirement for this use in the contract. “Sustainability Analysis” was another BIM use that has application level of 1; however, it has higher variability. For this BIM use, even if not all the projects required an environmental certification by the clients, some projects had their own environmental requirements to be satisfied, for which BIM was helpful.
BIM uses “Site Analysis,” “Space Programming,” and “Code Validation” were associated with Level 2 of application, while “Design Review” and “Engineering Analysis” were associated with Level 3. Regarding “Engineering Analysis,” according to the interviewees, it is difficult to ask the external designers to develop the whole process in BIM software.
Finally, 6 out of the 10 evaluated BIM uses presented high variability in terms of the application levels. This is due to the differences among the projects in terms of their characteristics, the companies that execute them, and their clients. For most of the projects, an improvement opportunity was identified in the BIM uses.
Additionally, hypothesis tests were conducted to identify significant differences according to the characteristics of the evaluated projects. Nonparametric hypothetical tests were applied with a significance level of 95% since the variable of each evaluation is ordinal qualitative. The Mann–Whitney U test was used to compare pairs of samples, and the Kruskal–Wallis test was used to compare three samples. No significant differences were identified between projects with a public or private client. There are also no significant differences between projects with an internal or external client. There are no significant differences between infrastructure projects and building projects. Differences were identified in terms of the countries of origin of the projects; values of less than 0.05 were obtained between the pair Chile-Colombia and Spain, where in the latter country, the projects had significantly lower evaluations. No significant differences were identified between the projects in Chile and Colombia.
Therefore, from this initial evaluation, it can be noted that there is a high variability in the types of BIM uses in the design planning phases, where the most developed use is coordination between specialties, and the least developed is 4D planning. In addition, this sample shows high variability in the level of development and automation of each use, for example, in cost estimation or design authoring.
6. Conclusions
The proposed BUA tool that is presented in this study contributes to the diagnosis of the application of BIM uses, thereby enabling companies and clients to identify the BIM use state of the project, the way in which the BIM uses are being implemented and opportunities for improvement. Via this approach, it is possible to realize higher benefits from the BIM methodologies when they are applied in the earliest stages of the projects. Since countries are encouraging the use of BIM methodologies, having a tool that enables the assessment of the application of BIM uses in projects is advantageous for those who are in the process of implementing or are seeking to implement this methodology efficiently. Then, the BUA tool becomes crucial for companies in evaluating how they are utilizing all the “uses” that BIM can offer. Likewise, BUA tool can be used to evaluate companies to be contracted seeking for the specific BIM use level, or also for benchmarking the BIM use level in the industry.
In the validation process of the BUA tool, a high consistency value was obtained; i.e., the tool is reliable and measures what is being measured. In addition, when the same project was evaluated by two researchers, a high percentage of agreement was obtained; therefore, it can be concluded that the tool is free from the evaluator’s bias. The BUA tool has high concordance and consistency values; however, it is recommended that external specialists evaluate a project to eliminate biases. The BUA can be used as a self-assessment tool if the examiner is knowledgeable regarding BIM. In addition, this evaluation enables the comparison between projects of the same company or of different companies and promotes benchmarking and continuous improvement in organizations.
The descriptive analysis and hypothesis tests were conducted with the pilot test of 25 projects. The use of a larger sample size is recommended for obtaining more general conclusions according to the characteristics of the project. The BUA defines each level as a combinatorial of states associated with each characteristic of each use; however, this combinatorial could be different. The uses “sustainability analysis” and “engineering analysis” group several specialties, respectively; this simplification could generate an information gap if the objective of the assessment is to understand the use of each specialty individually.
Future work will focus on the extension of BUA, which was developed for implementation in the planning and design phases, to the other project lifecycle phases, namely, the construction and operation phases. In addition, it would be of substantial value to assess the level of socialization of each BIM use since, for effectively using BIM, the integration of information among all factors of the project is necessary. This would involve assessing the way in which the information is managed, shared, and stored. Additionally, with a greater number of projects, a deeper descriptive analysis can be made of the states in which each characteristic of each BIM use is located.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work was supported by FONDECYT (1181648 to Alarcón L. F. and Mourgues C.) and CONICYT, Chile (PCHA/National Doctorate/2018-21180884 to Herrera R. F.).
Supplementary Materials
The BUA is composed of three sheets in an Excel file. The first two sheets (“Plan” and “Design”) define the conditions for belonging to each level for the BIM uses and are similar to help tables. The third sheet, namely, “Interview Template,” should be used to conduct the interview. In this template, the project data are obtained by writing an “x” in the corresponding box and using the first sheet, the level of implementation for each BIM use in the project can be identified. (Supplementary Materials)