Abstract
Background. In Mexico, several studies have been conducted under the ISAAC methodology; nevertheless, no validation studies of the ISAAC questionnaire based on objective clinical testing in our country have been published. Aims of the Study. To validate the ISAAC questionnaire, used in a study of prevalence of allergic diseases, based on medical, respiratory, and allergic evaluations of schoolchildren being 11 to 16 years old in Mexicali, Mexico. Material and Methods. We conducted a cross-sectional study to validate the ISAAC questionnaire through the generation of an index (considered as gold standard) using pre- and postbronchodilator spirometry test, exhaled nitric oxide measurements, and atopic evaluations. 114 schoolchildren were included (23 asthmatics with respiratory symptoms and 91 nonasthmatics without respiratory symptoms) and we evaluated the sensitivity, specificity, and positive and negative predictive value of the questionnaire using discriminant analysis. Results. We observed sensitivity of 35.2% and specificity of 93.3% and the positive and negative predictive values were 82.6% and 61.5%, respectively. Conclusions. Our findings indicate that the ISAAC Mexican version questionnaire was less sensitive and more specific when compared to the gold standard; however, it is adequate and able to discriminate children with and without asthma and a useful tool to use in epidemiological studies.
1. Introduction
Asthma, allergic rhinitis, and atopic dermatitis are three of the allergic diseases that most affect children worldwide [1–3]. Asthma is a multifactor disease characterized by chronic inflammation of the airways and a reversible obstruction of air flow. Concern about these diseases has arisen because of their increase over the years, with prevalence and mortality rates having increased worldwide [4].
The International Study of Asthma and Allergies in Childhood (ISAAC) methodology was established in 1990s [4]. This methodology is a useful and practical strategy to most effectively identify these types of affections in children. It has been used in many countries where different languages are spoken, as well as in countries where the language does not have a way to describe asthma symptoms.
The ISAAC is designed with three phases. Phase I is designed to assess the basic prevalence and severity of asthma and other allergic diseases by administering target questionnaires. Phase II investigates possible risk factors related to these diseases and those suggested in phase I. Phase III repeats phase I to assess the prevalence tendency of these diseases and to examine the hypothesis established by conclusions from phase I [5].
The ISAAC protocol maximizes the value of epidemiological research involving transversal studies by providing a standard tool to enable comparisons with other populations. Therefore, it is advisable to establish a mechanism to validate the tool in order to assure the quality of information and support the findings. Though ISAAC validation studies have been conducted worldwide, some of these studies have been performed without considering objective clinical tests, contrasting only the responses of parents with those of children or using other tests for validation [6–9]. On the other hand, different studies using this methodology have been conducted in Mexico [10–12]; however, there are no published studies that validate the ISAAC questionnaire using objective clinical tests. Thus, the purpose of the present study is to validate the ISAAC questionnaire used in a study of prevalence and risk factors in Mexicali, Baja California, Mexico, conducted in 2004, measuring the tool’s sensitivity, specificity, and concordance by contrasting previous results with clinical tests for lung function and other markers in children.
2. Material and Methods
2.1. Design of Study Population
To validate the questionnaire using objective clinical tests, a convenience sample of 114 schoolchildren from 11 to 16 years old was chosen from a total of 5287 schoolchildren who participated in the transversal study of allergic diseases during 2004 in Mexicali, Baja California, Mexico.
2.2. Data Collection
An ISAAC questionnaire was used to collect data on sociodemographic variables, past health history, and respiratory health. Each participant was assigned a personal number and was classified as asthmatic or nonasthmatic, according to information reported on the questionnaires, which were self-administered by the parents. All procedures were explained to the parents, who signed an informed consent form. The children also gave their informed assent. The study protocol was reviewed and approved by ethics committees of the National Institute of Public Health. Of 114 children selected, 23 reported a medical diagnosis of asthma and have respiratory symptoms in the last 12 months and 91 did not report symptoms in the last 12 months or medical diagnosis of asthma.
2.3. Mechanisms Used to Validate the Questionnaire
The questionnaires asked two questions. Has a doctor ever diagnosed your child as asthmatic? And have you had wheezing or whistling in the chest in the last 12 months? After children were identified as asthmatic or non-asthmatic based on the answers, different clinical tests were conducted to obtain a gold standard index to validate both questions. The medical diagnosis of asthma reported in the ISAAC questionnaire was validated based on the following tests.
2.3.1. Reversibility Test for Lung Function
A spirometry test (Easy One, Zurich, Switzerland) was administered to each participant in accordance with American Thoracic Society 1994 criteria [13]. After an acceptable and reproducible test was obtained, a 200 μg dose of albutherol (a short-acting B2 adrenergic receptor agonist) was administered. After 15 minutes, a second spirometry was conducted to assess the change in lung function. To determine the bronchodilator response we use the degree of reversibility between prebronchodilator and postbronchodilator of FEV1 (≥12% and ≥200 mL) according to the GINA criteria [14]; however, we also calculated the percentage of change between prebronchodilator and post bronchodilator test, quantified as a percent relative change value and derived from the number of schoolchildren who had a positive reversibility test using both criteria was almost similar, we decided to use this percent change value for the generation of the gold standard index.
2.3.2. Exhaled Nitric Oxide (eNO) Concentrations
eNO concentrations were measured for each child during visits to the health center and according to American Thoracic Society recommendations [15]. As recommended, the child was asked to be seated for five minutes before and during the test. All tests were conducted indoors to reduce possible contamination from environmental nitric oxide. The exhaled nitric oxide was instantly measured by chemiluminescence using a portable NIOX MINO, with a measurement range from 5 to 300 ppb and a detection limit of 5 ppb [16]. For each participant, the two best tests were selected and the average concentration for those tests was considered.
2.3.3. Skin Test
A skin test was conducted using multitest testing [17, 18]. This enabled identifying a specific allergen response to standard allergens (Dermatophagoides farinae, grass, trees, dogs, cats, and cockroaches) glycerinated in a concentration of 1 : 10 (weight/volume) with negative control and positive to histamine (1 : 100). The test was performed by trained medical personnel. A diameter of ≤3 mm for the allergens tested was considered to be a null or negative result. A positive skin test was determined by a wheel greater or equal to 3 mm or equal to the same diameter of the wheel produced by the histamine control, after 15 minutes of placing the skin test. We determined atopy status as having at least one reaction to an antigen tested.
2.4. Statistical Analysis
First, the questionnaire’s reliability was determined by assessing the inner consistency and concordance of each answer using kappa statistics. Second, the quality and validity of the medical diagnoses of asthma were assessed by comparing answers with the index generated according the clinical tests, with the latter being considered the gold standard. The rate was obtained by discriminant analysis [19], which is a multivariate statistical technique that analyzes significant differences among groups of individuals versus variables measured in those groups. If there are any differences, it tries to explain them and provides systematic classification procedures for new observations with unknown origins in one of the groups studied. In that respect, we considered 3 classification variables: (1) percentage change in FEV1 or reversibility test, (2) average concentration of FeNO levels with logarithmic units, and (3) atopic condition, with 2 comparison groups (asthmatic individuals and non-asthmatic individuals) in a sample size of 114 children (23 asthmatic individuals and 91 nonasthmatics individuals). Additionally and with the aim of comparing the results obtained, we also analyzed the information considering only the reversibility test according to GINA criteria as the gold standard and tested again the sensitivity and specificity of the ISAAC questionnaire. Finally, a new classification was obtained based on a constructed discriminant function, a canonic correlation, and the transformed lambda value. ANOVA tests were used as validation parameters for the classification obtained.
3. Results
Twenty-three (20.2%) of the 114 children participating in the research reported in the questionnaire having a medical diagnosis of asthma, with a higher percentage (60.9%) being female, 12 to 14 years old. A total of 79.8% of schoolchildren participating in the research did not report symptoms and were considered as non-asthmatics, 59.1% of which were female, from 11 to 16 years old.
Table 1 presents the results of the descriptive statistics and main outcomes, where it can be seen that the parameters for lung function do not differ between the two groups of individuals studied (asthmatic and non-asthmatic participants), although the FEV1 reversibility percentage was slightly greater for the asthmatic group (). Average FeNO concentrations were higher in the group of individuals that did not report a clinical diagnosis of asthma. Differences were not statistically significant.
Positive skin tests were observed in 48% of the total sample, with a higher prevalence in the asthmatic group versus non-asthmatic schoolchildren (82.6% versus 39.6%, resp., ) (Table 1). The allergens that provoked a positive response with greater frequency were different between the groups. The most frequent allergens for asthmatic children were trees and dermatophagoides farinae (14% and 9%, resp.) whereas for non-asthmatic children they were cockroaches and cat and grass allergens (23% and 16%, resp.) (Table 2).
A new classification of participants was obtained as a result of the gold standard. Fifty-four of the 114 children were classified as asthmatic and 60 as non-asthmatic, as shown in Table 3. Nineteen of the 23 children who reported a diagnosis of asthma on the questionnaires were classified by the gold standard (sensibility = 35.2% and positive predictive value = 82.6%) and 56 of the 60 schoolchildren classified as non-asthmatic by the gold standard were identified correctly as non-asthmatic individuals by the questionnaire (specificity = 93.3% and negative predictive value = 61.5%). On the other hand, when we use the reversibility test considering GINA guidelines only as gold standard, the sensitivity and specificity were 22% and 81%, respectively (Table 4).
Based on the data obtained, we can observe that the probability that the questionnaire correctly classifies a child as asthmatic (sensibility) was lower than the probability of correctly classifying him or her as non-asthmatic (specificity). Table 5 shows the characteristics of individuals according to the gold standard classification.
With the ROC analysis, a probability of 64% was observed for asthmatic and non-asthmatic individuals and both groups were correctly classified by the discriminant analysis. Concordance (kappa test) for both tests was 65.8.
4. Discussion
The ISAAC study and its phases have been developed in our country [10–12]; the prevalence and risk factors associated with allergic diseases among schoolchildren are known. Nevertheless, to our knowledge the present study is the first conducted in our country to validate the ISAAC questionnaire through objective clinical testing.
Our results indicate that 54 of the 114 children participating in the research were classified as asthmatic according to the gold standard (percentage change in FEV1, average of FeNO concentrations, and skin test positive). Meanwhile, according to the questionnaires, 23 individuals reported having an asthmatic condition, which represents a very low sensibility (35.2%) of the instrument compared to other studies. This could indicate that interpreting all recent symptoms in the questionnaire was not adequate and may result in subestimating the prevalence of childhood asthma; that is, the 35 schoolchildren classified as negative false by the questionnaire are probably those who are not familiar with the term of wheezing and did not correctly report the respiratory noises and sensations, and the 4 schoolchildren who were found as positive false had some respiratory disease or respiratory symptoms when the questionnaire was applied. On the other hand, we observed that our gold standard classified 60 children as non-asthmatics versus 91 who were reported as healthy by the questionnaire, resulting in specificity of 93.3%. Similarly, the positive predictive value of the questionnaire was high (82.6%). These results are very important in public health because this tool can be used with confidence to identify the health situation in communities.
With regard to validation studies conducted worldwide, specifically among the Latino school population, it is important to point out that, unlike the majority of studies published, our research is one of the first studies to use several clinical tests (spirometry with bronchodilatation, measurement of FeNO levels, and skin tests). Other studies validate questionnaires mainly by contrasting responses of parents with those of children or by exclusively using spirometry.
Research conducted in a Latino population in Los Angeles, CA, USA, [20] evaluated the validation of responses to a questionnaire using spirometry and children’s responses. The study compared parents’ responses with children’s responses regarding asthma diagnosis. The more accurate report was obtained from children, for which 47% of children reported as asthmatic in the questionnaire were observed to be positive by at least one of four spirometries conducted.
Another validation study was conducted with Spanish speakers in Spain [21]. This study demonstrated how much variability an event can present, such as bronchial hyperreactivity, when compared with other studies performed in the same country.
In addition, a multicentric validation study conducted in 40 countries and 90 centers was performed by comparing responses before and after children were shown a video to identify respiratory diseases. This study found that teenagers interpreted asthmatic symptoms differently. In 20 centers there was concordance of ≥0.4, indicating that the questionnaires should obtain the same results when administered using different modalities, at different times, and by different observers [22]. Thus, total validation of the questionnaire is a strategy that more greatly guarantees obtaining reliable data. In the present research, the same questionnaire was administered to an additional subsample at two different but consecutive moments. According to kappa statistics, that verified the reliability and quality of the instrument, we observed a questionnaire reproducibility of 66%.
With regard to the classification of asthma in this study using the ISAAC questionnaire, it is important to mention that this was obtained mainly with two questions. Has a doctor ever diagnosed your child as asthmatic? And during the last 12 months has your child observed wheezing? Therefore, we are probably excluding children with wheezing during another period of life, which could indicate an asthmatic condition.
A limitation of this study may be that the gold standard was based on three different tests; therefore the sensitivity of the questionnaire could be affected. For this reason, additionally we used only the reversibility test considering the GINA guidelines [14] as gold standard and the sensitivity of the ISAAC questionnaire was low; Jenkins MA and collaborators found sensitivity of 0.39 (0.21–0.61) in adults and 0.54 (0.48–0.67) in children using the bronchial hyperresponsiveness (BHR) as gold standard for asthma, noting that the BHR was highly specific but not sensitive [23]. Nevertheless, the literature has reported that the tests that we used can be used for the diagnosis of asthma. Therefore, we believe that, in this case, lack of affirmative criteria to diagnose asthma would underestimate the sensibility of questionnaire. Finally, our sample size is small which may have limited the power of our study.
5. Conclusions
In summary, our findings indicate that the ISAAC Mexican version questionnaire was less sensitive and more specific when compared to the gold standard; however it is adequate and able to discriminate children with and without asthma and a useful tool to use in epidemiological studies. Nevertheless, to improve the consistency and quality of information, measurement instruments need to be continuously validated because, though they are the standardized ones, it is important to remember that language, culture, and accessibility can affect the results. This will enable verifying the authenticity of the data and assuring that the efforts of governmental, social, and scientific institutions benefit high-risk populations.
Conflict of Interests
The authors declare they have no competing conflict of interests.
Acknowledgments
We want to extend our gratitude to the schoolchildren that participated in the study. This research was supported by a research grant from the National Center for Environmental Health, Center for Disease Control and Prevention, Atlanta, GA.