Questionnaire for Assessing Preschoolers’ Organizational Abilities in Their Natural Environments: Development and Establishment of Validity and Reliability
Despite the consensus in the literature regarding the importance of organizational abilities in performing daily tasks, currently there is no assessment that focuses exclusively on such abilities among young children. The study aims to develop a Questionnaire for Assessing Preschoolers’ Organizational Abilities (QAPOA), Parents’ and Teachers’ versions, and to examine their reliability and validity. QAPOA was distributed to preschool teachers and parents of 215 typically developing 4–5.6-year-old children. The teachers’ and parents’ versions demonstrated good internal consistency. Factor analysis performed to examine the tool’s content validity yielded two factors: motor-based and language-based OA. Furthermore, both versions of the questionnaire demonstrated significant differences between OA among boys and girls. Concurrent validity was demonstrated between the QAPOA total scores and the equivalent subscale of the BRIEF-P. Given these findings, different cut-off scores were established for identifying boys and girls with either motor-based and/or language-based OA. The results indicate that both the teachers’ and parents’ versions of the QAPOA are reliable and valid measures of children’s organizational abilities. The questionnaires can assess and identify risk for organizational disabilities as early as preschool age. Thus, it can contribute to the planning of appropriate intervention programs and the prevention of difficulties in the future.
Children require organizational abilities to participate in age-appropriate daily occupations [7–10]. For example, so as not to disrupt the progression of activities performed at home and in school, they must ensure that the necessary objects are readily accessible, do not become dispersed, or get lost. Spatial organization (school, playground, home) enables children to orient themselves and perform activities independently in familiar environments. Activities of daily living (such as brushing teeth, dressing, eating) should be temporally organized so that the child can meet his/her schedule demands (getting to school/friends/after-school activities) in a timely fashion. Organization of speech is also necessary to communicate one’s experiences in the proper sequence so that they can be understood by the listener. Environmental demands for performing school tasks are easier to manage for children with efficient organizational ability [2, 11–14]. The relationship between organizational ability and client factors such as perceptual, motor, cognitive, and language abilities is articulated in both the cognitive and neurodevelopmental frames of reference. Fried  emphasizes that organizational ability functions as a “conductor,” orchestrating the client functions mentioned above. She presented a cognitive approach that ascribes organization as an executive function [16, 17]. Executive functioning (EF) refers to higher level cognitive functions, such as formulating and executing goals, planning, organization, inhibitory control, working memory, metacognition, and directing brain activity between the cognitive and emotional processing systems [18–23].
The neurodevelopmental approach describes the concept of organization according to the sensory integration (SI) framework . SI is a neurological process that organizes sensory input from all sensory channels from within the body and the environment to enable effective functioning. Ayres introduced the concept of praxis, defined as a basic organizational process that incorporates motor ideation, planning, and execution of a purposeful action. Accordingly, praxis refers to the process through which people formulate an idea (what to do) and organize it (how to do it) through a timed sequence of activities, depending on their current spatial context, enabling them to participate in meaningful occupational activities 
Organization abilities have been demonstrated in studies of 3-4-year-old children [15, 25] Although executive functions in general and organization in particular evolve at a rapid pace during the preschool years , there are few reports of standardized assessment tools for preschoolers focusing on organization . Moreover, despite the fact that the young child’s everyday environments, both at home and at preschool, are important venues for observing routine manifestations of executive functions in general  and organizational abilities in particular, standardized and ecological assessments for young children are scarce [26–28]. A few standardized assessments, such as the Goralnik , the Shatil test , and the Kaufman Assessment Battery for Children , have been developed but they do not assess the child in his/her natural environment. In addition, the Israeli Ministry of Education has published “Mabatim,” a tool for use with 3–5-year-old preschoolers to assist teachers in observing and tracking children’s preschool performance [32, 33], but it relates only to their functioning within the educational environment and has not yet been validated. Other standardized, ecological tools for preschoolers include the Brief-P Inventory of Executive Function for Preschoolers [34, 35] and the Performance Skills Questionnaire (PSQ) . These assessments focus on EF in young children, but only partially relate to organizational abilities. Thus, the QAPOA was developed to meet the need for an ecological standardized assessment focusing on organizational abilities among preschoolers and includes both a parents’ and teachers’ version. This tool enables educators and parents of preschoolers to identify children with rudimentary risk for organizational disabilities at an early age and refer them for further appropriate intervention. The purpose of this study was to establish the validity and reliability of both versions of the QAPOA.
The study examined a convenience sample of 215 typically developing healthy children ranging from 61 and 68 months (M = 67.21, SD = 6.08), including 118 girls 55% and 97 boys 45% attending mainstream educational system preschools, as reported by their parents. All were Hebrew speakers. Exclusion criteria included children diagnosed with ADHD, neurodevelopmental disorders, and developmental delays.
(1)Demographic Questionnaire for the Preschoolers, which includes items relating to the children and their familial, developmental, and medical background.(2)The Questionnaire for Assessing Preschoolers’ Organizational Abilities (QAPOA, Parents’ version (QAPOA-P) and Teachers’ version (QAPOA-T)) : these questionnaires are designed to gather information about the child’s organizational abilities in his/her everyday environments, home and preschool. Both versions of the QAPOA contain 48 items, 44 of which were accepted through an expert validation process. The other four items, relate to the parent’s or teacher’s general impression of the child’s organizational ability in each occupational area (play, activities of daily living, school, and social interaction), for example, “What is your general impression of the child’s organizational abilities at school.” The tool is scored on a 5-point Likert scale (1 = “never,” 2 = “rarely,” 3 = “sometimes,” 4 = “usually,” 5 = “always”). The total score comprises the sum of scores of the specific items relating to organizational abilities (ranges from 48 to 240). Before completing the questionnaire the respondents are asked to rate their general impression of the child’s organizational abilities according to the above scale, presented at the beginning of the questionnaire.(3)Behavior Rating Inventory of Executive Function, Preschool version (BRIEF-P) : the BRIEF-P is a reliable and valid questionnaire for parents and teachers of preschool-aged children that enable professionals to assess (EF) behaviors in everyday contexts (home and preschool). This questionnaire is designed for children ages 2:0–5:11. It is a 63-item measure with five related but nonoverlapping EF domains: Inhibit, Shift, Emotional Control, Working Memory, and Plan/Organize. It takes 10–15 minutes to complete and is scored on a 3-point scale (1 = “never,” 2 = “sometimes,” 3 = “often”). Raw scores are converted to -scores (M = 50, SD = 10), and the cut-off score is 65 (1.5 SD). Studies have demonstrated good internal consistency, temporal stability, and concurrent validity .
2.3.1. Development of the QAPOA and Examination of Content Validity
Initially, 45 statements were selected for the questionnaire based on the cognitive [5, 15, 17] and neurodevelopmental  theoretical models. The face validity of the tool was examined by six pediatric clinicians (3 occupational therapists OTs and 3 speech-language therapists SLTs) with a minimum of 10 years’ experience working with preschoolers who were asked whether the statements did (yes) or did not (no) represent the organizational abilities of preschoolers. In accordance with their recommendations, ten items were added.
Next, the content validity of the revised 55-statement item pool was examined through the use of feedback questionnaires sent to 15 expert professionals (preschool teachers, OTs, and SLTs), with over 10 years of experience working with preschoolers. They were asked to rate each item as suitable, somewhat suitable, or very suitable in representing the organizational abilities of preschoolers. Ten questionnaires were completed by four OTs, four SLTs, and two preschool teachers. The maximum total score for each statement was 30 (3 points 10 questionnaires, per statement). Only the 44 statements that received 18 or more points (60% of maximum score) were retained . Examples of statements in the area of play include “selects preferred games at home” and in the area of social interactions “relates and responds appropriately to what others say.” In addition, four statements relating to the parent’s or teacher’s overall impression of the child’s organizational ability in the four occupational areas (i.e., play, activities of daily living, preschool, and social interaction) were included, resulting in a total of 48 statements.
Following approval of the Chief Scientist of the Ministry of Education the questionnaires were distributed to the preschool teachers and parents of the participants to initiate the validation process for both versions of the QAPOA (P/T).
3.1. Internal Consistency Reliability
The internal consistency (ICC) of both versions of the QAPOA (P/T) was found to be very high ( and 0.94, resp.).
3.2. Construct Validity
As can be seen from these tables a high ICC was found between the items in each factor. Notably, according to both the teacher’s and parents’ reports, a significant difference was found between boys’ and girls’ scores (, ; , , resp.) (Tables 1 and 2).
3.3. Convergent Validity
Pearson’s correlational analyses revealed significant correlations between the preschool teachers’ total organization score on the QAPOA and the -score on the BRIEF-P Planning and Organization scale, kindergarten teacher’s version (; ), and between the QAPOA total organization scores and the teacher’s overall impression of the children’s organizational abilities (. ). Similar findings were found for the parents’ scores on these scales (i.e., parent’s total QAPOA organization score and the BRIEF-P -score, , ; QAPOA-P total score, and parents’ overall impression of children’s organization, , ).
3.4. Questionnaire Cut-Off Scores Indicating Difficulty in Organization
The QAPOA cut-off scores were found to be different for boys and girls, both with respect to total scores and the motor and language subscale scores. Total and subscale scores at or under the 16th percentile were identified as cut-off scores, indicating a risk for difficulties in organization. Thus, the identified total organization cut-off score for boys on the QAPOA-T was 173 and 185 on the QAPOA-P, and for girls it was 192 (QAPOA-T) and 196 (QAPOA-P). The same held true for cut-off scores for each subscale, in accordance with the findings of the factor analysis reported in Tables 1 and 2.
Specifically, the total cut-off score for boys on the QAPOA-T motor organization subscale was found to be 68, and it was 71 on the QPAOA-P, representing scores at or under the 16th percentile. For girls the total cut-off score signifying risk for organizational difficulties in the QAPOA-T was 76, and on the QAPOA-P it was 77. On the language-related organization subscale, the following cut-off scores were identified: for boys on the QAPOA-T = 90 and on the QAPOA-P = 94; for girls on the QAPOA-T = 99 and on the QAPOA-P = 100.
4. Discussion and Summary
The aim of the current research was to describe the development and examine the psychometric properties of the QAPOA, a practical, user-friendly tool for evaluating children’s organizational abilities in ordinary life situations at kindergarten and at home. The statistical analyses found the QAPOA to be a valid and reliable tool.
4.1. Internal Consistency
Internal consistency refers to the extent to which the individual items in an instrument measure the same construct. The statistic generated is a correlation coefficient ranging from 0 and +1.00, and a correlation of 0.80 is considered to demonstrate adequate reliability .
The item-selection procedure for the QAPOA was based on cognitive and neurodevelopmental theoretical models, as well as on the clinical experience of experts in preschool education and related therapy services (OT, SLP). The meticulous item-selection process described above testifies as to its content validity, supported by the high Cronbach’s coefficient values found for both versions of the tool (QAPOA-P, ; QAPOA-T, ).
According to the neurodevelopmental  and cognitive [15, 17] theoretical approaches upon which the initial item selection was based, organization is conceptualized in terms of its spatial and temporal dimensions. However, during the process of item selection, the need arose to also consider the organizational abilities needed for effective social interaction, as exemplified by items such as “relates to initiatives of peers during free play” or “seeks help when needed from adults or children through gestures, spoken words, or sentences.” Thus, we propose the following expanded definition of organizational skills: the ability to set meaningful goals and plan and execute them efficiently and successfully in the appropriate order and sequence, temporally and spatially with objects and people.
4.2. Construct Validity
Construct validity is defined as the extent to which an operationalization measures the concept it is supposed to measure [38, 39]. Factor analysis is one approach used to establish construct validity for the purpose of determining how unobserved or latent factors exert directional influences on participants’ responses of observed variables. It helps the researcher identify the number and nature of such latent factors .
The QAPOA questionnaire items were categorized according to the major occupations characterizing children’s everyday activities: playing, learning, and daily activities of living and social interaction . Therefore, it was hypothesized that factor analysis would group the items into four factors corresponding to these four occupational areas. However, surprisingly, the items clustered into two factors: motor-based organizational skills, reflecting children’s motor abilities (what and how to do), and language-based organizational skills, reflecting children’s language ability (what and how it is said).
In light of this finding, three different cut-off scores were established to determine whether a child is within the normal range of organizational abilities: one for total organization, for motor-based organization, and for language-based organization.
The disclosure of these two factors enable us to conduct a more in depth, comprehensive assessment process that relates to client factors underlying organizational abilities. For example, “puts personal belongings in their proper place” (QAPOA-P, item 35) represents a motor-based expression of a child’s organizational abilities in the natural environment. If the child scores below the cut-off, he/she can be referred for further evaluation by an Occupational Therapist (OT). Likewise, “seeks help from an adult or a child when needed using gestures, words, or sentences” (QAPOA-P, item 40) represents a language-based expression of a child’s organizational abilities in his/her natural environment. Children obtaining a score below the cut-off in this area can be referred to a Speech-Language Pathologist (SLP). If a child scores below the cut-off for both motor- and language-based organization, he/she can be referred for both occupational and speech therapy and may need to be referred to other types of professionals.
4.3. Gender Differences
Another way to examine the construct validity of a tool is to examine whether it distinguishes between populations. The QAPOA-P/T was found to distinguish between boys and girls, such that for both versions the cut-off score for boys for total organizational abilities as well as for motor- and language-based organizational abilities was lower than that for girls. The BRIEF-P  also has different cut-off points for boys and girls, and there too the cut-off score for boys is lower than for girls. In contrast, no gender differences were found in the My Child’s Play Questionnaire , a parental questionnaire that examines play abilities among 3–9-year-old children, even within the subscale of executive functioning.
Convergent validity measures the extent to which the underlying constructs of an assessment tool are similar to those measured by established, validated instruments . In the current study, the convergent validity of the QAPOA-P/T was supported by moderate-significant correlations found with questions from the BRIEF-P that assess planning and organization, in both the parents’ and kindergarten-teachers’ versions. This finding provides evidence that the construct of organization, as examined in both tools, is similar. However, the degree to which the observation of behavior is representative of this construct differs between them. Specifically, the BRIEF-P is comprised of 63 statements, of which only 10 questions refer to organizational abilities. In contrast, the essence of the QAPOA-T/P is to examine organization abilities in a more comprehensive manner, such that all 48 items examine these abilities. For example, while only one item deals with organization abilities during play in BRIEF-P, QAPOA specifies 11 of such abilities as preferred game selection, story initiation, and respecting game rules. Likewise, nonitems in BRIEF-P concern organizational abilities during social communication, while QAPOA includes nine items/statements dealing with these abilities, like focusing on the topic of discussion, experience sharing, seeking help, referring to what was said by the other (see Tables 1 and 2 for the all the items). Since several areas in the QAPOA-T/P are not covered by the BRIEF-P, further validation should be considered with other subtests or assessment tools that examine organizational abilities (e.g., 30, 31, 37).
The findings of the current study support the reliability and validity of preschool teachers’ and parents’ versions of the QAPOA in assessing organizational abilities among Hebrew-speaking preschoolers. Data on reliability and validity should be taken into account before using QAPOA in other cultures.
As noted above, the process of developing the questionnaire raised the need for a new, more comprehensive definition of the construct of organization that relates to motor- and language-based organization abilities and to establishing cut-off scores according to gender. This questionnaire represents a more focused foundation upon which professionals can base their continued evaluation according to the various body functions.
This process informs us of the need for a more comprehensive top-down model for evaluating children’s organizational abilities and should relate to the relevant abilities needed in their natural environment. Further assessment should then include the evaluation of body functions within the various therapeutic domains. Figure 1 illustrates such a model, which provides a structured and comprehensive guide that can serve to streamline the evaluation process of organizational abilities among children. The use of the approach delineated in this model can enable the creation of a more personalized intervention program for children with organizational disabilities.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
The authors are grateful to the parents and teachers of the preschoolers who participated in the study. The authors wish to thank the students in the research seminar courses in the Speech and Language Pathology and Occupational Therapy Departments at Ono Academic College for their assistance in data collection. Finally thanks are due to Professor Noomi Katz, Director of the Research Institute for Health & Medical Professions at Ono Academic College, and to Dr. Yossi Frier for their support and counsel through all phases of the present study.
A. J. Ayers, Sensory Integration and Praxis Test, Western Psychological Service, Los Angeles, Calif, USA, 1989.
M. Levine, A Mind at a Time, Simon & Schuster, New York, NY, USA, 2002.
N. Lifshitz, N. Josman, and E. Tirosh, “Validation of the Questionnare for Assessing Students’ Organizational Abilities (QASOA),” The Israeli Journal of Occupational Therapy, vol. 22, no. 3, pp. 53–69, 2013.View at: Google Scholar
N. Lifshitz and N. Josman, “Development of the Questionnaire for Assessing the Student’s Organizational Abilities (QASOA): establishing reliability and validity,” Israeli Journal of Occupational Therapy, vol. 15, pp. H5–H29, 2006.View at: Google Scholar
P. Dawson and R. Guare, Executive skills in children and adolescents, The Guildford Press, New York, NY, USA, 2004.
T. Aharon, T. Menachem- Eitan, M. Kalingel-Levi et al., “The Relationship between organizational abilities and sensory processing among preschoolers,” Israeli Journal of Occupational Therapy, vol. 26, no. 1, pp. H28–H43, 2017 (Hebrew).View at: Google Scholar
G. Tubul-Lavy and N. Lifshitz, “Organizational abilities among kindergarten children: comparison between children with developmental delay and typical children,” Speech & Hearing, vol. 35, pp. 57–70, 2016 (Hebrew).View at: Google Scholar
M. Malekpour and S. Aghababaei, “The effect of executive functions training on the rate of executive functions and academic performance of students with learning disability,” International Journal of Developmental Disabilities, vol. 59, no. 3, pp. 145–155, 2013.View at: Publisher Site | Google Scholar
R. Fried, “Neuropsychology of Attention-Deficit Hyperactivity Disorder (ADHD),” in Proceedings of the ADHA Worldwide - 1st Joint Meeting, Tel Aviv, Israel, 2013.View at: Google Scholar
N. Josman and S. Rosenblum, “A metacognitive model for children with atypical brain development,” in Cognition, Occupation and Participation Across the Life Span, N. Katz, Ed., pp. 223–247, AOTA, Bethesda, Md, USA, 3rd edition, 2011.View at: Google Scholar
J. Toglia, “The multicontext approach to cognitive rehabilitation of executive function,” in Proceedings of the Workshop presented at the Research Institute for Health & Medical Professions, Ono Academic College, Israel, 2011.View at: Google Scholar
P. J. Anderson, “Towards a developmental model of executive function,” in Executive Functions and the Frontal Lobes: A Lifespan Perspective, V. Anderson and P. J. Anderson, Eds., pp. 3–22, Psychology Press, New York, NY, USA, 2008.View at: Google Scholar
E. R. Blanche and L. D. Parham, “Praxis and organization of behavior in time and space,” in Understanding the Nature of Sensory Integration with Diverse Populations, S. S. Roley, E. I. Blanche, and R. C. Schaaf, Eds., pp. 183–200, Communication Skill Builders, 2001.View at: Google Scholar
P. A. Crist, “Framing ecological validity in occupational therapy practice,” The Open Journal of Occupational Therapy, vol. 3, no. 3, pp. 1–8, 2015.View at: Google Scholar
A. Golos and N. Weintraub, “The structured Preschool Participation Observation (SPQ): development and validation,” he Israeli Journal of Occupational Therapy, vol. 24, pp. H59–H72, 2015.View at: Google Scholar
M. Badia, M. Begoña Orgaz, M. Gómez-Vela, M. A. Verdugo, A. M. Ullán, and E. Longo, “Do environmental barriers affect the parent-reported quality of life of children and adolescents with cerebral palsy?” Research in Developmental Disabilities, vol. 49-50, pp. 312–321, 2016.View at: Publisher Site | Google Scholar
E. Goralnik, Language Screening Test for Hebrew-Speaking Preschool Children, Gai Agencies, Netanya, Israel, 1995 (Hebrew).
E. Shatil, Shatil Test for the Assessment of at-risk Preschool Children, Ach Publishing, Haifa, Israel, 2002 (Hebrew).
D. Loomis, R. Holt, J. C. Kaufman, and A. S. Kaufman, “Kaufman Assessment Battery for Children,” in Corsini Encyclopedia of Psychology, 2004.View at: Google Scholar
O. Goldhirsch, E. Wagner, and M. Vinokour, Mabatim: Looking on Children in Their Natural Environment, Ministry of Education Press, Tel-Aviv, Israel, 2002 (Hebrew).
A. Golos, M. Sarid, M. Weill, and N. Weintraub, “Efficacy of an early intervention program for at-risk preschool boys: a two-group control study,” American Journal of Occupational Therapy, vol. 65, pp. 400–408, 2011.View at: Google Scholar
G. Gioia, K. A. Espy, and P. K. Isquith, “Behavior Rating Inventory of Executive Function–Preschool Version (BRIEF-P),” in Psychological Assessment Resources, Odessa, Fla, USA, 2003.View at: Google Scholar
N. Lifshitz and G. Tubul-Lavy, “Questionnaire for Assessing Preschoolers’ Organizational Abilities (QAPOA),” in Proceedings of the The Annual Conference of Israeli Speech Therapists, Jerusalem, Israel, 2016.View at: Google Scholar
E. DePoy and L. N. Gitlin, Introduction to Research, Elsevier, St. Louis, Mo, USA, 4th edition, 2011.
N. O'Rourke, R. Psych, and L. Hatcher, A Step-by-Step Approach to Using SAS for Factor Analysis and Structural Equation Modeling, Sas Institute, Cary, NC, USA, 2013.
American Occupational Therapy Association, “Occupational therapy practice framework: domain & process,” The American Journal of Occupational Therapy, vol. 68, pp. S1–S51, 2014.View at: Google Scholar