International Education Studies: Increasing Their Linguistic Comparability by Developing Judgmental Reviews

Table 2

Method for conducting the Judgmental review.

(1)	Carefully reading the items and the texts with which they occurred and getting acquainted with the processes and strategies assessed in the items.
(2)	Consulting the report on Finland’s dodgy items [29] and the national judgmental report 2000 [32].
(3)	Comparing the Finnish items and texts to the corresponding items and texts in the English and French source versions. The comparisons were made systematically at all linguistic levels (morphology, syntax, semantics and pragmatics), paying special attention to the sources of translation-related DIF presented in Table 1. The linguistic elements were compared against the cognitive processes required to respond to the items and the factors that have been found to affect item and text difficulty, such as familiarity, explicitness and transparency (e.g., [8, pages 40–4] [36–39]). The objective was to explore whether there were linguistic elements in the Finnish versions which differed so much from the corresponding elements in the source versions that they could have changed the cognitive content and/or difficulty of the items.