Indicators & Profiles SIG
A service to members of the American
Educational Research Association
The Importance of Identifying Levels in Multilevel Analysis: An Illustration of Ignoring Top or Intermediate
Levels in School Effectiveness Research
Jan Van Damme
In educational effectiveness research, multilevel models are increasingly
used because these models take the multilevel structure of the data into
account. In multilevel research, the data structure in the population is
hierarchical (e.g. pupils in schools) and the sample data are viewed as
a multistage sample from their hierarchical population. In such samples,
the individual observations are generally not completely independent because
of selection processes and because of the common history they share by
belonging to the same group. Multilevel models take these dependencies
into account (Hox, 1994).
The identification of a level above the individual level implies the
existence of different levels of variation: if there are effects of the
social context (the group level) on individuals, these effects must be
mediated by intervening processes that depend on the characteristics of
the social context. Groups can have a direct effect on individuals (different
intercepts) or there can be cross-level interaction effects (different
slopes). The latter requires the specification of processes within individuals
that cause those individuals to be differentially influenced by certain
aspects of the context.
However, what are the criteria for choosing levels and the number of
levels? There are three kinds of criteria: the theory under investigation
(Hox, 1994; Snijders & Bosker, 1996) or the research question, the
kind of sampling used (cf. multistage sampling) (Hox, 1994; Snijders &
Bosker, 1996) or the number of units belonging to a level e.g. when there
are only three units in the highest level, it makes no sense to consider
that level in the analysis. Sometimes, none of the criteria can be used
to determine the (number of) levels. The international literature about
multilevel modeling has paid very few or no attention to the determination
of (the number of) levels nor to the effect of ignoring one or more levels
in the analysis.
Purpose of the Study
The purpose of our contribution is to explore the effect of ignoring
one or more levels of variation in hierarchical regression analysis. In
a first analysis four hierarchical levels will be considered: the individual
pupil, the class group, the teacher and the school. We will investigate
the impact of ignoring the highest level (the school level), the highest
two levels (the school and the teacher level) and ignoring one or two intermediate
levels (the teacher and/or class level) on the attribution of the variance
to the levels taken into consideration .
Data Sources and Method Used
The data set consists of longitudinal data of pupils and secondary schools
in Flanders (Belgium), studied for a period of seven years (see Van Damme
et al., 1996). We use a subsample of 2719 pupils in 154 classes, nested
within 85 teachers in 49 schools. Mathematics achievement at the end of
the common first grade is the dependent variable.
To analyze the data, the computer program MLn (Woodhouse, 1995) was
used. By manipulating the number of levels and the kind of level ignored
(top level or intermediate level) and comparing the results of these models
with the four level model, we investigated the implications of ignoring
levels of variation. The results of the four level model indicates that
each level is important: 54.39% of the overall variance in mathematics
is linked to the individual level, 14.68% to the class level, 17.38% to
the teacher level and 13.55% to the school level. Exemplary, also models
with a relevant independent variable at each level will be used to show
the effect of ignoring levels on the effect size of main effect variables.
Hox, J.J. (1994). Applied multilevel analysis: TT-publicaties.
Snijders, T.A.B., & Bosker, R.J. (1996). Introduction to multilevel
analysis. London: Sage. (in press).
Van Damme, J., De Troy, A., Meyer, J., Minnaert, A., Lorent, G., Opdenakker,
M.-C., & Verduyckt, P. (1996). De aanvangsjaren in het secundair onderwijs.
Een eerste bundeling van resultaten van het LOSO-project. Leuven: K.U.Leuven,
Onderzoekscentrum voor Secundair en Hoger Onderwijs.
Woodhouse, G. (Ed.) (1995). A guide to MLn for new users. London: University
of London, Institute of Education.