process to see that their relationships remain intact. In
verifying the model itself a special check should be made
on all fitted parameters and on the method of selecting
typical values.
VERIFICATION
Computer
Program
Model L
Validation is the trying out or testing of the abstract
system of a model to see if it yields reasonable and satis-
factory approximations to the real world. The validity of
an abstract system is determined by its ability to predict
the behavior of the real world system, and this can best
be done by comparing its output with real world data.
Since no model can be perfectly valid, the essential
question becomes how close to reality must the output be
to have practical usefulness in planning.
The output of an urban simulation model usually is in the
form either of time-series data, that is developments
over time, or patterned data, which is the distribution of
a phenomenon at a point in time. For the validation of
time-series data the analysts can apply various statistical
tests directly, focusing on the number of turning points
and their timing and direction, and the characteristics
of the distribution of output data, their amplitude, var-
iation about the mean, and other distinctive features.
Appropriate statistical methods include the chi-square
test, analysis of variance, regression, factor and spectral
analysis and nonparametric tests.
More elaborate procedures are required for the validation
of patterned data. A technique for describing spatial
phenomena must be worked out before the analyst can
apply statistical techniques. The fields of animal and
plant ecology suggest some methods for this. A typical
approach is to plot the phenomena as a series of dots on
a map, both the simulated and the corresponding real
world data, and to superimpose a regular grid over the
map. Three basic techniques can then be used todescribe
the distribution of phenomena. Counting the number of
data points in a bounded area is the simplest one. The
nearest-neighbor technique and contiguity analysis are
more sophisticated methods. After the distributions are
described by one of these methods, the statistical tests
listed in the previous paragraph can be applied.
The following diagram illustrates the steps or points at
which verification and validation theoretically can take
place. At least in principle, verification can be at-
tempted between each and every level of abstraction,
working from the top down, as it were, while validation
is essentially a comparison of the final product with the
real world.
Together verification and validation tell the analyst how
well designed and useful the simulation model is. Only
after knowing this can it be properly used as a pseudo-
laboratory for urban experiments.
ILLUSTRATIONS OF VERIFICATION AND VALIDATION
OF THE TOMM MODEL
It is practically impossible, of course, to document a
complete verification and validation of a simulation
model representing so complex a system of population,
employment and land use as does TOMM. The process of
verification goes on continuvously from the inception of
the modeling effort, through each step of abstraction,
and throughout the time of the model’s use. Validation
usually is attempted only after the model is programmed
Manageable
Set of
Relationships
General
=
Conceptual
Scheme
VALIDATION
Real World
Figure 4: Verification and Validation
Work Down the Levels of Abstraction
and by its nature can never be complete. There is no such
thing as a perfectly valid model, and every model be-
comes less valid with the passage of time, as the structure
and pheonomena of the real world change. What follows
below, then, are partial and illustrative examples of
verification and validation, with reference to the TOMM
model.
Verification
The process of verification can be conceived of aswalk-
ing down the staircase of abstraction and comparing each
level for veracity with the one below. We start by de-
bugging the model. One may look first for errors in the
computer program itself, such as variables that might be
added twice, or the use of improper subscripts. Most such
errors reveal themselves during the program run, when it
attempts to divide by zero, or generates obviously un-
realistic numbers, or simply falters and stops.
More subtle errors may have been made in translating the
model to a program. Failing to place limiting conditions
to prevent the number of households from becoming ne-
gative, failing to distinguish between continuous and
discrete variables, or forgetting to establish initial con-
ditions for a variable, are examples of such errors. These
are oversights, details the analyst normally takes as given
in his own thinking, but that the unimaginative computer
does not understand. It cannot appreciate the ridiculous-
ness of a tract of land populated by a negative number of
households.
Coming down the levels of abstraction, we next look for
errors in the logic of the modeling process itself. When
the manageable theory, or set of reldtionships, was con-
verted to a model the analyst may have failed to capture
a subtle or apparently minor but essential aspect of the
system. Such errors may reveal themselves only in special
cases, or for a very few combinations of values of the
variables, and thus pass undetected through many runs of
the model that appear reasonable. Such a logical error
was found in the TOMM model only after it had been run
repeatedly. The relationship of population density to
distance from the city’s center had not been properly in-
corporated into the model, and, under certain circum-
ARCH+ 2 (1969) H.&