stances, the model tended to force people out into the
suburbs, creating a wholly unrealistic population distri-
bution.
Unfortunately for the process of verification, logical
errors tend to be the hardest to find and their searching
out forces the analyst to rethink theoretical relationships
about which he may feel unsure. Less critical program-
ming errors, on the other hand, tend to pop up for
attention immediately when the program is run. To over-
come this perversity and get down to the heart of the
verification, the only recourse is a long series of runs
with a working model. The analyst experiments with the
model through repeated runs until he is confident he can
anticipate its behavior. He then introduces a carefully
selected and measureable change and compares the out-
put with that of previous test runs. When he is sure he
understands the behavior of the model under the first
modification, he adds a second and reruns, and then a
third, and so on. Should the model prove inadequate in
any of these comparison tests, it is changed accordingly,
and in this way it is possible for a model to emerge from
a prolonged process of verification as a quite different
model from that which started.
Validation
The validity of an urban simulation model is determined
by the accuracy with which it can predict the future.
Only the passage of time can tell this. But even com-
paring the model’s forecast with eventual reality is not
a true test of validity. Conditions may arise over time,
that are not included in the original theory of the model
to make its forecasts inaccurate, although in its first
conception the model was valid. Or, conversely, a
poorIly conceived model may give an accurate forecast
by accident. It may, as it were, do the right thing for
the wrong reason. We approach the process of validation
with feelings of inadeauacy.
Perhaps the most common method of validation is retro-
spective prediction. In most cases of large simulations
this is possible only for the years 1950 and 1960 for which
adequate U.S. Census data are available. The method
is to insert historical data for any earlier year and to
predict the 1950 data, comparing the results with the
1950 Census. The process is repeated with a 1960 pre-
diction. When the results of the two tests are compared
some insight can be gained into the validity of the mode
In validating the TOMM model this procedure could not
be used because the Census data of 1950 were not ade-
quate. The earliest employment base data available were
for 1960. An inter-Census prediction was therefore nec-
essary. TOMM accepts as its major input externally pre-
pared projections of basic employment. Fortunately such
projections were available for the period 1960 to 1966.
Using these data as inputs a projection was made of the
number and location of households among the 160 land
tracts for the year 1966. Lacking actual data on house-
holds for the year 1966, it was necessary to use substitute
data that happened to be available for the year 1967.
These had been collected through a sampling procedure
by a market research agency for quite different purposes.
It was therefore necessary to base the validation test on
a comparison of the model’s projections from 1960 to
1966 with sample-based data of actual households for the
0
year 1967, a less than rigorous procedure, to say the least
A regression analysis yielded:
TOMM-Pprojected households for 1966 = ‚8235 Actual
1967 households, with an R2 of ‚7354.
This is a reasonably good projection of the distribution. A
T-test yielded .2905, again a reasonably good figure,
considering the nature and use of the model.
When we run the chi-square test for goodness of fit we
discover the relative inappropriateness of such statistical
methods for testing urban models. With 160 degrees of
freedom (N-1, or 161-1. There are 160 individual pro-
jections and one external TOMM projection), the result
is approximately 21, a figure indicating so good a fit that
it is outside the range of most tables. Since we know that
there are discrepancies between the two distributions, the
reason for the apparently good fit must be found in a mis-
taken notion about the character of the distributions
themselves. A little reflection reveals that the gravity-
attractiveness theory underlying the TOMM model makes
all allocations of households to tracts interdependent.
The concept of degrees of freedom has thus been errone-
ously applied and the chi-square test really is not appli-
cable.
It is unfortunate that the statistical methods available for
testing the validity of urban simulation models seem to be
inappropriate to some extent. Most urban relationships
tend to be qualitative, interdependent, and somewhat
unclear. On such marshy ground statisticians had best
tread carefully.
When all is said and done, simulation models are the only
practical kind of models for use in urban planning efforts
of any size or complexity. Analytic models are limited by
lack of theory and Ey difficulties of formulation and
solution either to research applications or very narrow
applied problems. If we are going to model urban systems,
it must continue to be, for some time, with simulation
models, despite their limitations.
(1) Lowry, Ira, "A Model of Metropolis", Rand, August
1964
(2) Crecine, John P., "A Time - Oriented Metropolitan
Model", University of Michigan, paper No.
March 1969
Teplitz, Paul, Urban Analysis for Corporate Planning:
A Model of Banking Potential, unpublished DBA
Thesis, Harvard Business School, Boston, June 1969
Kilbridge, O’ Block and Teplitz, "A Conceptual
Framework for Urban Planning Models", Management
Science, February 1969
(Appreciation is expressed to Richard Corbin of the
Harvard Business School for his help in formulating the
content of this paper.)
ARCH+ 2 (1969) H.8