Comments at Census Conference on Estimates, July 19, 2006
by Steve H. Murdock
Department of Demography and Organization Studies
The University of Texas at San Antonio
I wish to thank the Bureau and COPAFS and Sabre for the invitation to attend and participate in this important conference.
I appreciate the substantial effort being made by the Bureau to examine its estimates program in light of changing data availability and demographic trends. It is obvious that not only state and local jurisdictions but also many parts of the federal establishment have a vested interest in the Bureau’s estimates program.
I, like Stan, am going to spend most of my short time discussing the Bureau’s plans so I will only briefly mention the other papers:
I liked the Salvo and Warren paper and think it makes a number of good points foremost of which may simply be the fact that assessments have always shown that reliance on a single method, as is been followed by the Bureau using IRS data, is problematic. It has never been fully clear to me why the Bureau proceeded to violate this principle when it went to a one method procedure.
I gained extensive knowledge from reading the Thomas Nardone paper because I was not aware of how major the role played by the population estimates is in the labor estimates. I must admit to being a bit disarmed by the large changes in employment that resulted from changes in the population estimates (e.g. the January 2003 estimate increase of 943,000 that resulted in a 540,000 increase in employment). It is clear BLS estimates are dependent on accurate population estimates.
Jennifer Nadans paper makes it equally clear how important such estimates are to NCHS.
Linda Gage’s remarks I found very helpful as a FSCPE member state. Her comments about the difficulty in evaluating estimates was particularly useful and I hope will be taken very seriously by the Bureau. As currently done one largely has a total estimate number with which one can only agree or disagree. One of our in-state reviewers for the estimates we produce apart from the Census summarizes his review policy in a very appropriate way. He notes “If I examine the methodology including data, assumptions, and algorithms and concur with them I must logically accept your estimates but without access to these materials I cannot properly evaluate a set of estimates and therefore cannot legitimately comment except on the basis of impressions which is, of course, an unacceptable form of evaluation. Under the latter circumstances I prefer not to comment.” His conclusions may reflect the view of many of we state reviewers of Census estimates and may suggest why so many simply do not comment. Linda’s comments relative to challenges and emphasis are also very useful and similar to those I will note below.
Dave Swanson’s paper may be the most adventuresome suggesting that the MAF could become the universal base for all our estimates work. I like the concept but share all the concerns and limitations that he notes are the criticisms of this concept. Perhaps the ACS has changed this but the updating directly before the Census has always shown dramatic changes as a result of more resources being allocated to enumerating units and I wonder whether this will not continue to be the case.
Let me now turn to my comments about the Bureau’s plans. The Bureau’s delineation of the deadlines that must be met for particular programs and needs is useful and revealing (as I discuss more fully below). Its assumptions relative to release dates, vintages, the parts summing to the whole, and its list of standard statistical and use criteria (such as parsimony), and the need to ensure consistency in methodology across units are essential but, of course, not terribly surprising. I am pleased to see that the Bureau recognizes the need to improve its methods relative to such critical areas as:
Detailed plans for how to proceed in meeting the enhancement goals. What specific alternatives will be examined, how, and in what time frames? For example, the objectives of the enhancement of international migration estimates are to produce annual estimates, improve migration related questions on ACS, and conduct evaluations to determine the best method to incorporate ACS data into population estimates. The work suggested of course begs the general question of how well ACS (or CPS for that matter) actually measures all international immigration, both authorized and unauthorized immigration. At least to many when you tell them that your estimates of the unauthorized are based on questions about respondents’ foreign born status, which you assume they answer accurately, and that your sampling techniques actually obtain a valid sample of the unauthorized there is some scepticism. Yet we in the “scientific” community seem to have no such questions, perhaps because we have no alternative. Is there no work to be done on the credibility of our use of comparisons of CIS and foreign born data? Ignoring such questions, are there no current ideas about how to use ACS data to assess immigration? What are these proposed ways and how will they be assessed?
Similarly, what are the alternatives being considered for determining the geographic distribution of immigrants which is a critical issue for subnational areas? In these and many other areas I found myself wanting more detail on this and virtually all of the other proposed enhancements. I would have liked to have seen a detailed plan of esimate evaluation and criteria for changing estimation procedures? Perhaps it is too early for such detail but such detail must obviously be determined before the evaluation begins. What is obvious, of course is that it is not too soon, given the demands that the updating of data bases will place on the Bureau after the 2010 Census. There will be limited time after the Census so the time must be made before the census.
I am also perplexed about a general lack of clarity about how ACS in general will inform estimates. Clearly it cannot be used to estimate total population but can it not help us estimate characteristics such as age, sex and race? The description of the work on “developing a new methodology for estimating subnational population characteristics does not mention ACS? Since measuring characteristics is what the long form in the past, and ACS in the Future, are to be used for is it only non-demographic factors that can be estimated by ACS? This cannot be the case since much was made by the Bureau from ACS data in the recent Katrina release about changes in race/ethnicity and other compositional patterns. I need further clarification. The impression one has is that ACS has very little utility for estimates because it is so little used in estimates. (except for vacancy rates and household size).
I am, in fact, increasingly concerned about what we do know about ACS and other census products’ comparability. The fact that ACS currently includes only the household population (although this should be remedied next year), that it is not based on usual residence like the decennial census, that it is an aggregation of monthly surveys rather than tied to a specific date, the extent to which it is duplicative of other Census surveys, etc. all seem to being only indirectly addressed by the Bureau and evaluations if they do exist are seldom shared with users. I am seeing an increasing number of uses, and in many cases I believe misuses of ACS data, in the media and elsewhere that are not being addressed by the Bureau. The Bureau has an obligation to clearly state what the ACS data can and cannot be used for and how its data should be used. Such critical information cannot be left for the users to determine.
A third concern is that there is no delineation of the resources and other administrative procedures that will be used to ensure that the evaluation, etc. actually occur to do the enhancements. What is made clear throughout the document is that the production schedule must be met and drives the process. My impression is that there is very little time for anything else so how will the time be created to ensure that an adequate evaluation occurs? I know in our state estimates we have a long list of items to be assessed but we never have enough time to get to all of them. Our staff and other resources are not sufficient. I hope the Bureau has a specific plan for making such time and providing such resources, otherwise we are likely to not have many of the enhancement actually occur and, without these enhancements the very credibility of Census estimates is at stake.
Again, let me thank the Census and let me simply tell them that all of us in the States want them to succeed in implementing their enhancements but that one of the areas in which we would like them to demonstrate their enhancements, relative to FSCPE and other stake holders participation, is in allowing us to evaluate more detailed plans and procedures for pursuing the enhancements. If you really believe we are your partners, please treat us as such and let us help you to achieve your very important goals.