Incoming COPAFS Chair Maurine Haver started the meeting with comments on the previous night’s reception for retiring Executive Director Ed Spar, and thanking Linda Jacobsen, Ralph Rector and the others who worked to arrange the event.

Board member Judie Mopsik then introduced the slate for the 2013 COPAFS board as follows:
Chair                                    Maurine Haver (NABE)
Vice Chair                             Linda Jacobsen (PRB)
Treasurer                              Chet Bowie (NORC)
Secretary                              Ken Hodges (PAA)
Past President                      Felice Levine (AERA)
Board                                   Judie Mopsik (Lewin Group)
Board                                   Daniel Newlon (AEA)
Board                                   Stephen Tordella (Decision Demographics)
Board                                   Andy Weiss (Abt Associates) 

A motion to accept the slate was made and seconded, and the Board was voted in by acclamation.

Update on Recent Developments and Council Activities

Kitty Smith, who became Executive Director as of October 1, described the meetings she has had with COPAFS members – asking how COPAFS can best serve them, and what kinds of interactions with federal agencies they would find most helpful. She also has met with the heads of many of the statistical agencies, and looks forward to meeting with the rest. Smith’s objective is to promote links between COPAFS members and the federal statistical agencies – something she described as especially important during this “scary time” for the federal statistical system.

Smith described the recent FCSM meeting, and said she will post links to the presentations, and a video of Bob Groves’ address, on the COPAFS website. Dates for the 2013 COPAFS Quarterly meetings are March 1, June 7, September 20, and December 6. Smith hopes some of these meetings can focus on themes of current relevance (such as confidentiality), and encouraged COPAFS reps to suggest meeting themes. She also expressed appreciation for being hired three months prior to Spar’s departure – as she has learned a tremendous amount from him during the transition.

NASS Methodological, Operational and Structural Transformation
Cynthia Clark. National Agricultural Statistics Service

Clark noted that NASS has experienced little change over a long period of time, and after joining as Administrator in 2008, she saw the need to bring about change in more than a research program. It was clear she would have to address issues related to methodology, operations, and structural change.

In order to spark transformational change, Clark said one must first create the impetus for change, and she noted that the panic associated with continuing budget resolutions can be an impetus, as they force one to confront the realities of program and staff reductions. They force one to improve operational efficiency – reducing the cost of data collection while maintaining data quality, and enhancing career opportunities for staff. Clark noted there was also the need to shift the organizational culture at NASS from status quo to innovation. This has been difficult for some at NASS, but she has identified senior leaders to champion improvements in areas including research initiatives, information technology, centralized databases, CAPI interviewing, and the centralization of activities such as frames maintenance.

Clark also described the need to look beyond operational efficiencies to long range goals consistent with the principles of a federal statistical agency. Among the long range goals are to provide more opportunities for staff development, align programs with traditional and developing agricultural needs, and to increase data products to support changes in agriculture.

In the past, Clark noted that NASS recruited people with expertise in agriculture, and would teach them about statistics. Now they are looking for people with expertise in statistics, and will teach them what they need to know about agriculture. And the intent is to have academics make greater contributions to the NASS research agenda. These are just some of the ways NASS is changing how and where it does business. Others include increased use of teleconferencing, a centralized LAN, and the use of wireless broadband tablet technology for field data collection.

Structural changes include a streamlined decision making process with a flattened management structure, and centralized budget and human resources functions. Clark credited such changes with numerous benefits – such as enabling increased interviewer training, increased dissemination capabilities, and the publication of quality measures.

To maintain the momentum for change, Clark described a proposed 2013 reorganization based on recommendations by a team within Field Operations, and reflecting anticipated budget cuts. The plan calls for field operations to be distributed across just 12 regional centers – a sharp reduction from 46. The plan also calls for establishing a Methodology Division that will centralize all operational survey and statistical methodology work.

The changes described by Clark reduce costs associated with training and meetings, duplicative systems, and the large field staffs required before the transition to CAPI. There is also a payoff in data quality, as centralized data collection facilitates quality control, and standardized processes reduce variability. NASS expects a full return on investment by 2016, but Clark acknowledged the impact on staff. The changes create jobs requiring different skills, but eliminate many support and telephone interviewer positions.

The payoff for NASS is survival in a time of reduced budgets while maintaining successful programs. And the outcome is a streamlined, centralized organization positioned to provide flexibility in services, enhanced career opportunities for staff, and higher quality data for its users.

Looking at Presidential Polling After the Election
Clyde Tucker. Consultant

Tucker started by comparing the results of the 2012 presidential election with the average of major polls, which have smaller errors than individual polls. The final result puts Obama at 50.9 percent of the vote and Romney with 47.3 percent – a spread of 3.6 percentage points. The average of polls taken the week before the election showed a narrower race, with Obama at 48.8 percent and Romney 48.1 percent – a spread of only 0.7 points.

Individual polls showed the following.
1.0 R
1.0 D
1.0 R
ABC/Wash Post
3.0 D
1.0 D
Pew Research
3.0 D
The ABC/Washington Post and Pew polls were closest to the actual result, and Tucker noted that with the electorate so polarized, many elections are very close, so polls can be off by just a few points, and get the outcome wrong. For example, the Democrats won a number of key “battleground” states where polls favored Republicans. In others, polls showed a Democratic win, but by a smaller margins than the final result. The tendency was to underestimate the Democratic vote, and Tucker argued that it was not just better turnout by the Democrats, but a tendency for polls to miss Democratic voters.

As evidence Tucker cited cell phone only households, which are about one third of all households (more if one counts “cell mostly”). Polls have a difficult time reaching these households, which tend to be young, and more likely to vote Democratic. Increasing non-response rates are another factor making it difficult for polls to reflect election results. Contributing to the decline have been the growing numbers of single-person households, and technologies such as answering machines and caller ID.
If non-respondents did not vote, missing them would be of little consequence. There is evidence they are less likely to vote, but non-response is becoming widespread, and there is concern that attempts to compensate with weights can introduce unknown errors.

The determination of likely voters is critical, and based on screening questions on political interest, voting history, length of residence, and intention to vote. Based on the screeners, voters are assigned a probability of voting, and Tucker cited results finding that screens help produce better results. But he cautioned that this is not always the case, as likely voter numbers can be volatile. The allocation of undecideds – another component – can be handled in three ways: 1) dividing evenly among the candidates, 2) assigning based on the proportions of decided voters, or 3) model based assignments based on voter characteristics. The best approach can vary from election to election. A problem with evaluating polling methods is that any improvements need to be tested over several elections, and dynamics often change from election to election.

Tucker also mentioned “house effects” or differences tracing to factors such as political bias, differences in methodological sophistication, funding, and data collection modes. He described the “tyranny of electoral choice.” Election polls are only one of the many products of surveys, but one of intense interest involving a binary choice and a constrained distribution. Due to polarization, each party will get at least 40 (maybe more like 43) percent of the vote, so a poll with maybe 1,000 respondents and a margin of error of +/- 3 percent is trying to determine a very small percent of the vote. And in election polling, there is a definitive right versus wrong verdict.

Tucker concluded with a brief discussion of methods offered by Nate Silver at the New York Times and Drew Linzer at Emory University. Both correctly predicted the 2012 electoral vote margin by relying in part on averages of polls in individual states. Both methods rely on simulations of model predictions to establish a probability distribution for determining the chance that each candidate has of winning.

In response to a question about the implications of early voting, Tucker noted it used to be largely among old and Republican voters, but we now see increased early voting among Democrats. He suggested the greater impact might be on election night media coverage, as the immediate release of early voting numbers (when polls close) impacts the pace of what is known and reported. In closing, Tucker observed that despite the many challenges, telephone surveys did well in predicting the 2012 presidential election outcome – even on a state by state basis.

Comment from COPAFS Chair

Before breaking for lunch, incoming COPAFS Chair Maurine Haver remarked on the continued lack of a commissioner for the Bureau of Labor Statistics. She described a National Association for Business Economics letter advocating for Senate action, and encouraged COPAFS reps to consider the importance of this issue.

Launching Adaptive Design at the U.S. Census Bureau
Peter Miller. Census Bureau
Michael Thieme. Census Bureau

Miller described Adaptive Design as an approach the Census Bureau is taking as it needs to save money while at the same time dealing with declining response rates. As former Census Director Bob Groves stressed, the Census Bureau cannot keep doing what it had done before. Adaptive Design is an alternative to what the Census Bureau had done before. The idea is to identify phases in survey data collection – start with Phase 1, reach “phase capacity,” then do something else in Phase 2, and so on, adapting survey designs to new realities.

The Census Bureau has established a Center for Adaptive Design that works in three areas. The first is research – experimentation and testing to prove that adaptive design principles can be effective at increasing efficiency. The second is outreach and education – using research results to communicate and demonstrate to stakeholders that adaptive design is a viable tool for increasing census and survey efficiency. The third is design and build – building and deploying systems that enable near-real-time cost-error trade-off decisions in data collection operations.

An important first step is to assemble the data needed for Adaptive Design – what Miller called “paradata” on things like contact histories with specific respondents. The objective is to couple survey response and address frame data to enable mode switching during the course of data collection. With such information, one could know in advance that certain respondents will never respond by mail, but will respond to another mode. Using that mode first would increase efficiency and save money.

With this approach, one can also categorize non-respondents – identifying those whose response would have the biggest impact on resulting data. If one finds that adding many more responses in a certain mode would make little difference in the resulting data, one could redirect resources to responses that would have a greater impact. This approach is a departure from traditional practice of collecting data until the money runs out.

Another step is to establish and test “business rules” to guide mode switching, resource allocation, and other adaptations. Business rules of this type are currently being tested in the National Health Interview Survey, National Survey of College Graduates, and others. The Census Bureau also has been doing trainings for survey teams, and is working to resolve issues related to confidentiality. An overarching idea is that the survey response rate is not the be all and end all, but rather a means to the more important end of data quality. Adaptive Design is about what produces the best data.

Thieme called Adaptive Design “moneyball for surveys,” and noted the challenge of doing such work with legacy systems. He described the need to show that Adaptive Design saves money, but that it is not an exercise in cutting corners. The Census Bureau is sensitive to the need to show that Adaptive Design does not adversely affect the resulting data or the job of the analyst – that it is a more efficient way to produce a comparable product.

Among the challenges facing Adaptive Design are legislative and political considerations (such as the use of administrative data and paradata), alternative contact methods, and the early termination of data collection – something one cannot do in a decennial census. The transition from fixed to Adaptive Designs will be a challenge, so the Census Bureau is moving ahead with plans for Adaptive Design in the ACS (2015), the Economic Census (2017), and plans for the 2020 decennial census. The strategy is to explore the potential of Adaptive Design now, in order to be ready to use it when budgets decline.

Asked about the political sensitivity of Adaptive Design in the measurement of the unemployment rate, the response was that that is one reason the CPS is not one of the early surveys to adopt the approach. But Miller and Thieme stressed again the importance of selling the idea that while Adaptive Design is needed to save money, its real focus is on data quality.

Revisions to BEA’s Estimates of GDP and GDI
Dennis Fixler. Bureau of Economic Analysis

Fixler started with a pre-emptive clarification that revisions are not errors – a point often misunderstood by journalists and policymakers. Revised estimates, he explained, are made because data release schedules are fixed, and available data sources become more complete and of higher quality with the passage of time.

For example, BEA produces multiple vintages of estimates of GDP, each one incorporating better and more complete source data. Fixler described the production of three current quarterly vintages (advance, second and third), three annual revision vintages (first, second, and third annual), and comprehensive benchmark estimates (produced about every five years, and incorporating data from the economic censuses and definitional revisions reflecting the changing economy). He also described revisions to the estimates of gross domestic income (GDI). Fixler followed up describing the flow of data from sources such as BLS and IRS, and how much they can vary from vintage to vintage.

Nearly two thirds of advance estimates are based either entirely or partially on trend projections. Trend projections drop to about one quarter by the second projections, and are reduced further for the first annual estimates. Fixler then reviewed in some detail, changes in preliminary and revised data related to nondefense capital goods, business inventories, and retail sales.

Fixler reiterated that the GDP and GDI estimates are revised to improve accuracy, and noted that BEA judges the accuracy of its early estimates by whether they present the same picture of the economy as its latest estimates in terms of long-term growth rates, trends in key components of GDP and GDI, and broad cycles such as the timing and depth of recessions.

Looking at long-run patterns, Fixler noted that revisions to GDP growth have averaged less than 0.1 percent in the period 1985-2009. Revisions to the key components have been small, and the pattern of change in GDP is changed little by revisions. From 1983-2009, GDP estimates successfully indicated the direction of change in real GDP 97 percent of the time, the acceleration of growth about 72 percent of the time, and has corresponded well with cyclical peaks and troughs.

The largest revisions for GDP and other estimates are observed between the first and second annual estimates, while revisions between the second to third annual estimates tend to be smaller. There are exceptions to this pattern, but it makes sense, as the first revisions introduce important source data.

Turning to short-term patterns, Fixler said the focus now is on the performance of the three quarterly vintages. Mean absolute revisions (MAR) have been about 0.1 to 0.3 percentage points for all three vintages. MARs for current dollar GDP have been about 1.1 to 1.2 percentage points, and about 1.3 percentage points for real GDP. The focus tends to be on the introduction of new data, but Fixler noted that revisions are the combination of new data and revisions to seasonal factors. In fact, research shows that revisions to seasonal factors can dominate the total revisions – or sometimes the two can offset to a net of very little overall revision.

Fixler concluded with a look at revisions to the 2007-2009 recession – noting that revisions were unusually large, as early estimates significantly underestimated the extent of the decline. Some revisions came quickly as actual data replaced assumptions, but other revisions came later with annual revisions. The total revision of 5.1 percentage points from advance to latest is the largest downward GDP revision on record.

Discussion with COPAFS Constituencies

A question was raised about the possibility of a COPAFS blog. Kitty Smith responded that this was a timely suggestion, as the topic had been discussed at the previous day’s COPAFS Board meeting. Another question concerned the letter Maurine Haver described urging resolution to the BLS commissioner situation, and Smith agreed to send a copy of the letter to COPAFS reps. And responding to a question about the advocacy stance of COPAFS with respect to questions such as data synchronization, Haver assured that the intent is for COPAFS to be more involved in advocacy. Smith affirmed the intent, but noted the need to consider how to best distribute those efforts.

There was no further discussion, and the meeting was adjourned.

Presentations from December 7, 2012