Final Report from the Blue Ribbon Panel Conducting an External Review of the US Department of Agriculture Conservation Effects Assessment Project
Soil and Water Conservation Society, 2006
Contents
Executive Summary
Introduction
Change Direction
Strategic Resource Management Blueprint
Build the Science Base
Cautions
Suggested Citation
Soil and Water Conservation Society. 2006. Final Report from the Blue Ribbon Panel Conducting an External Review of the US Department of Agriculture Conservation Effects Assessment Project. Ankeny, IA: Soil and Water Conservation Society.
The U.S. Department of Agriculture (USDA) asked the Soil and Water Conservation Society (SWCS) to help design and implement the Conservation Effects Assessment Project (CEAP). One of SWCS’s roles was to facilitate an external, policy-level review of CEAP. The external review was intended to help make CEAP more useful, responsive, and credible and help assure that CEAP’s products will have wide utility for policymakers, program managers, and the conservation community.
The U.S. Department of Agriculture (USDA) asked the Soil and Water Conservation Society (SWCS) to help design and implement the Conservation Effects Assessment Project (CEAP). One of SWCS’s roles was to facilitate an external, policy-level review of CEAP. The external review had two primary purposes: (1) to seek, analyze, and synthesize input from future users of information generated by CEAP as a means of helping USDA understand how best to design CEAP and package its outputs and (2) to recommend new approaches or refinements of planned approaches that will enhance the capacity to produce comprehensive national assessments. The project is intended to help make CEAP more useful, responsive, and credible and help assure that CEAP’s products will have wide utility for policymakers, program managers, and the conservation community.
SWCS assembled a blue-ribbon panel of academics and leaders of nongovernmental organizations, state agencies, and tribes. The panel was constructed to represent the communities who will use, interpret, and shape opinion regarding the meaning and value of outputs from CEAP. The panel conducted its work through meetings, teleconferences, and outreach to user groups. The panel focused its work in three areas: (1) understanding the expectations and needs of eventual users of CEAP outputs, (2) scrutinizing detailed plans for CEAP and interacting with USDA staff members responsible for CEAP national assessment and watershed research projects, and (3) exploring the relationship of CEAP to agency reporting systems and similar assessment efforts undertaken by other federal and state agencies. Federal agencies involved in the design and implementation of CEAP assigned a staff liaison(s) to the panel to serve as an information source and point of contact.
The blue ribbon panel met for the first time January 12-14, 2005, in Washington, D.C. to discuss the detailed CEAP study plans USDA had provided and to interact with federal staff through intensive briefings and discussion sessions. In March 2005, the panel issued a report of its preliminary findings that summarized the panel’s initial reactions to USDA’s overall plans for building and implementing CEAP. In their preliminary findings the panel strongly and unanimously endorsed the purpose of CEAP, but recommended an immediate change in direction and emphasis for the project to ensure CEAP’s purpose is achieved.
The panel met for the second time in May 2005 to respond to a request from USDA to help identify opportunities to use CEAP to inform the 2007 farm bill conservation title debate. The first recommendation the panel made in its preliminary findings to change CEAP’s direction was for USDA to redirect CEAP in the short-term to strategically inform the 2007 farm bill. The panel issued its second report in September 2005 outlining what they considered the most promising opportunities for CEAP to inform the 2007 farm bill conservation debate.
In this final report, the panel takes up the second major recommendation the panel made in its preliminary findings for changing CEAP’s direction: “USDA must also change its stated long term goal for CEAP to ensure the program is built primarily to look to the future to enable rigorous and science-based evaluation of options to improve conservation efforts in the future.”
Panel Strongly Endorses Purpose of CEAP
CEAP’s stated purpose is to help policymakers and program managers implement existing, and design new, conservation programs to more effectively and efficiently meet the goals of Congress and the Administration. The panel unanimously endorses this purpose. A coherent and science-based assessment and evaluation system is urgently needed to ensure conservation programs cost-effectively produce the gains in environmental quality taxpayers and agricultural producers expect. The panel commends USDA for taking on this critical and difficult task. The panel hopes its work will help USDA and its partners succeed in their efforts. The panel, however, recommends important changes in direction for CEAP to ensure it achieves its purpose.
CEAP Must Change Direction to Achieve Purpose
CEAP must change direction to become the coherent, science-based assessment and evaluation system policymakers, program managers, and the conservation community urgently need. The panel found that CEAP was conceived largely as a way to supply annual, quantitative, and generalized estimates of the effects of conservation practices needed to support a suite of program-specific performance measures. The panel concluded this vision of CEAP is too limited and is likely to produce misleading information—even in the long term and after large investments of resources to quantify annual, program-by-program performance.
Solving Problems versus Estimating Effects
The panel found little value in even the best estimate of the environment effects of a conservation program unless that estimate could be (1) compared to established environmental goals and (2) linked to the ecological and economic context in which the estimated effect occurs. As one panel member put it: “What does it mean if we are told EQIP (Environmental Quality Incentives Program) reduced nitrate losses from farm fields by 80 million tons? Was that enough, should we have done more, how much more?”
Monitoring versus Simulation or Extrapolation
Uncertainties and error introduced by broad practice definitions, missing quantitative links between variability in practice application(s) and environmental effects, and the difficulty of simulating real world interactions among conservation practices in process models, will seriously impair the scientific credibility of simulated, quantitative estimates of environmental effects being produced by conservation programs. Simulations and extrapolations cannot—and must not—substitute for on-the-ground monitoring and inventory systems designed to determine if anticipated conservation and environmental benefits are being achieved.
The panel is encouraged by changes USDA has made in its short- and long-term plans for CEAP during the course of the panel’s deliberation. The panel commends USDA for making such changes and urges USDA to take additional steps to ensure the investment in CEAP produces the credible, science-based assessment system so critical to the future of USDA conservation programs.
CEAP Must Inform Strategic Resource Management
CEAP should be built to answer the question “What should we do next year?” rather than “What did we do last year?” The panel recommends two critical roles for CEAP: (1) CEAP should become an integral part of a larger, collaborative, and ongoing system to inform and adapt strategic resource management and (2) CEAP should define and test the science-base for adaptive management of conservation programs.
The panel is also acutely aware that it is recommending a significant change in CEAP and recommending a strategic resource management system be built at a time when agency budgets are tight and likely to get tighter. The panel identified six opportunities to facilitate the change in direction it is recommending and to reduce the cost and increase the efficiency with which such as system can be built: (1) USDA must achieve a broader consensus on the purpose and future direction for CEAP, (2) CEAP must expand and strengthen collaboration, (3) Congress should update and reauthorize the Soil and Water Resources Conservation Act of 1977 (RCA), (4) the strategic resource management system should initially focus on a few critical environmental goals, (5) the system should look more to regional- rather than national-level assessments, and (6) more weight should be given to strategic components when evaluating program performance.
Blueprint
The strategic resource management system envisioned by the panel must be able to accomplish six tasks: (1) construct and update the conservation baseline, (2) set meaningful goals, (3) evaluate alternative strategies, (4) monitor program implementation, (5) monitor environmental benefits, and (6) reevaluate strategies.
Components Included in Current CEAP Plans
The panel is encouraged that USDA’s current plans for CEAP could produce several important components of this strategic resource management system. The CEAP Cropland-CRP national assessment simulation capabilities will enable large-scale estimation of the baseline effects of the conservation effort represented by USDA programs and allow for more rigorous assessment and reevaluation of strategies for employing staff and programs to meet goals—at least at large regional and national scales and in the short-term for cropland only. CEAP watershed studies will help refine and validate the methods used to simulate effects of conservation practices and programs. The panel recommends greater priority be given in CEAP watershed study plans to building the capacity to conduct regional assessments of the environmental benefits of conservation activities. Performance reporting systems, already in place in NRCS and FSA, will provide the nuts and bolts information about program implementation needed to document the current level of conservation effort supported by USDA. Efforts to more precisely geo-reference program implementation data are essential and should be the highest priority for enhancing current performance reporting systems.
Components Missing from USDA Plans
The panel is concerned that several critical components of a strategic resource management system are not included in USDA’s current plans for CEAP.
Monitoring. The most important and troubling missing piece is the absence of plans for on-the-ground monitoring of change in the environmental indicators and outcomes conservation programs and activities are intended to improve. The panel recommends that Congress mandate that at least 1 percent of the funding for each authorized program —about $40 million of the $4 billion taxpayers are investing in conservation—be set aside to support monitoring and evaluation of those programs.
Conservation Needs and Priorities. The panel did not see any evidence that provision was being made for rigorous and comprehensive identification and assessment of the extent and magnitude of environmental and resource management problems that are not being met through current conservation efforts. The panel strongly recommends that clear links be forged between CEAP staff, USDA strategic planners, and staff responsible for the RCA process. Linked staff should be charged with producing a coordinated plan for data collection, resource inventory, and resource assessment activities that will produce credible assessments of unmet needs and priorities. The Natural Resources Inventory (NRI) system, in particular, should be revisited to determine if and how the system could be revamped to produce statistically valid estimates of the extent and geographic distribution of conservation needs.
The assessment of needs and priorities must be developed in collaboration with other federal and state conservation agencies, nongovernmental organizations, and the private sector. Meaningful and cost-effective assessment can only be completed if they are based on a collaborative effort between federal entities such as USDA, the Environmental Protection Agency (EPA), the National Oceanic and Atmospheric Administration (NOAA), the U.S. Fish and Wildlife Service (USFWS), the U.S. Geological Survey (USGS); state conservation, natural resource, and environmental protection agencies, nongovernmental conservation organizations and the private sector.
Beyond USDA Programs. Many private and public sector entities contribute to the nation’s conservation effort on working land. The CEAP national survey on cropland will collect some information on such efforts, but it is not clear to the panel what plans are in place to regularly update and enrich information available about non-USDA driven conservation on working land. The panel recommends a collaborative effort among USDA and its partners to design and implement a cost-effective system to collect and consolidate information about the level of conservation effort supported by other federal, state, and local units of government, nongovernmental organizations, and the private sector.
Build the Science Base
Building the science-base for strategic resource management on working land must be a primary purpose for CEAP. The panel applauds USDA’s commitment to using CEAP to help in building this science base. The panel is particularly enthusiastic about the unique value of the watershed study component of CEAP. CEAP should support a broad, collaborative effort to create a set of "representative" watersheds or other salient geographic units in various agricultural settings on a regional basis. The network of watershed studies should be constructed in a comparative fashion to answer three broad, inter-related, and critical questions: (1) What are the most effective interventions—practices, alternative farming systems, landscape or hydrologic restoration—to achieve specific environmental benefits?, (2) Where such interventions should be placed on the landscape to most effectively achieve specific environmental benefits?, and (3) How can we design cost-effective and cost-feasible systems to monitor changes in core environmental indicators?
Panel members are acutely aware of the importance of the technical services infrastructure—research, education, and technical assistance—as the foundation of the nation’s conservation effort on working land. Even the best science base will produce few results unless technically proficient people and technical tools are available to translate that science into on-the-ground changes in farm, ranch, and watershed/geographic unit management. Moreover, rapid scientific advances are improving our ability to understand and manage the spatial and temporal variability in agricultural landscapes in ways that could dramatically improve the performance of conservation programs. Taking advantage of these opportunities, however, will require better information and people trained to use that information to more precisely apply conservation treatments at field, farm, ranch, watershed, and other salient geographic scales. An in-depth assessment of the strengths and weaknesses of the current technical services infrastructure should be among the first priorities undertaken by a strategic resource management system.