Five-Year
Program Review Summary
In
early 1997, seven independent scientists and specialists skilled
in matters pertaining to monitoring design, data analysis, quality
assurance, and science administration and management evaluated the
Regional Monitoring Program (RMP). This external program review
was part of the initial program design. The purpose of the review
was to examine the technical underpinnings, structure, function,
and performance of the RMP and its staff, contractors, and administrative
structure. The following is the Executive Summary and the Conclusions
and Recommendations for Implementation from the panelπs Final
Report.
The
Regional Monitoring Program for Trace Substances in the San Francisco
Estuary (RMP) has successfully produced high-quality data on chemical
contaminants and their toxicity throughout San Francisco Bay. Since
its inception in 1993, it has combined shared support, direction,
and participation by regulatory agencies and regulated organizations/industries
in a model of collective responsibility. As a result, it is developing
an expanding database of information that has helped to address
important decision-making needs of regulatory agencies and other
program Participants.
This
report presents the findings and recommendations of an in-depth
review of the RMP carried out during its fifth year of operation.
This review was an integral part of the program's initial five-year
plan and was carried out by a panel of nationally recognized experts
in a range of fields. Its objectives were to
õ determine
the successes and shortcomings of the RMP,
õ identify
parts of the program that should be retained or amplified to
maintain performance at a high level, and
õ suggest
changes or additions to meet present and future needs.
The
RMP has faithfully addressed its guiding objectives and has achieved
notable successes during its first five years of operation. These
include:
õ Establishing
and carrying out a large, complex technical program with few,
if any, problems.
õ Gathering
extremely high-quality data that describe the present state
of the Bay.
õ Producing
data that have been used in a variety of environmental management
decisions by regulatory agencies, dischargers, and industry.
õ Establishing
a climate of cooperation and a commitment to participation among
an extremely wide range of regulators, dischargers, industry
representatives, and scientists.
õ Fostering
the involvement of other government and academic scientists
with valuable knowledge and expertise.
õ Preparing
and widely disseminating thorough and accurate yearly reports
on the program's data and accomplishments.
õ Implementing
a thorough quality control system for laboratory analysis and
data management.
õ Setting
up a World-Wide-Web site to make the program's data more widely
available to potential users.
As
a result of these successes, the Review Panel found widespread support
for the RMP, many instances of its usefulness, and a firm commitment
that it should be continued for at least another five years.
The
Review Panel also found, however, that these very successes, along
with five years' experience and the benefit of hindsight, have raised
serious issues that must be addressed if the RMP is to fulfill its
potential. The Review Panel believes that complex programs such
as the RMP must continue to evolve in response to their users' needs
if they are to avoid the ¯monitoring trap” of simply collecting
data for its own sake. In the RMP's case, two core themes consistently
arose in the evaluations the Panel carried out from a variety of
perspectives (basic objectives, study design, data analysis, information
management, organizational dynamics, and management).
The
first theme is the need for more detailed definitions of all aspects
of the RMP, in particular of
õ core
program objectives,
õ specific
management and scientific questions needed to focus study design
and data analysis,
õ the
roles, responsibilities, and authorities of all parties to the
RMP,
õ decision-making
processes, and
õ methods
of identifying and resolving healthy conflict.
The
RMP's original objectives provided effective guidance during the
program's early years. However, at present they are not sufficiently
detailed or specific enough to effectively focus the program's efforts
on management's key information needs. As a result, much of the
current data analysis, interpretation, and reporting is diffuse
and not particularly relevant. Similarly, the program's commitment
to consensus-based management has helped build an important degree
of involvement and commitment on the part of all parties to the
RMP. On the other hand, it has also resulted in an inability to
directly address important issues, such as developing more detailed
objectives, where there is disagreement among some of the parties.
The Panel recommended that the RMP make it a high priority to address
the issues listed above as part of developing a new five-year plan.
The
second theme is the need for the RMP to broaden its scientific horizons
in order to increase the usefulness of its results in decision making.
The Panel strongly recommended that the RMP undertake modeling and
analysis to place the RMP data in the context of other data from
San Francisco Bay. In particular, historical data can provide a
larger perspective within which to interpret the relatively short
time series of data developed to date by the RMP. These other datasets
represent a valuable resource that is currently being under-utilized.
In
addition, the Panel recommended that the RMP address a wider range
of fundamental scientific issues that are key to any attempts to
interpret the implications of the RMP's monitoring data. These issues
include such questions as the annual input of key pollutants to
the Bay, the response of the Bay system to past reductions in pollutant
input, and the relationship between observed patterns and trends
of key pollutants and various kinds of sources, both human and natural.
The
Review Panel believes that such issues are not unique to the RMP
but are challenges that typically face complex environmental monitoring
and management programs. The Panel further believes that the parties
to the RMP have the commitment, understanding, and ability to successfully
meet these challenges and to continue to make the RMP a model of
cooperative environmental problem solving.
The
Review Panel outlined a large number of recommendations to improve
both the short- and long-term performance of the RMP. Some of these
recommendations require little if any additional funding and can
be implemented relatively quickly. Others are larger in scope or
more fundamental in nature and require more time and effort to implement.
These include, for example, special studies to integrate data from
other studies into the RMP and to begin developing mass-balance
models to provide a context for interpreting RMP results. They also
include efforts to clarify the roles and responsibilities of the
parties to the RMP and to develop a revised set of program objectives.
The Review Panel believes that this last set of recommendations
is of the utmost importance and should be given the highest priority.
One
of the RMP's major strengths is that the technical and administrative
personnel involved in the project believe very strongly in it. For
example, SFEI has already begun to implement many of the more straightforward
recommendations in the draft report of this review issued on 20
May, 1997. Other recommendations, however, are more difficult to
implement. They may address more fundamental and potentially contentious
issues (e.g., the development of new study objectives) or ones that
require the full involvement of all parties to the RMP (e.g., clearer
definition of roles, responsibilities, and authorities). In the
final analysis, each recommendation will be evaluated and considered
for its overall value to the program and only those considered necessary
to the program will be implemented as interest, time, and money
allow.
This
chapter provides some guidance for this evaluation and for planning
the implementation of high-priority recommendations. It documents
the overall conclusions the Review Panel derived from interviews,
analysis, and discussions with technical and administrative personnel
associated with the RMP. These in turn led to a preliminary prioritization
of the recommendations made in the body of the report and a suggested
plan for implementing the most critical ones. The Review Panel understands,
nevertheless, that it is the responsibility of the Regional Board,
the Steering Committee, and SFEI to evaluate each recommendation
and determine whether or not it should or can be implemented.
Overall
Conclusions of the Review
õ The
Regional Monitoring Program for Trace Substances in San Francisco
Bay is a valuable environmental monitoring program based on
a unique partnership between regulatory agencies and dischargers
that can serve as a model for others.
õ The
data from the RMP are of very high quality and reflect, in many
cases, state-of-the-art analysis for environmental parameters
that is unequaled in a monitoring program of this size.
õ Participants
in the RMP believe that the program is important and valuable
to them and will, in the long run, be of benefit to regulators,
dischargers, and the population of the Bay Area.
õ The
RMP has operated on a consensus management model to date. The
quality of the program can best be preserved in the future by
more specific description of the roles, responsibilities, and
authorities of the parties involved, as well as of key decision-making
processes.
õ Participants
in the RMP agree that the program should be continued for at
least another five years; a strategic plan is needed to guide
the development of the program through those years.
õ The
original objectives of the RMP served it well during its early
years; however, they are diffuse and non-specific. Study design,
field execution, data analysis, and reporting would benefit
from development of more specific objectives based upon the
needs of the Regional Board and the Participants.
õ The
overall value of the RMP can be improved by applying a greater
degree of interpretation to the data being collected, as well
as a more thorough integration into the RMP of the results from
other monitoring and research programs in the Bay area, both
past and present.
Prioritizing
Recommendations
The
Five Year Review compiled many recommendations to be considered
for implementation. Whereas some of these can be implemented immediately
and with little effort, many that focus on the objectives and design
of the program are closely interrelated and should be implemented
only following a careful consideration of their relationships.
Recommendations
for Immediate Implementation
Table
2.1 summarizes those recommendations that the Review Panel believes
can be incorporated rather easily into the RMP's operations.
With
two exceptions (recommendations 5a and 5b), implementation for all
the recommendations in Table 2.1
fall to SFEI. Recommendation 5a calls for the Regional Board to
clarify and define precisely what their responsibilities are in
the RMP. This item should receive high priority within the Regional
Board since a definition of the Regional Board's responsibilities
affects the implementation of other recommendations that directly
address the design and execution of the program.
Recommendation
5b calls for the Executive Officer of the Regional Board to request
that parties to the RMP devise a new five-year plan for the program.
That five-year plan would cover the years 1998 through 2002, and
would be the primary vehicle for implementing the major recommendations
made by the Review Panel (see below).
Most
of the recommendations for immediate implementation would have a
minor financial impact on the RMP budget. By and large they represent
slight to moderate increases in labor at the technical level. The
Review Panel believes that a different division of labor within
SFEI would aid implementation and keep financial impact to a minimum.
The Review Panel suggests that SFEI emphasize greater use of less
highly trained personnel in the more routine data processing, analysis,
and report-writing functions, leaving staff at the higher levels
to concentrate on more conceptual evaluations.
Perhaps
the most expensive of the recommendations in Table
2.1 is the expansion of the laboratory intercomparison program.
This would require that SFEI contract with additional laboratories
for chemical analysis of split samples taken from the routine sample
stream. While additional QA/QC would not necessarily improve the
overall quality of RMP data, it would improve its credibility. The
relative value of this recommendation should be weighed against
other claims on budget resources.
Recommendations
for Gradual Implementation
The
remaining recommendations fall into two main categories. The first
includes specific studies the Review Panel believes are needed to
address important scientific and technical issues. These are summarized
in Table 2.2 in a sequence that
reflects the Review Panel's judgment of their relative importance.
It is most essential to integrate data from both current and historical
studies into the RMP. This will provide the context needed to assess
sources, define impacts, and evaluate design issues such as the
potential value of using TSS to define exceedances2,
defining the seasonality of the data, and estimating the rates of
burial of contaminant-laden particles in the Bay ecosystem.
The
other category consists of recommendations that go to the very heart
of the program: the design of the sampling, analysis, and interpretive
components of the RMP, and the formulation of new objectives for
the RMP. The Review Panel considers these ¯developmental” activities
the most important part of the Five Year Review report. Failure
to address and reach some reasonable resolution about these issues
would likely lead the RMP into the ¯monitoring trap” (Chapter 2,
Chapter 3 of the Final Report) of collecting data for the sole purpose
of collecting data. To avoid the regression of the RMP, therefore,
the Review panel believes that all parties should give the highest
priority to implementing the following recommendations (see also
Table 2.3):
õ To
undertake to define carefully the roles of the parties;
õ To
define the real data needs and the uses to which the RMP data
will be put;
õ To
expand the program objectives in detail (the form of the questions
asked) and scope (the conditions evaluated by the RMP and its
geographic scope); and
õ To
evaluate the design of the RMP so that it provides the data
needed to answer the questions stated in the revised objective
statement.
Implementing
the recommendations summarized in Tables
2.2 and 2.3 will require considerable
effort from all parties to the RMP. They will involve additional
committee and workgroup meetings for planning, discussion, and negotiation.
Just as importantly, the studies listed in Table
2.2 will demand additional financial resources to support new
subcontracts, or to enable SFEI to hire additional personnel to
maintain their day-to-day scientific, administrative, and management
activities as these additional studies are performed by the senior
scientific staff. The Review Panel believes that such additional
funding should be made available to initiate implementation of these
suggested studies in order of their prioritization (Table
2.2).
The
Review Panel also perceives different parties to the RMP as having
primary responsibility for implementation of these recommendations.
However, each will require collaboration among and between the Regional
Board, the Steering Committee, and SFEI. Most will require that
work plans be formulated, and that workgroups with representatives
of the Technical Review Committee be convened to evaluate the topic
and recommend actions to the Steering Committee.
Finally,
it is important to note that the full suite of recommendations for
gradual implementation (Tables 2.2
and 2.3) are interrelated. The Review
Panel suggests that the first step in implementing these recommendations
should be a critical path analysis that shows which actions must
necessarily precede others. This will assist the parties to the
RMP in analyzing the overall implications of each recommendation
and in placing them in a logical sequence for implementation and
for development of the new five-year plan.
Review
Panel:
Dr.
Donald Boesch, University of Maryland, Center for Estuarine and
Environmental Studies
Mr.
Robert Cushman, Oak Ridge National Lab., Carbon Dioxide Information
Analysis Center
Mr.
William Crooks, private consultant
Dr.
Alan Mearns, NOAA Ocean Assessment Division
Dr.
Susan Metzger, Lawler, Matusky and Skelly Engineers
Dr.
Thomas O'Connor, NOAA National Status and Trends Program
Dr.
Allan Stewart-Oaten, University of California at Santa Barbara
Review
Coordinators:
Dr.
Brock Bernstein, formerly of EcoAnalysis, Inc.
Dr.
Joseph O'Connor, private consultant
2
The RMP has begun to develop regressions between total aqueous concentrations
of many trace contaminants and total suspended solids (TSS). This
should be expanded to test the validity of using only TSS measurements
to monitor exceedances of water quality criteria. It seems that this
should be possible because invariably those exceedances are due to
high concentrations of particle-bound copper, mercury, nickel, or
PCB. These data strongly suggest that present exceedances are due
in large part to the historical pool of contaminants in Bay sediments.
The Review Panel suggests that this implication be considered in any
attempt by the RMP to link water quality patterns to current sources
of contamination. |
RMP:
RMP
Home | Program Information
| Publications | RMP
Data
