RMP Pilot & Special Study Selection Process

Roles and Responsibilities | RMP Pilot/Special Study Selection Procedure

This document describes key aspects of the RMP's process for identifying and selecting Pilot and Special Studies. It identifies specific steps in this process and relates them to the RMP's multi-year planning cycle. It also summarizes important understandings among the three parties to the RMP (Regional Board, Steering Committee, SFEI), as well as the Technical Review Committee), regarding their respective roles and responsibilities in developing selecting Pilot and Special Studies.

Pilot and Special Studies fulfill an important function for the RMP. Pilot Studies are projects conducted on a trial basis in order to determine whether they are suitable for eventual inclusion in the Base Program. Special Studies are projects that will help improve monitoring measurements or the interpretation of monitoring data, in particular by elucidating cause-effect relationships. Both kinds of studies constitute the primary mechanism for responding quickly to new information and/or concerns, assessing new technical approaches, investigating particular questions that have defined endpoints, and evaluating new directions for the Base Program. By fulfilling these functions, they act as a key adaptive process for the RMP as a whole. Because of the range of issues that arise for consideration as Pilot and Special studies, it will continue to be necessary to solicit outside expertise as needed for these efforts. This benefits the RMP in the short term by ensuring that studies are performed appropriately and in the long term by broadening the scientific horizons and skills of all parties to the RMP.

A key point in the decision process described below is the determination of whether a particular Special or Pilot Study should be performed by outside investigators or by some combination of current Program participants, investigators, SFEI scientists, or Regional Board staff. Another critical point is the decision about whether a Study should be modified, somehow incorporated into the Base Program, or terminated. The following set of procedures is intended to help resolve these and other related issues in a transparent and equitable process that follows a consistent set of guidelines.

top

Each party to the RMP has a specific role to play in identifying Pilot and Special Study topics and in determining who should perform them.

Regional Water Quality Control Board

While the overall regional direction toward water quality management results from a collaborative interaction among the Board, dischargers, and other interested parties, the Regional Board has the particular statutory responsibility of articulating, and then ensuring the implementation of this approach. Since the management approach helps to determine the range of problems on which Pilot and Special Studies might potentially focus, it is appropriate that the Board also ensure that the overall set of Studies chosen for review are in accord with this approach. Specifically, the Regional Board will:

• participate in setting the overall budget for Pilot and Special Studies
• propose specific topics for consideraton
• review and comment on RFPs
• ensure that selected topics conform to the RMP's overall priorities
• actively participate in the review and selection process
• participate in midcourse progress reviews
• provide feedback to the Steering Committee and SFEI about the usefulness of completed Pilot and Special Studies.

top

Steering Committee

As members of the group that funds the RMP, the Steering Committee has a natural interest in maintaining the focus, efficiency, and effectiveness of the RMP and in ensuring that Pilot and Special studies address core RMP issues. The Steering Committee has the authority for managing the actual operation of the RMP and, with regard to Pilot and Special Studies, will:

• participate in setting the overall budget for Pilot and Special Studies
• propose topics for consideration of interest to the Program Participants
• manage the process for selecting the list of topics for consideration each year
• oversee the development, implementation, and updating of the solicitation and selection process
• oversee and participate in midcourse progress reviews
• provide feedback to the Regional Board and SFEI about the usefulness of completed Pilot and Special Studies
• solicit needed input and support from the Technical Review Committee
• help define specific questions.

top

SFEI

As the entity responsible for implementing the RMP, SFEI acts as the central focus of scientific and technical expertise. Specifically, SFEI will:

• provide input to the budgetary decisions
• develop standardized formats for topic submission (Step 1 below), concept descriptions (Step 4), and RFPs (Step 9)
• maintain an updated list of potential Pilot and Special Study topics derived from all sources
• propose specific topics for consideration
• with the assistance of the Technical Review Committee, organize, summarize, and perform an initial prioritization of proposed Studies
• draft RFPs for review by the Steering Committee and the Regional Board
• act as the contracting agent for Pilot and Special Studies
• develop and implement a management process for the technical performance of Pilot and Special Studies
• organize and manage midcourse progress reviews
• prepare summary evaluations of the outcome and overall value of completed Pilot and Special Studies
• provide for external scientific review as appropriate.

top

Technical Review Committee

The TRC consists of technically knowledgeable representatives from the agencies represented on the Steering Committee. Their overall responsibility is to provide needed technical review and input at key stages in the Study development process and to furnish the Steering Committee with an additional source of information to assist their decision making. Specifically, the TRC will:

• suggest to be added to the pool of potential Studies
• participate with SFEI in organizing and prioritizing the list of potential Studies
• provide an independent review of the Study concepts developed by SFEI for consideration by the Steering Committee
• furnish summaries, interpretations, evaluations, and other information as requested by the Steering Committee during its consideration of proposed Study concepts
• provide input to SFEI to support the development of Study plans and RFPs
• participate on proposal review panels for those Studies put out to bid
• assist in tracking and evaluating performance of Studies

top

Unlike the Base Program, which is relatively consistent from year to year and which is structured through a set of negotiations among the parties to the RMP, Pilot and Special Studies are typically one-time activities that may involve parties not normally a part of the RMP. The intent of the Study selection process described here is to:

• ensure that a broad range of ideas is available for consideration
• prioritize and evaluate these in light of the RMP's objectives and questions
• strike a proper balance between the costs and benefits of contracting with outside parties
• provide a consistent set of criteria for decision making at each stage of the selection process.

Figure 1 illustrates the overall flow of the Study selection process. Ideas for Pilot and Special Studies can come from a variety of sources (1) and it is important at this early stage to have a wide funnel to channel potentially useful ideas into the process. In addition, SFEI is likely to generate additional new questions and problems (2) as a direct result of their data interpretation efforts. All of these possibilities should be added to a raw list (3) maintained by SFEI and publicly available on SFEI's Web site to interested parties. In the summer of Year 2, SFEI should, with input from the TRC, begin to organize and prioritize this raw list (4) in light of the RMP's overall objectives as well as the current set of management questions guiding data collection and interpretation.

This prioritized list will then provide the basis for a discussion among the Regional Board, the Steering Committee, and SFEI intended to produce a set of draft Pilot and Special Study needs, priorities, and concepts (5). Following a formal review and recommendations by the TRC (6), the Steering Committee will finalize the list of Study concepts for the coming year (7). At this point, it will be necessary to determine (8) if these Studies can be accomplished by existing participants, contractors, SFEI, and/or Regional Board staff, or whether it is desirable to solicit outside parties to perform the work. For the former case, SFEI, with input from the TRC, will guide the development and implementation of Study plans. For the latter case, outside bidders will be selected through a formal solicitation process (10, 11).

The following sections discuss each of these steps in greater detail, indicating where more formal procedures and/or criteria should be used to shape the process. The desired completion dates are discussed in each section and summarized in Figure 1 below. As Figure 1 illustrates, this procedure crosses the three distinct tracks that occur simultaneously in the RMP.

top

Step 1: Generate ideas

Figure 1 shows five main sources for suggestions about what Studies should be considered for inclusion in the Program. These suggestions could arise from a variety of activities, such as data interpretation, observations during field work, review of relevant literature, existing Studies, and unsolicited suggestions from other interested parties. There need be no formal procedure at this point other than a statement to the effect that SFEI is maintaining a raw list of Study topics and is open to suggestions at any time from the community at large (scientists, managers, public interest) as well as from those directly involved in the Program. However, SFEI publishes on its Website a standard submission form that, at a minimum, concisely defines the potential Study topic, specifies how it relates to the Program's current objectives and management questions, and briefly discusses the technical issues involved. This will help streamline the process of organizing and prioritizing the possible Study topics.  

Step 2: Maintain topic list

SFEI should devote a section of its Website to soliciting topics for Pilot and Special Studies. It should include a statement of the RMP's current management questions, descriptions of topic areas of major interest, synopses of ongoing Studies, and a standard format for submitting additional topic ideas for consideration. SFEI should clearly state that Studies are selected through a formal review process and that those suggesting a Study topic are not guaranteed of performing the Study if it is selected. SFEI should also state that unsolicited proposals will be returned unread with a description of the Study solicitation and selection process.

SFEI should update a comprehensive topic list periodically to enable Program participants, contractors, SFEI, and/or Regional Board staff, as well as other interested parties, to begin comparing and evaluating the topics as they desire.

top

Step 3: Organize and prioritize list of topics

At this point, SFEI should take the lead in organizing, evaluating, and prioritizing the raw list of Study topics. The TRC should be actively involved in this process, which should include the following steps:

• SFEI will solicit any needed additional information from submitters
• SFEI and the TRC will have a preliminary, open-ended discussion of topics
• SFEI and the TRC will sort topics into those that might be included in the Base Program (need successive measurements over time), approached as Pilot Studies (need methods development or proof of concept work), or implemented as Special Studies (focused and bounded questions)
• SFEI and the TRC will separately rate each topic on specific criteria, including
• overall relevance to objectives and management questions
• the appropriate level of scientific detail needed to address the topic
• technical issues that need to be resolved prior to implementation
• probable level of effort and cost
• need for outside expertise
• SFEI will solicit the Board's views on the degree to which each topic responds to Program objectives and the current management questions
• SFEI and the TRC will meet together to discuss each topic, using the criteria listed above as guidance of developing a prioritized list of topics.

On a case by case basis, SFEI and the TRC will determine whether to apply these criteria in a subjective or a more formal quantitative fashion, documenting the factors that entered into their decision.

This priority list should have a three-year time horizon and an important part of the prioritization process will be to identify which Studies might be performed sooner and which later.

Step 3 should be completed by June of each year in order to provide adequate lead time for the remaining steps leading to contracting and implementation (see Figure 1).

top

Step 4: Develop Study concepts

Based on the prioritized list of possible topics, SFEI will prepare a set of more formal descriptions of concepts or issues. These will form the basis for Pilot and Special Studies in the coming year and in subsequent years. These concepts need not correspond one-to-one with individual topics on the list developed in Step 3. Instead, these concepts may incorporate more than one topic or may use one or more topics as a starting point. At this point, the Regional Board may decide, based on its unique authority, that one or more particular issues should be included on the prioritized list. The goal of this step is to make the transition from a simple list of independent Study topics derived from a range of sources to a set of coordinated and concisely stated Study concepts that fit within the Program's Five-Year Plan.

An integral part of each Study concept should be a statement of its relationship to the Program objectives and the current management questions. Further, each should also include a description of the anticipated duration, a tentative budget for the Study, and decision points regarding when the Study should be considered complete and/or included in the Base Program.

Step 4 should be completed by June of each year (Figure 1). While based primarily on results from previous years, this timing will permit the RMP to take advantage of the data analysis and synthesis underway in the current year.

top

Step 5: TRC and External Scientific Review

The TRC will provide an independent review of the Study concepts developed in Step 4. The goal of this step is to prepare background information that will help the Steering Committee's decision making in Step 6. The TRC should evaluate each Study concept in detail, identifying potential benefits and pitfalls, evaluating and improving tentative budget estimates, and soliciting specific information needs from individual members of the Steering Committee. In particular, the TRC will ensure that each concept includes a description of how the problem, and the approach to it, reflects an awareness relevant to system processes and mechanisms. This conceptual model can be brief, but should provide the basis for later decision making about specific aspects of study design such as site selection and sampling frequency.

The TRC will identify any potential conflicts of interest stemming from the involvement in the review process of parties with a financial interest in specific Studies. While such parties can and should be involved in the discussion of such Studies, they should recuse themselves from the preparation of the final review document for such Studies.

In order to assure that study concepts are scientifically sound, external scientific review may be solicited, as necessary.

Step 5 should be completed by October of each year and should take full advantage of the current year's data analylsis and interpretation efforts.

top

Steps 6 & 7: Steering Committee decision

In this step, the Steering Committee will select a certain number of Studies to include in the Program plan for the coming year. The Steering Committee will consider information provided by both SFEI and the TRC in its discussion and will base its final decision on consensus. It may use a variety of additional rating or evaluation methods to help in its decision making. In particular, the Steering Committee will consider the completeness and accuracy of each Study's budget estimates and their possible impacts on the Program's overall budget.

Once specific Studies have been selected for the coming year, the Steering Committee will decide, for each Study, whether it will be performed by parties involved in the Program, or whether it will be put out to bid. The factors involved in this decision will differ somewhat from case to case, but will typically include issues such as the:

• need for expertise that is unavailable within the Program
• opportunity to significantly enhance the intellectual content of the Program
• availability of lower-cost services
• potential for building and/or enhancing valuable relationships with others
• existing workloads of the parties already involved in the Program.

As in Step 5, parties with a direct financial interest in specific Studies will recuse themselves from the final decision about such Studies. Steps 6 and 7 should be completed by January of each year, in order to provide sufficient lead time for the preparation of RFPs and coordination with the contracting track (see Figure 1 in Data Interpretation Procedure).

top

Steps 8 & 9: Develop Study plans/RFPs

SFEI, with necessary input from the TRC and the Steering Committee, will develop Study plans for those projects to be performed by current parties to the Program and RFPs for those to be put out to bid. In both cases, the plan/RFP should include at a minimum a concise statement of a limited number of objectives, milestones, deliverables, expected budget, and evaluation criteria. Where particular technical methods must be used, these should be specified. In particular, SFEI should give careful consideration to whether the desired work can be performed within the budget and time constraints. Where they do not believe this can be achieved, they should inform the Steering Committee, which may then reconsider the goals for the Study. Final drafts of the plans/RFPs will be reviewed and approved by the Steering Committee.

Steps 8 and 9 should be completed by March of each year, in order to ensure coordination with the contracting track (see Figure 1 in Data Interpretation Procedure).

Step 10: Evaluate proposals

Proposals will be evaluated by a small workgroup drawn from SFEI, the Steering Committee, and the TRC. The workgroup will discuss their respective interpretations of the evaluation criteria to help ensure that their scoring will be relatively consistent. The workgroup members will then evaluate each proposal independently and meet to discuss and consolidate their evaluations. They will compute an average score for each proposal and, combined with appropriate subjective criteria, use these as the basis for selecting a winning bidder.

Step 10 should be completed by June of each year, in order to ensure coordination with the contracting track (see Figure 1 in Data Interpretation Procedure).

top

Steps 11 & 12: Implement Studies

At this point, Studies will be integrated with the RMP's normal contracting and implementation process. Studies will be implemented beginning each January as an integral part of each year's work plan. SFEI will monitor the progress of all Pilot and Special Studies and report on their progress to Steering Committee at least quarterly.

Step 13: Analysis, interpretation, and reporting

Following implementation, results of the Pilot and Special Studies should be evaluated in accordance with the data interpretation plan. In particular, the Steering Committee, with input from both SFEI and the TRC, should make explicit decisions about, first, whether and how Pilot Studies should be incorporated into the Base Program and, second, whether results of Special Studies suggest changes to the Base Program and/or further Studies. External scientific review may be solicited as necessary.

top

Figure 1. Relationship of the Special/Pilot Study selection process to the RMP's multi-year planning cycle. Specific steps are tied to the contracting cycle for the Base Program and the availability of data analysis and synthesis results. (See also Figure 1 in the Data Interpretation Procedure.)

top