Study selection process

Unlike the Base Program, which is relatively consistent from year to year and which is structured through a set of negotiations among the parties to the RMP, Pilot and Special Studies are typically one-time activities that may involve parties not normally a part of the RMP. The intent of the Study selection process described here is to:

• ensure that a broad range of ideas is available for consideration
• prioritize and evaluate these in light of the RMP's objectives and questions
• strike a proper balance between the costs and benefits of contracting with outside parties
• provide a consistent set of criteria for decision making at each stage of the selection process.

Figure 1 illustrates the overall flow of the Study selection process. Ideas for Pilot and Special Studies can come from a variety of sources (1) and it is important at this early stage to have a wide funnel to channel potentially useful ideas into the process. In addition, SFEI is likely to generate additional new questions and problems (2) as a direct result of their data interpretation efforts. All of these possibilities should be added to a raw list (3) maintained by SFEI and publicly available on SFEI's Web site to interested parties. In the summer of Year 2, SFEI should, with input from the TRC, begin to organize and prioritize this raw list (4) in light of the RMP's overall objectives as well as the current set of management questions guiding data collection and interpretation.

This prioritized list will then provide the basis for a discussion among the Regional Board, the Steering Committee, and SFEI intended to produce a set of draft Pilot and Special Study needs, priorities, and concepts (5). Following a formal review and recommendations by the TRC (6), the Steering Committee will finalize the list of Study concepts for the coming year (7). At this point, it will be necessary to determine (8) if these Studies can be accomplished by existing participants, contractors, SFEI, and/or Regional Board staff, or whether it is desirable to solicit outside parties to perform the work. For the former case, SFEI, with input from the TRC, will guide the development and implementation of Study plans. For the latter case, outside bidders will be selected through a formal solicitation process (10, 11).

The following sections discuss each of these steps in greater detail, indicating where more formal procedures and/or criteria should be used to shape the process. The desired completion dates are discussed in each section and summarized in Figure 1 below. As Figure 1 illustrates, this procedure crosses the three distinct tracks that occur simultaneously in the RMP.

  back to top

Step 1: Generate ideas

Figure 1 shows five main sources for suggestions about what Studies should be considered for inclusion in the Program. These suggestions could arise from a variety of activities, such as data interpretation, observations during field work, review of relevant literature, existing Studies, and unsolicited suggestions from other interested parties. There need be no formal procedure at this point other than a statement to the effect that SFEI is maintaining a raw list of Study topics and is open to suggestions at any time from the community at large (scientists, managers, public interest) as well as from those directly involved in the Program. However, SFEI publishes on its Website a standard submission form that, at a minimum, concisely defines the potential Study topic, specifies how it relates to the Program's current objectives and management questions, and briefly discusses the technical issues involved. This will help streamline the process of organizing and prioritizing the possible Study topics.  

Step 2: Maintain topic list

SFEI should devote a section of its Website to soliciting topics for Pilot and Special Studies. It should include a statement of the RMP's current management questions, descriptions of topic areas of major interest, synopses of ongoing Studies, and a standard format for submitting additional topic ideas for consideration. SFEI should clearly state that Studies are selected through a formal review process and that those suggesting a Study topic are not guaranteed of performing the Study if it is selected. SFEI should also state that unsolicited proposals will be returned unread with a description of the Study solicitation and selection process.

SFEI should update a comprehensive topic list periodically to enable Program participants, contractors, SFEI, and/or Regional Board staff, as well as other interested parties, to begin comparing and evaluating the topics as they desire.

back to top

Step 3: Organize and prioritize list of topics

At this point, SFEI should take the lead in organizing, evaluating, and prioritizing the raw list of Study topics. The TRC should be actively involved in this process, which should include the following steps:

• SFEI will solicit any needed additional information from submitters
• SFEI and the TRC will have a preliminary, open-ended discussion of topics
• SFEI and the TRC will sort topics into those that might be included in the Base Program (need successive measurements over time), approached as Pilot Studies (need methods development or proof of concept work), or implemented as Special Studies (focused and bounded questions)
• SFEI and the TRC will separately rate each topic on specific criteria, including
• overall relevance to objectives and management questions
• the appropriate level of scientific detail needed to address the topic
• technical issues that need to be resolved prior to implementation
• probable level of effort and cost
• need for outside expertise
• SFEI will solicit the Board's views on the degree to which each topic responds to Program objectives and the current management questions
• SFEI and the TRC will meet together to discuss each topic, using the criteria listed above as guidance of developing a prioritized list of topics.

On a case by case basis, SFEI and the TRC will determine whether to apply these criteria in a subjective or a more formal quantitative fashion, documenting the factors that entered into their decision.

This priority list should have a three-year time horizon and an important part of the prioritization process will be to identify which Studies might be performed sooner and which later.

Step 3 should be completed by June of each year in order to provide adequate lead time for the remaining steps leading to contracting and implementation (see Figure 1).

  back to top

Step 4: Develop Study concepts

Based on the prioritized list of possible topics, SFEI will prepare a set of more formal descriptions of concepts or issues. These will form the basis for Pilot and Special Studies in the coming year and in subsequent years. These concepts need not correspond one-to-one with individual topics on the list developed in Step 3. Instead, these concepts may incorporate more than one topic or may use one or more topics as a starting point. At this point, the Regional Board may decide, based on its unique authority, that one or more particular issues should be included on the prioritized list. The goal of this step is to make the transition from a simple list of independent Study topics derived from a range of sources to a set of coordinated and concisely stated Study concepts that fit within the Program's Five-Year Plan.

An integral part of each Study concept should be a statement of its relationship to the Program objectives and the current management questions. Further, each should also include a description of the anticipated duration, a tentative budget for the Study, and decision points regarding when the Study should be considered complete and/or included in the Base Program.

Step 4 should be completed by June of each year (Figure 1). While based primarily on results from previous years, this timing will permit the RMP to take advantage of the data analysis and synthesis underway in the current year.

  back to top

Step 5: TRC and External Scientific Review

The TRC will provide an independent review of the Study concepts developed in Step 4. The goal of this step is to prepare background information that will help the Steering Committee's decision making in Step 6. The TRC should evaluate each Study concept in detail, identifying potential benefits and pitfalls, evaluating and improving tentative budget estimates, and soliciting specific information needs from individual members of the Steering Committee. In particular, the TRC will ensure that each concept includes a description of how the problem, and the approach to it, reflects an awareness relevant to system processes and mechanisms. This conceptual model can be brief, but should provide the basis for later decision making about specific aspects of study design such as site selection and sampling frequency.

The TRC will identify any potential conflicts of interest stemming from the involvement in the review process of parties with a financial interest in specific Studies. While such parties can and should be involved in the discussion of such Studies, they should recuse themselves from the preparation of the final review document for such Studies.

In order to assure that study concepts are scientifically sound, external scientific review may be solicited, as necessary.

Step 5 should be completed by October of each year and should take full advantage of the current year's data analylsis and interpretation efforts.

back to top

Steps 6 & 7: Steering Committee decision

In this step, the Steering Committee will select a certain number of Studies to include in the Program plan for the coming year. The Steering Committee will consider information provided by both SFEI and the TRC in its discussion and will base its final decision on consensus. It may use a variety of additional rating or evaluation methods to help in its decision making. In particular, the Steering Committee will consider the completeness and accuracy of each Study's budget estimates and their possible impacts on the Program's overall budget.

Once specific Studies have been selected for the coming year, the Steering Committee will decide, for each Study, whether it will be performed by parties involved in the Program, or whether it will be put out to bid. The factors involved in this decision will differ somewhat from case to case, but will typically include issues such as the:

• need for expertise that is unavailable within the Program
• opportunity to significantly enhance the intellectual content of the Program
• availability of lower-cost services
• potential for building and/or enhancing valuable relationships with others
• existing workloads of the parties already involved in the Program.

As in Step 5, parties with a direct financial interest in specific Studies will recuse themselves from the final decision about such Studies. Steps 6 and 7 should be completed by January of each year, in order to provide sufficient lead time for the preparation of RFPs and coordination with the contracting track (see Figure 1 in Data Interpretation Procedure).

  back to top

Steps 8 & 9: Develop Study plans/RFPs

SFEI, with necessary input from the TRC and the Steering Committee, will develop Study plans for those projects to be performed by current parties to the Program and RFPs for those to be put out to bid. In both cases, the plan/RFP should include at a minimum a concise statement of a limited number of objectives, milestones, deliverables, expected budget, and evaluation criteria. Where particular technical methods must be used, these should be specified. In particular, SFEI should give careful consideration to whether the desired work can be performed within the budget and time constraints. Where they do not believe this can be achieved, they should inform the Steering Committee, which may then reconsider the goals for the Study. Final drafts of the plans/RFPs will be reviewed and approved by the Steering Committee.

Steps 8 and 9 should be completed by March of each year, in order to ensure coordination with the contracting track (see Figure 1 in Data Interpretation Procedure).

Step 10: Evaluate proposals

Proposals will be evaluated by a small workgroup drawn from SFEI, the Steering Committee, and the TRC. The workgroup will discuss their respective interpretations of the evaluation criteria to help ensure that their scoring will be relatively consistent. The workgroup members will then evaluate each proposal independently and meet to discuss and consolidate their evaluations. They will compute an average score for each proposal and, combined with appropriate subjective criteria, use these as the basis for selecting a winning bidder.

Step 10 should be completed by June of each year, in order to ensure coordination with the contracting track (see Figure 1 in Data Interpretation Procedure).

  back to top

Steps 11 & 12: Implement Studies

At this point, Studies will be integrated with the RMP's normal contracting and implementation process. Studies will be implemented beginning each January as an integral part of each year's work plan. SFEI will monitor the progress of all Pilot and Special Studies and report on their progress to Steering Committee at least quarterly.

Step 13: Analysis, interpretation, and reporting

Following implementation, results of the Pilot and Special Studies should be evaluated in accordance with the data interpretation plan. In particular, the Steering Committee, with input from both SFEI and the TRC, should make explicit decisions about, first, whether and how Pilot Studies should be incorporated into the Base Program and, second, whether results of Special Studies suggest changes to the Base Program and/or further Studies. External scientific review may be solicited as necessary.

back to top

---

Figure 1. Relationship of the Special/Pilot Study selection process to the RMP's multi-year planning cycle. Specific steps are tied to the contracting cycle for the Base Program and the availability of data analysis and synthesis results. (See also Figure 1 in the Data Interpretation Procedure.)