B.

The Academies' Evaluation Process and Report

By prior agreement, the Academies were asked by the DOE to assist in the SSC site evaluation process by providing an independent evaluation of the qualified site proposals against the set of requirements issued by the DOE in the Invitation on April 1, 1987, and to recommend an unranked Best Qualified List.

The Academies' assistance was sought in the interest of enlisting an independent evaluation that would further the goal of a credible and objective site selection process. It was the opinion of the Department that the Academies' participation would provide a review of the proposals that met the highest standards in light of the National Academies' reputation for fairness and objectivity.

The Academies' Committee was established in June 1987, to evaluate qualified sites referred by the DOE and to recommend an unranked list of best qualified sites. The Academies' Committee was composed of persons particularly well qualified for the site proposal evaluation, having appropriate technical, professional, and management experience. Of particular note, 8 of the 21 members have extensive personal experience in the management of large scientific enterprises.

The DOE determined on September 17, 1987, that 36 of the 43 proposals it received in response to the Invitation were qualified and forwarded those proposals to the Academies' Committee. (One proposal was subsequently withdrawn by the proposer during the course of the Academies' Committee evaluation.)

The Academies' Committee formed seven working groups (one for each technical evaluation criterion and one for the life cycle cost). Each working group included at least one member who also served on another working group. The Academies' Committee agreed to a set of written evaluation procedures which it provided to the Task Force. These procedures were discussed with the Academies' Committee staff.

The Academies' Committee determined that adopting a rigid set of weightings to allow a mechanical method of rating proposals was essentially impossible and had potential technical and practical flaws. In lieu of this, each of the Academies' Committee working groups was charged with providing an initial evaluation (good, satisfactory, and questionable) on each subcriterion of each proposal, based upon the particular criterion for which the working group was responsible, as a basis for presentation to and discussion by the full Committee. The Academies' Committee

directed its staff to develop alternative methods of aggregation as tests of the Academies' Committee evaluation methodology. These test results revealed no basis for the Committee to change its conclusions.

The Academies' Committee did elaborate on several of the elements within the technical evaluation criteria, subcriteria, and life cycle cost considerations (e.g., for geology and tunneling, groundwater inflow into the tunnel and experimental halls during construction) that were considered in the evaluation of proposals. These elements are identified in the Academies' Committee report as "those items within the DOE-announced criteria and subcriteria and their relative importance, that are likely to be most critical in determining scientific productivity of the SSC laboratory." Specifically, "Because the SSC will be a very large national laboratory, its ability to recruit and retain a first-class staff is of utmost importance to its scientific success." Given the extensive experience in the management of large scientific enterprises represented on the Academies' Committee, it was well qualified to judge factors crucial to determining scientific productivity.

The Academies' Committee received and discussed reports from its working groups on the strengths and weaknesses of each of the 35 proposals during its final meeting. In order to make an explicit decision to include (or not to include) each site on the Best Qualified List, the Chairman asked that the full Committee discuss each site until a consensus was reached that it should or should not be placed on the Best Qualified List. The recommended Best Qualified List is unranked and at no point did the Academies' Committee consider what would be an appropriate number of Best Qualified List sites. Geographic distribution was not a factor in the Academies' Committee decision regarding the Best Qualified List, nor did the Academies' Committee process limit Best Qualified List sites to one per state.

The Academies' Committee report, "Siting the Superconducting Super Collider," which was forwarded to the DOE on December 24, 1987, identified eight sites that "merited inclusion" on the Best Qualified List. Listed in alphabetical order, the recommended Best Qualified List sites are:

Arizona (Maricopa)

Colorado

Illinois

Michigan (Stockbridge)

New York (Rochester)

North Carolina

Tennessee

Texas (Dallas-Fort Worth)

The Academies' Committee expressed full confidence that the recommended Best Qualified List represented the best collective judgment of the Committee (whose members were carefully chosen for their expertise and impartiality), and reflected a selection of those sites that best meet the selection considerations included in the Invitation.

Before release to the DOE, the Academies' Committee report was reviewed and approved by a committee of the National Research Council Governing Board to assure that it met their quality and content standards. It was formally transmitted to the Acting Director of the Office of Energy Research by the Presidents of the

Academies.

The Task Force, under the auspices of the Acting Director of Energy Research, was directed to review and validate the Academies' Committee report and recommended Best Qualified List upon receipt to determine whether the Academies' Committee evaluation process gave appropriate consideration to the technical evaluation criteria and subcriteria, and their relative importance, as well as cost considerations consistent with the Invitation.

In its review and validation of the Academies' Committee report, the Task Force has sought specifically to determine whether:

The Academies' Committee complied with the requirements of the
Statement of Work of its contract with the DOE;

The results reflect a fair selection process, evaluating sites on their technical merit and cost considerations as described in the Invitation;

The Academies' Committee has recommended a credible Best
Qualified List for siting the SSC.

Prior to receipt of the Academies' Committee report, the Task Force discussed, in depth, each proposal in order to arrive at a more complete understanding and familiarity. The Task Force proposal review was accomplished by assigning lead responsibility for each technical evaluation criterion and the life cycle cost to an individual Task Force member. In consultation with Task Force members and advisors, as appropriate, each lead member assessed each proposal in his respective area of responsibility. Additionally, a summary presentation of the life cycle cost results was made to the Task Force by the DOE contractor, RTK, a joint venture of Raymond Kaiser Engineers, Inc., Tudor Engineering Company, and Keller & Gannon-Knight, on December 14, 1987.

The Acting Director of the Office of Energy Research received the Academies' Committee report on December 24, 1987, and copies were distributed to members of the Task Force. The members assessed the report during the period of December 24-28, 1987.

The Task Force met on December 28, 1987, to formulate questions to be discussed with the Academies' Committee staff. The questions covered the following areas: the Academies' Committee evaluation procedure; the application

and relative importance of the technical evaluation criteria, subcriteria, and cost considerations; the basis and rationale for determining the Best Qualified List sites; and a review of the significant characteristics of each of the qualified proposals.

On December 29, 1987, the Task Force had an all day meeting with the Academies' Committee staff members, including the Project Director, Project Manager, and Staff Officers. The Project Director outlined the evaluation methodology followed by the Academies' Committee in their evaluation process.

This process included the requirement that the Academies' Committee reach a consensus that a site was to be included or not to be included on the Best Qualified List. The Academies' Committee staff confirmed that the Committee evaluation was consistent with the guidance specified in the Invitation with regard to the relative importance of the technical evaluation criteria and cost considerations. The staff answered all questions raised and provided elaborations to the Task Force, where appropriate.

During the briefing by the Academies' Committee staff, the Task Force was provided with a description of the significant characteristics of each qualified site as discussed by the Academies' Committee. The Academies' Committee considered each technical evaluation criterion and the life cycle cost estimates in its evaluation of sites. The two most important technical evaluation criteria, as listed in the Invitation, i.e., geology and tunneling and regional resources, provided substantial discrimination among the sites. Some sites had characteristics in the environment criterion that distinguished them from other proposals, but as the Academies' Committee noted in its report, "no site proposal presented environmental problems that could not be prevented or minimized." There were some differences among proposed sites with respect to the setting and regional conditions criteria, but no site had negative setting or regional conditions sufficient to warrant its elimination on these bases alone. The last of the technical evaluation criteria, utilities, provided little discrimination among proposals. Life cycle cost estimates, as calculated for the DOE by RTK, did play a role in the Academies' Committee evaluation, but that role was minor because the estimates fell within such a narrow range.

The Task Force met on December 30, 1987, to review results of its meeting with the Academies' Committee staff and in subsequent days to prepare its evaluation of the Academies' Committee report.