INFORMATION FOR INDUSTRY BACKGROUND

19 June 1996

COMPUTING ENVIRONMENT STRATCOM ARCHITECTURE (CESAR) CONTRACT

PROPOSED CONCEPTS & TIMELINES

• X - 1 Concept (Traditional) being considered

- One contractor is awarded the contract for both the planning and performance phases

RFP Release TBD

Award TBD + 3-5 Months

Planning/Performance Phase Award + 7 Years

• X - 2 - 1 Concept (Fly-off) being considered

- Two contractors are selected for multiple award of the contract and compete during the planning phase

- The contractors' proposals built during the planning phase will be evaluated and costed and one will be selected and continued for the performance phase. The other contractor's option to extend will not be exercised and he will not be continued

RFP Release TBD

Award (Multiple) TBD + 3-5 Months

Planning Phase Award + 6-12 Months

Produce Requirements (Continuous Govt Interaction)

Produce Solutions (Technical & Process)

Produce Cost Proposal

Solution Selection Planning Phase End + 1 Month

Performance Phase Selection + 7 Years

• Contractor Duties

- Establish a process for modernization and sustainment of the computing environment

- Activities include requirement analysis, solution development, provide solution alternatives to the government (with cost and technology justification)

- After government decision, acquire, install, test, operate, maintain solution. Solution includes equipment, COTS software, developed software, maintenance and technical services

- Manage/perform the computing environment process

- Integrate application software systems

- Develop application system requirements including performance

- Provide computing environment technical specifications to applications contractor - design to

- Provide computing environment for application system development, test, and production activities

NOTE: Please be aware that information you furnish as a result of this questionnaire will be on a voluntary basis and will be for informational purposes only. Information that you feel is proprietary shall not be furnished, since this is not a solicitation.

It should also be understood that the Government may use some of the information, ideas and recommendations you provide as an aid in developing the solicitation/contract. The Government will not be liable for any information obtained as a result of this questionnaire.

INDUSTRY QUESTIONNAIRE REPLIES

COMPUTING ENVIRONMENT STRATCOM ARCHITECTURE (CESAR) CONTRACT

1. It is envisioned that the scope of this contract will encompass multiple missions, multiple services, hardware/software purchases, and maintenance. Please comment on the scope.

A. Scope is broad but necessary given the overall goal of eliminating "stovepipe" subsystems. Cuts across lots of rice bowls which is understandably necessary but will make it extremely difficult for CESAR contractor team to be successful in relationships with various government functional experts. Using the conceptual framework of DII will help keep the right high level focus, but the real difficulty will be getting functional agreement on replacement hardware/software if they have to reduce some of their nearest and dearest requirements to comply with the DII. Functionally, the involved systems (software and hardware), processes, maintenance and support, are all opposing operations with very different architectural designs and futures. The complexities, different domain applications, autonomous controlling authorities, and hybrid arrangement would result in a scope too broad to articulate succinctly or manage successfully. The best way to ensure complete control and understanding of the scope of work is to break areas into more understandable parts, but keep those parts to an absolute minimum. The strategic and tactical plans of each respective directorate i.e., J2, J3, J6 should be combined, refined and contained into common USSTRATCOM goals. Thus, the CESAR should be viewed as more than a contract but also as the establishment of a Contractor-Operated USSTRATCOM organization. The government may want to consider an arrangement where multiple contracts are awarded under an umbrella contract arrangement. The key to the success of this arrangement is that a single contractor may be involved as a partner in more than one area, but a single contractor may only prime one area. Thus, you entice contractors to stick to what they do best, and offers an avenue for small business venture. Areas are Umbrella, Quality Control, Software Development, Software Maintenance, Software Testing, Hardware Maintenance, Hardware Procurement, Hardware Supply, Configuration Management Services, Training Services, LAN Support Services. Questions: Can any DoD agency/Service use this contract vehicle or is it restricted to STRATCOM only? What specific services will be included? For example, will the SOW include the need for requirements analysis, architecture development, system engineering, and training or is only focused on acquisition, installation, integration, and maintenance of h/w and s/w systems?

B. The scope is manageable, because the contractor would respond to a single organization.

C. A contract of this scope will require a well established, experienced system integrator leading a large team of contractors with complementary skills and specialized capabilities. This approach poses inherent risks. The broad scope of the program may drive the system integrator to focus on technology and integration of technical efforts rather than the business issues driving the program. This could result in program slips or cost overruns as well as an emphasis on one contractor's overall technological solution at the expense of inadequate evaluation of superior alternatives. This contract is open-ended enough to allow system integrators a vested interest in contract changes as revenue generators, greatly increasing the overall estimated cost of the program. Large systems integration companies will have the capability to perform many of the tasks required by the CESAR contract. As such, the prime integrator may view new projects as in-house tasks rather an pass the work to a sub contractor. Programs of this scope see an inevitable disappearance of the originally proposed subcontractors in favor of an increased role for the prime, with the resultant loss of the subcontractors' specialized skills and insights. Large teams performing a multitude of interdependent tasks requires extensive management. As a result, management costs for a contract of this scope can exceed the total management costs for a variety of smaller contracts. We believe that the above issues pose contract risks on broad-scope contracts. To maximize the Government's financial investment, the Government should decrease the scope of this contract and award multiple contracts that require smaller, more focused teams.

D. Agree that the scope of CESAR may be broader than the customary STRATCOM contract, but a scope of this size is not that unusual in the contracting community. Your winner should be a prime with the capability to manage the large number of subs that will be required. You must pay careful and early attention to contract tracking and management, however. One possible approach to this is to include the requirements documents that the contractor use some sort of automated system to handle the acquisition processes as they come up during the life of the contract. In addition, special attention should be given to CDRLs. An automated system to develop and track these would serve to reduce risk on an effort of this scope.

E. The scope is extremely large and ambitious. One of the primary challenges will be to meet the needs of a broad base of users, from strategic war planners to force managers to desktop computing and LAN users. The benefit of such a broad scope, which includes multi-year maintenance, is that contract stability will allow for more efficient contractor planing and a more stable work force.

F. The scope is broad but the right prime contractor teamed with strong and specialized subcontractors (including small and disadvantaged businesses) can accomplish the envisioned scope more efficiently and effectively than multiple "store-pipe" contractors.

G. Incorporating multiple missions and services will require adherence to open systems principles and will place a premium on interoperability. This vendor strongly encourages this approach.

H. As an outsourcing company, we have assumed the responsibility for the development of entire corporate I/T environments from the construction of buildings through the operation of a data center and support of customers applications and databases. There is no reason why multiple prime contractors should be needed to provide this service to you. There will be a need to subcontract out certain work, but that should not be your concern as long as the prime is meeting the agreed to service level agreements. The larger the scope of responsibility, the greater the savings to the customer. The work can be phased so all mission areas are not under change at the same moment. Eventually, the prime contractor should be able to assume complete control.

I. We believe the stated scope and associated objectives are a very cost-effective and operationally-effective way of achieving the goals and vision developed by USSTRATCOM J6. The program will offer an opportunity to realize both the maintenance of existing systems as well as the migration of those systems into a common operating environment with a firmly established way of accomplishing future developments and modernizations. The migration to a single architecture could also lead to additional efficiencies as common functions, accomplished independently for each program, (such as training, logistics, acquisition and maintenance), are consolidated and redundancies are reduced. This will provide cost savings that could then be applied to the program development allowing more work to address the primary issues. In addition, there will be obvious savings from economy of scale. For a program of this complexity, scale and highly unique applications there could be a much higher degree of risk than would be experienced for the average program. As a result, control and mitigation of the inherent risk becomes the significant factor related to the success of this program. It is therefore suggested that as this program comes to maturity all aspects focus on effectively managing the inherent risks. From a contractor point of view, our approach to providing high quality, high productivity and low risk will be to team for strength. To keep program risks to a minimum, we envision a large contractor team with contractor specialists that have established excellent USSTRATCOM presence and credentials. Our team will have professional representation in each of the program disciplines within each of the program domains. The initial team will cover the necessary functions and disciplines but also to have the structural and organizational flexibility to accommodate evolving missions, requirements and technology. Due to the complexity of such an all encompassing program, a team led by or comprised mainly of non-USSTRATCOM-experienced contractors would pose extremely high risks in all phases of the program. The typical tactic of such a team would be to suggest that they would simply hire the various incumbent engineers. To some degree this could work, but to a larger degree a majority of the current engineering talent would seek local civil employment with both higher rates and improved security/longevity. It is well known that there is a constant drain on the current local contractor community and it is becoming more difficult to keep highly specialized talent. The local contractor community has between 20 and 30 years of effective program support and management and the most significant risk to this program would be the loss of that essential pool of specialized talent. Additionally, it will be important to clearly delineate the boundaries of CESAR responsibility. For example, in the Command Center there are several communications systems that, under some definitions, may be considered computing equipment and under other definitions would not be considered as such. It would be awkward to have CESAR responsible for some Command Center hardware and another contract responsible for the rest in terms of maintenance, procurement, configuration control, engineering and QA. We recommend that the Government clearly define responsibilities for C2 and War Plans equipment as part of CESAR via a detailed equipment list and an enumeration of responsibilities. We further suggest that either CESAR be responsible for all C2/War Plans hardware or responsible for none. This responsibility could include only configuration management, engineering and operational support as is now the case for C2, or it could also include maintenance and operations support which is currently performed by government personnel. An essential element of CESAR is to maintain and modernize the current systems without impacting the mission and daily operations.

J. In general, the scope, although broad, should not pose any significant problems since the majority of the effort will continue to center around the USSTRATCOM headquarters with some efforts required from time-to-time at other remote locations. However, the Government needs to define explicitly which missions and potential work sites are "within scope" to the greatest extent possible. The breadth of the potential USSTRATCOM missions dictates that multiple services and Government agencies (USAF, USN, USA, USMC, DIA, DNA, etc.) could all be involved with determining the required support at and from USSTRATCOM (mission and requirements, data exchange, database structure, applications, models, simulations, and exercises, etc.).

K. The broad scope envisioned for the CESAR contract is appropriate to ensure that the solution provides USSTRATCOM with a truly interoperable, scaleable solution. Given the desirability for a single contractor to be responsible for end-to-end system performance, USSTRATCOM should explicitly include in the CESAR statement of work the requirement for the CESAR contractor to provide all interface specifications and standards for new applications developed during the contract's performance period. To assure compliance with these specifications and standards, the CESAR contractor should be designated as the Validation and Verification (V&V) contractor or as the independent test and evaluation (T&E) contractor for future application development programs. With respect to ongoing applications programs, the CESAR contractor must accommodate to the contracted "build to" or "design to" specifications in their proposed migration plan. To assure compliance, the CESAR contractor should be tasked with an V&V or T&E role for these on going programs.

L. Scope as written (e.g. multiple services, multiple missions, etc.) is too broad to comment on, but if the Government can write a definitive, logical RFP, Industry should be able to satisfy the requirement.

M. The government should award to a single prime contractor for all work required for successful CESAR implementation. The driving reason for this recommendation is the "tightly coupled" nature of the work to be performed and the inter-dependencies for its execution. Should the government choose to partition the work under multiple contracts (and potentially multiple contractors) the government assumes the integration risk. The scope of work to be performed under a single CESAR contract is consistent with the tasks and scope on similar modernization and migration contracts and will result in an overall low risk and low cost solution.

N. The scope of the contract is quite broad and management of the different facets of the contract will be a major task. The contract is probably too large for any single contractor.

O. As the primary vehicle supporting C4I modernization for USSTRATCOM, the CESAR contract will necessarily require a broad scope. To effectively and efficiently provide the range of H/W, S/W and services, the contract would best be structured to provide a core COE as a basic deliverable and series of ID/IQ CLINS that will provide the flexibility to task the contractor to acquire, implement and integrate hardware and software components for the COE and for interfaces to the STRATCOM systems, as well as to provide for the continued O&M of existing applications/systems, and the development of new applications as the need arises to implement reengineered Office Automation, War Planing, C2 and Intelligence processes. The ID/IQ vehicle should include CLINs for technology insertion/refresh; S/W licenses and updates; and ODCs for obtaining one of a kind items required to support integration efforts. In addition, the contract should include various CLINs for labor and services. To ensure flexibility and provide Government controls and contractor incentives, where appropriate, the ID/IQ CLINs should be structured to provide the opportunity for FFP, CPAF, LOE and T&M task delivery orders.

P. It is essential that the desired tasks under this contract be well defined. The breadth of the anticipated effort requires on site support for certain portions, while at the same time highly specialized architectural development activities may require skills that are not present in the Omaha area. A clear understanding of the requirements will enable a prime contractor to assemble a team that will satisfy the diverse activities in an optimum manner. Library documents should clearly explain the mission tasks that are envisioned for this contract.

Q. The scope of this contract is big by definition (includes SWPS, C2, LANs and supporting infrastructures). Certain advantages accrue to "big". Accountability falls on a single integration contractor. The Government only has one contractor to confer with in order to resolve problems. Theoretically, less overhead might be required to manage the single contractor. Standardization could accrue if the Government ensured that the integration contractor enforced a common set of tools and standards. This may provide some economies of scale for bulk buys of hardware and more cost-effective licensing of software. Risk could be diversified, since the integration contractor could theoretically move resources from one area to another. However, big is not always better. Risk is consolidated to one integrator. If a poor performing integrator is in place, the entire program can languish or fail. Although the Government only has one contractor to manage, management of the multiple contractors now falls to the integrator, and the Government relinquishes its direct control of subcontract performance of vital mission-essential elements. The risk now becomes whether the integration contractor can manage the multiple contracts any better than the Government. This risk will be passed on at higher cost. While there are potential economies of scale, there are also potential diseconomies of scale. As software or system effort becomes larger and more complicated, team interaction becomes less efficient; coordination becomes more complicated; and overhead functions may increase especially if it becomes necessary for the Government to increase its program oversight. From this bidder's viewpoint, the risk of scope is an important factor from another standpoint. If the size of the contract does not turn out to be as large as forecasted (such as AFCAC 281), then the return on investment is diminished. Faced with uncertain future defense budgets, the Integrator may necessarily have to offset this risk with higher than expected rates. With a large scope covering many years, the integrator expects the Government's requirements to change and evolve over time. While new work may be viewed as under the general scope of the contract, the Government can expect to negotiate new rates for sufficiently different or specific hardware, software, and services. The scope will probably discourage competition to the degree that such acquisition forces companies, accustomed to being prime contractors, to subjugate their roles to that of subcontractor which is not acceptable to many companies.

2. What approach to a total system responsibility contract performance do you envision? For example: Can this be done by a single contractor, or would it involve several contractors in a teaming arrangement? What could be done to provide opportunities for Small Business?

A. Total System Responsibility. The tasks, as described during the 19 June session, must be done by a team. Difficult to envision one contractor having the current capability to accomplish all tasks without stripping incumbent "stovepipe" contractors of their most qualified personnel. Recommend the creation of a "virtual" contractor program management office as well as a clearly stated opportunity for small business to participate through a "plant your flag" type process. The aforementioned process basically means that small business is invited to design, develop, and market products with the CPMOs knowledge and then if awarded, will be permitted to perform with minimal oversight and extremely low "wheat tax". The total system responsibility will be difficult to discern from the enormous scope being contemplated. This difficult responsibility will likely be too large for a single contractor (too much), and assuming a single award is achieved, small and medium size businesses will be excluded. The magnitude definitely suggests teamed or aligned relationships for the multiplicity. Small Business. There are opportunities for small business if multiple contracts were awarded in area such as support, testing, procurement services etc. Small business subcontract goals should be established in the RFP. The incentive provisions of the contract must be established to reward the prime contractor when small business subcontracting meets certain pre-established thresholds. What is the envisioned SB SIC code? Will the SB be based on dollars or size?

B. The ideal solution will include a team members who have (1) integration experience; (2) domain (SIOP and other SWPS missions) knowledge; (3) knowledge of all technologies and expertise in leading technologies; (4) local presence; and (5) physical familiarity with HQ USSTRATCOM. We don't believe there is a single company that meets these credentials maximally. The prime should possess as many of these attributes as possible, but a well-managed team of "experts" can do a better job of meeting all program requirements than a one-size-fits-all approach. The best way to provide small business opportunities is not to carve work out, but to set small business teaming goals and to incentivize (via award fee) the Prime to meet or exceed these goals.

C. Contract performance on this scale will require a team of contractors with complementary core capabilities that, in total, offer the full range of required services. Contracts of this size tend to fail for a variety of reason, including: (1) a focus on technology rather than the business issues driving the program (2) a vested interest in contract changes as additional revenue generators (3) the inevitable disappearance of the subcontractors' specialized skills and insights. These risks can be mitigated by decreasing the scope of this contract and awarding multiple contracts that require smaller teams. All contracts should include a cooperation clause with associated contractors as well as participation in STRATCOM IPTs. For example, the Government could release one contract for development of an enterprise database, development of a common operating environment, and management of STRATCOM'S C4I system integration and migration. This management contract would provide the Government with a single belly button for C4I integration. In addition, a variety of smaller contracts will expand opportunities for small business participation.

D. Single contractor vs. A large team. No single company is likely to have the broad capabilities required to satisfy this contract. To provide best value to STRATCOM, the prime should have capability to handle a significant amount of the actual work, but the sub contractor management capability of the prime is a critical issue for you. You will have subs that are world class performers in very narrow areas, which is good. The scheduling, monitoring, management and oversight of these subs is the important task of the prime. You should look for a strong "business-oriented" approach by the winning team. To increase opportunity for small business, you should set percentages for the portions of work to be done by small business, and reward the prime when these goals are met or exceeded. In addition, some leeway should be given to the prime to assign work to small business on a competitive basis throughout the life of the contract. Cost and risk analysis capability on the contracting team could be of considerable value to you, and should be exercised early on in the acquisition.

E. The broad scope of the proposed contract strategy will either restrict competition to very large businesses or to large contractor teams. In order to achieve viable competition, it is recommended that one significant evaluation criteria be for subcontracting goals for small and small disadvantaged businesses. Setting aside a percentage of work to be sub-contracted to Small Business should encourage teaming arrangements which can encompass the broad range of expertise required to thoroughly analyze and implement the goals of CESAR.

F. This will require a team approach to bid and execute a program of this magnitude. 55th contracting could specify a specific percentage of the total contract be given to small business but the prime contractor should be the one to select the small business and what the tasks should be.

G. This vendor believes the scope of the contract necessitates the use of a major prime contractor for overseeing the work of a multi-sub team. Such an arrangement would also facilitate the inclusion of SDBs.

H. The work effort will require a team effort with the prime contractor subcontracting to corporations with unique capabilities to meet the total demands. Capabilities provided by small businesses can be incorporated into the solution once the total requirements are determined.

I. In our opinion, this would be a near impossible program to be successfully and efficiently completed by a single contractor. Based on our thorough comprehension and appreciation of the scope, scale and complexity of the CESAR environment, we suggest that this type of an all inclusive program initiative is very achievable provided it is structured in a very business like manner. The multiple mission, multiple services, unique domains and the highly specialized disciplines and functions involved should dictate that the lead/prime contractor chosen should have extensive credentials in managing large multiple-faceted upgrade and replacement programs. To be successful, a program of this nature with the wide ranging requirements, will require a large contractor team in order to effectively cover and manage all of the requirements now and in the future. Such a team will create a requirement for a lead/prime contractor that has extensive experience in effectively managing and integrating engineering teams comprised of multiple disciplines and multiple sub-contractors. The team prime contractor becomes the keystone to program risk mitigation and program success. Such a team lead should have a high degree of competence and knowledge in both the USSTRATCOM Command & Control and War Planning Domains and be experienced at integrating operational systems without impacting the operations in order to effectively manage existing systems and future modernizations. Then, to add to the strength of the team lead, the team should be structured in such a way as to provide focused levels of experience and expertise to each of the four CESAR engineering domains. Because of the unique nature of each of the four domains, they should be led by a domain team (not necessarily the prime integrating contractor) that has experience in that particular domain. The domain team leaders would then manage sub-contractor teams that would be highly experienced in their particular domain. All of this would be coordinated and managed by the prime contractor and the senior staff of domain leaders through an IPT process. In summary, the simplest and least risk approach is for the Government to have a single prime contractor with a large team of subcontractors that spans the multiple domains and technologies. This will provide the Government with a single interface and it will clarify the relationships between performers as a formal contractual arrangement. Associate contractor relationships, while they may be necessary, typically saddle the Government with a burden of coordination and integration that is difficult, if not impossible, to discharge whenever close technical interfaces are required. Success of CESAR depends to a great extent on the selected prime contractor. The prime contractor will have to carefully select a team that spans the domains and technologies of CESAR. Then the team needs to be organized in such a way so as to provide seamless support to the Government regardless of the particular parent companies. The team organization must preserve agility to react to sudden requirements and must have the flexibility to evolve as the domains and technologies evolve. Small businesses have a role in CESAR. As a team mate, they will provide niche capabilities to ensure all domains and technologies are covered. As vendors, they become important sources of hardware, COTS software and services. We support and advocate small business participation in DoD program developments. As a matter of course, we have a solid track record of actively involving small businesses in our engineering efforts and will continue to do so for the CESAR program. From our point of view, special incentives are not necessary. However, it may be appropriate for the Government to establish a small business set aside percentage to ensure that others planning to bid this program take steps to include the small business element. Care must be taken that if a percentage participation for small business is defined, it is not so large that it raises risk or causes a small business to graduate.

J. We believe that a teaming arrangement is essential given the scope of the new CESAR contract. We have teamed with a number of other companies in the pursuit of other contracts and we don't see any reduction in the need for teaming, with the combination of War Planning, Command and Control, Intelligence, and Office Automation requirements; we anticipate a greater need to team. The key question which needs to be asked is, "will the contractor have authority to service in this single integration capacity?" Responsibility without authority simply does not work. The government cannot hold any contractor ultimately responsible for operation of systems while government personnel, military and civil service, other than the contracting officer have day-to-day influence and/or authority to direct the contractor and his teammates, subcontractors, business relationships, etc. Serious consideration must be given as to how the government intends to implement the new contract, with appropriate clauses in the contract and procedures put in place, to allow the contractor to actually become that "single belly button" described in the 19 June Industry Day briefing and to preclude personal services aspects to creep in. The approach we would recommend for the achievement of total system responsibility is for the government and prime contractor to enter an arrangement where those two parties agree on task scope, and that the prime manages the work through its subcontractors. The government should not, as a rule, engage directly with the subcontractors. Periodic program reviews keep the government informed as to program progress. As to our ability to accommodate small/small disadvantaged businesses (S/SDB), we have found that opportunities in teaming and later during program/contract execution do arise. We believe that a reasonable goal is attainable but a goal that is arbitrarily assigned disregarding local demographics will likely not be met. Therefore, the Government should establish minimum participation as a goal and permit the prime to choose the small businesses. Incentives could be levied to encourage larger S/SDB percentages. Above all, let the prime contractor choose the small businesses since the prime is ultimately responsible for contract performance.

K. Although teaming is almost certainly required to optimally perform the broad scope of work required, a single Prime Contractor with in depth experience in managing large teams of subcontractors should be responsible to USSTRATCOM for the performance of the total system. There is ample opportunity for small contractors to perform work under the CESAR contract. Clearly, office automation, requirement's definition, networking, and system maintenance are areas which numerous small contractors are qualified to provide top quality goods and services. A small business requirements of 10-12%, (5% reserved for Small/Disadvantaged or Woman Owned Business) is appropriate and should be mandated. Incentives for exceeding the requirement could easily be incorporated into an award fee criteria.

L. Based on the scope of expertise required, a teaming arrangement combining unique strengths of several companies would be best. For Small Business (SB), the Government should cut out portions best suited for SBs, or carve out and protect a specified percent of the total--like 25-35%. Experience as a SB has taught us that if a SB is not protected, the prime takes over SB work, dismisses the company, and hires their employees. Enduring contract incentives/restrictions should ensure this does not happen.

M. We envision that a single contract will be awarded to a large contractor team consisting of the prime system developer/integrator, legacy system contractor(s) and Small Business contractors. The execution of contract tasks will be accomplished through Integrated Product Teams (IPT's) that are staffed with both government and contractor personnel. We do not anticipate subcontractor/vendors being allocated End Item responsibility. Selection of personnel assigned to an IPT will be based on the resource requirement to successfully perform the task and the availability of that skill within the project team. Industry should promote the scope of work to be allocated to Small Business (SB) contractors. Typically, SB contractors specialize in a technical area. Examples of tasks we envision being allocated to SB contractors as part of the CESAR IPT's include algorithm conversion, test conduct, training development and conduct and hardware maintenance.

N. With such a large scope the best contract performance will be enhanced if there is a single major contractor responsible for the overall system design with multiple subcontractors responsible for implementing specific portions of the contract. We see teaming arrangements as almost a necessity if Small Business contractors are to be let into this process. The opportunity will be much more attractive if the teaming arrangements allow for long term participation in the process.

O. Given the broad scope of the CESAR effort and its emphasis on total systems integration, we do not believe that a single contractor can adequately satisfy the wide range of services and total system responsibility. The best approach would appear to be selection of a team of contractors led by a prime with proven experience in managing large contractor teams providing systems integration services, as demonstrated by past performance, and teammates selected to fill all of the specialty skills required to satisfy the broad spectrum functional user needs. The scope of the CESAR effort affords many opportunities for the use of small businesses. The best way to encourage their inclusion in large contractor teams is to specify goals for small and small disadvantaged business subcontracting, and provide financial incentives to the prime for meeting to exceeding those goals while maintaining the high quality of services provided.

P. While a single contractor could probably find the necessary skills across the corporation to meet the contract requirements, an optimal solution would require a teaming relationship to insure that the necessary domain expertise is available to smoothly transition from the existing infrastructure to the envisioned system. TSPR could be assumed by the prime contractor, given adequate size. In our opinion, it is not viable to spread TSPR to different associate contractors. Program scope could be expanded to include the applications programs which would insure that the prime could satisfy TSPR. Subcontracts could be responsible through the prime for individual system elements. The broad scope of the proposed contract will provide niche opportunities for small businesses with specific capabilities.

Q. The scope of this contract (e.g., C2, SWPS, LANs, mainframes, client-server, etc.) dictates that one contractor cannot handle this contract alone. Anticipate that one well-established large vendor will manage contract with multiple subcontracts. Incentives should be built in the contract to provide for sustained and meaningful Small Business participation. These incentives should go beyond points being awarded during the competition phase of the acquisition. Realize that the greatest benefits of contracting accrue to the prime contractor. Subcontractor, however, assume their share of risks, yet have minimal control, and minimal decision-level interface with the Government. As a whole, subcontractors have a much lower level of overall benefits. This can produce an environment not totally conducive to maximum performance, optimum productivity, and satisfaction on the part of companies who possess vital program and technological expertise, and on whom the Government has placed significant reliance for mission accomplishment.

3. We believe you will be required to have relationships with our application contractors. What kinds of subcontractor/associate/teaming arrangements do you anticipate being necessary to implement the integration aspect of this acquisition? Are there potential pitfalls with this type of arrangement? What would be the best means to formalize this relationship?

A. The category "application contractor" is vague. We assume it refers to those contractors currently performing application software programming activities for the SWPS and CC subsystems. As currently briefed, the best arrangement for the integration contractor is to team with an application contractor. Pitfalls abound. The candidate prime contractors must "court" the current application contractors - the one with the most on his team has highest probability of win! Solutions are few. The government could place an OCI clause on all current application contractors, leveling the playing field for the integration contractors or the government could state that only application contractors need apply. It seems obvious that these interdependent business relationships will need to be governed and controlled through a system of memoranda of understanding/agreement (MOU/MOA), nondisclosure agreements (NDA), organizational conflict of interest (OCI) clauses, and sub and associate contractor teaming agreements. The requirement for a common operating system (open architecture), infers application software will operate across the platform. Thus, special teaming arrangements should be unnecessary, unless an application structure is so incompatible that the code is suspect--then proprietary problems could become a factor. As discussed in #1, the CESAR should not be viewed simply as a contract but also as an organization. If so viewed, application vendors will work with the process, providing the organization sufficient information on their software. These arrangements are not difficult and can be found throughout the DOD today.

B. It is always difficult when the Prime integrator cannot be given absolute TSPR, because there is legacy software integration or contemporaneous software development. We believe that the IPT concept provides the best approach to bring all relevant contractors together (even if they are part of the integrator's program/contract team). The Government should plan to modify legacy contracts so that they are structured to support IPTs and conceivably take direction from the integrator.

C. All proposed teams should have teaming agreements in place to govern execution of the CESAR contract. These contracts should specifically state the types of work performed by each contractor on the team (to minimize the loss of subcontractors' specialized skills and insights over the life of the contract). All STRATCOM contracts that will impact or be impacted by the CESAR acquisition should (1) include a cooperation clause with the CESAR contractor and (2) required contractor participation in related STRATCOM IPTs to facilitate communication among contractors. Finally, all contractors should be require to use a common or compatible tool set to simplify and enhance communications and analyses at the IPT level.

D. Relationship with application contractors. This is not an uncommon situation and is usually handled by detailed, formal associate contractor agreements, signed by all parties, bidders as well as application contractors, prior to contract award. You must protect your CESAR contractor from being charged with performances over which he has no control. That is, it must be made clear, early on, what the exact responsibilities of the application contractors are, and where the line is drawn between their responsibilities and those of the CESAR contractor. If your goal is a total system responsibility contract for CESAR, then the selection, performance and management of the application contractors must be given top visibility--somehow, you must give the CESAR contractor a hand in each of these, otherwise, you do not have a total system responsibility contract situation. Partnerships and teamwork should be the theme here.

E. The application contractors are obligated to coordinate with other Government contractors, as directed by the Contracting Officer or his/her designated representative. No contract is required between the successful offeror for CESAR and the current/future application contractors. It would be prudent to note these required relationships in the solicitation and to include an Organization Conflict of Interest clause into new application contractor or modification to add the clause into existing application contracts. Noting such relationships might also encourage teaming arrangements with current application contractors, paving the way for a smoother transition to CESAR implementation.

F. These would be no different than any other prime/sub relationship or associate contractors on a major DoD procurement. Allow the industry players to structure the agreements and relationships as they deem appropriate with Government participation if required.

G. This vendor believes the government should evaluate using existing USAF contracts as a mechanism to GFE the latest COTS technologies to the winning prime. With reference specifically to application contractors, we strongly encourage the government to make ICASE the development model for the program, as it was intended to unify and standardized software development practice in DoD. This will help contain cost to the government. And contracts, such as the Air Force Workstation Program, are direct relationships between at least one vendor and the USAF.

H. Access to software development organizations is imperative for success. We recommend the applications developing organizations be subcontractors to the prime. This give the prime overall responsibility which is in line with the direction that you appear to be heading.

I. The most effective arrangement will be for CESAR to include applications development in the prime contract. The prime contractor should assemble a team that can provide application programming support for all CESAR applications. Directed subcontracts to key developers should be used if the selected prime does not have all necessary application developers on the team. Associate contractor relationships can work in limited instances, but Government typically ends up assuming a major coordination role. When issues arise which can materially affect one or more associate contractors, the exact level and nature of the support provided between contractors is usually a result of Government intervention and direction and often results in constructive changes to one or more contracts. In summary, all contractors who have a significant role in CESAR should have a formal contractor/sub-contractor arrangement. It is our intention to put together a large team that will include many of the existing application contractors. Our approach to teaming for CESAR is to team in depth with as many of the existing/active on-site USSTRATCOM contractors as possible. In this process of teaming for strength and organizing for productivity, we plan to have the best possible team combination to offer in support of CESAR. This will help to mitigate the above concern and to reduce the inherent program risks.

J. Associate contractor agreements should not be absolutely necessary for this procurement but they can facilitate and clarify working relationships. The work previously accomplished by the application contractors was paid for with public funds and is not proprietary to the application contractor. However, to foster cooperation between developers and the CESAR contractor, we believe that all existing and future USSTRATCOM contracts should include a requirement for an "Associate Contract Clause" (ACS) which would require those Primes to negotiate an ACS agreement with the CESAR Prime. The risk in not doing this is that critical coordination would not be accomplished and data not provided at crucial times placing deliveries at risk. We have previously demonstrated that details of an ACS can be worked out between competitors.

K. The most cost effective, objective, and positive relationship between the CESAR Prime Contractor and the application contractors is that of associate contractors with the CESAR contractor being designed as the validation and verification or test and evaluation contractor. For this relationship to work effectively, it is essential that the Prime maintain a strict integrator role and commit to not compete with the associate contractors for procurements involving development of new or enhanced applications for integration into the CESAR environment. It is critical that the Government facilitate and proactively promote the requisite associate contractor relationships by appropriately amending in-place contracts to enable the CESAR Prime to effectively interact and operate with the "associates". This positions the Prime as a non-threat to both the legacy and future applications contractors and facilitates the establishment of joint working groups and technical exchange forums to focus on the USSTRATCOM architecture modernization and migration. In addition to the use of associate contractor relationships, if the Government desires to minimize the number of primes it has to deal with, the CESAR vehicle could be used to competitively procure new applications. The management cost of such actions by the prime could be established on a fixed rate basis.

L. There are potential problems: (1) Responsibilities and authority must be clearly defined to minimize finger pointing between development/ integration contractors and application contractors when a problem occurs; (2) Non-disclosure/proprietary agreements are needed between all parties. A demanding and specific official conflict of interest (OCI) section is also required.

M. We believe that all work to be performed on the existing contracts by the application contractors should be oriented towards the successful implementation of CESAR objectives. To achieve this goal we recommend the government create an IPT chaired by the government that promotes programmatic and technical dialogue and direction between the applications contractors and the CESAR prime contractor. As the existing contracts expire we recommend that any remaining work required on the legacy systems be performed via the CESAR contract.

N. No reply provided.

O. The lowest risk scenario is when all of the development items to be integrated into the computing environment are furnished by members of the prime contractor's team. In that scenario, the prime has full management authority over his subcontractors and final delivery responsibility to the Government. In an effort as broad in scope as CESAR, that scenario is not likely, and the prime will have to integrate items produced under separate contracts that are beyond his control/management authority. In this scenario, associate contractor relationships are required, and the prime's experience and expertise in identifying dependencies and managing interfaces becomes paramount. Since associate contractor relationships typically provide only for the exchange of information, and do not include any lines of control, the Government must assume a more active role in the management of schedules and interfaces to ensure that activities not under the direct control of the CESAR prime contractor are properly coordinated. A prime contractor with proven past performance in integrating multiple systems using associate contractor relationships and a sound methodology for integration management in an environment similar in scope to that of USSTRATCOM, is essential to minimize risk in this scenario. Therefore, evaluation factors should stress past performance in these area.

P. This is a critical issue for several reasons. Existing application programs were developed with different standards, non COTS products, proprietary elements, varying documentation completeness and different acquisition agencies and customers. Applications contractors will worry that the CESAR contract will evolve to replace them as the system evolves and application programs are reengineered to become DII compliant. There are programs underway that deal with these same issues. The TBMCS, GCSS-AF, and IMDS programs are out of ESC and the ICBM Prime Integration Contract is being planned out of OO-ALC. Experience and capability in this regime should be part of the past performance volume.

Q. It is common for contractors to form alliances (teaming agreements) when they deem it advantageous or necessary to enhance their competitive position. This can be done by increasing their overall capabilities through a combination of their individual strengths formed into a single contractor team. Such alliances are workable in contracts that are narrow in scope. However, it is neither common nor reasonable, in such a broad-scoped contract as envisioned in CESAR, wherein several large, major, yet disparate mission-essential programs, functions, and technologies are lumped into a single contract. On one hand, teaming is an absolute necessity since no single contractor possesses the desired breadth or expertise to excel in all functional areas, technologies, etc. On the other hand, because of the wide scope and the specific programs to be included, the concept of teaming to compete for CESAR creates an unnatural and probably unreasonable scenario for USSTRATCOM's current contractor support structure. To be more specific, the reason is that it is the nature of Government contracting and the significant differences between the real and perceived status of being a prime contractor versus a subcontractor. Contractors who have achieved the level of prime contractor and excelled as such, have developed valuable good will and professional relationships with the Government. They have experienced the advantages of interfacing directly with their Government counterparts and enjoyed both the professional and financial benefits. These prime contractors are not readily interested in relegating themselves to the disadvantageous role of subcontractor. The Government can certainly force the issue, but it will produce an environment that is unnatural and undesirable and will eliminate from the competition some highly qualified contractors who are unwilling to take the undue risks to their reputation and future standing as prime contractors, and put up with the grief of being a subcontractor. Incumbent prime contractors have invested much time, money, and effort toward the establishment of its credentials, capabilities, and reliability. They have gained a clear understanding and appreciation of the organizational mission, roles, priorities, and short and long-term objectives. Prime contractors have secured the Government's trust and created both a good rapport and a close working relationship with their Government program managers and users. These prime contractors have achieved preeminence and superiority in these programs through many years of successful contract performance. Thus, successful teaming relationships among existing USSTRATCOM prime contractors in a scenario with such a broad scope as CESAR will not be easily achievable, and--over the long term--will probably not be satisfactorily sustainable.

4. You will be provided some AS-IS architectures. We are also developing high level short term (1997 - 1999) TO-BE architectures.

A. See Below

B. See Below

C. See Below

D. It is early to be finalizing architectures, but Fiber Optics should certainly be a consideration. Yes, solutions will be complex, but doable by experienced contractors in the many disciplines required. Management of a contract of this scope should not be a major problem. A key architectural question that must be addressed early on is the integration of disparate legacy systems. Recommendation is that you make as part of the RFP a requirement for the bidders to describe in their proposal, in 15 or so pages or less, a recommended integrated architecture to satisfy the technical requirements document information and the additional information in the RFP. Modeling and simulation technology could be of value to you here.

E. See Below

F. Good, well-written requirements are the best way to start a program. The "TO-BE" architectures should not be implementation based, but rather a vision/definition of what the government wants the end product/system to perform.

G. See Below

H. The least risk approach is to assume the current operation and to develop a plan for the transition to the target system. Consequently, we need to understand your 1997-1999, "To-Be" architecture.

I. See Below

J. See Below

K. Given the significant investment in legacy applications at USSTRATCOM, it is essential for the CESAR Prime to receive in-depth insight into: the "AS-IS" functional baseline; the near-term "TO-BE" functional baseline based upon existing contractual obligations; and the anticipated funding profile for the goal or far-term "TO-BE" architecture. This information is critical for the CESAR Prime to develop both feasible and affordable migration strategies. The prudent approach for any far-term or goal "TO-BE" architectures would be to concentrate on logical/functional considerations, rather than the physical architectures. Some key aspects of the logical architecture definition are: the underlying business or mission process; driving, non-negotiable requirements; trade-off issues and criteria that could impact future architecture decision; the information architecture and the flow of that information; internal and external system interfaces; products produced and associated timelines; etc. Critical to the success of CESAR is the up-front involvement of the Prime Contractor in the key decisions and establishment of the target architecture and optimal migration path to that architecture. The more the Prime is a part of and responsible for the "TO-BE decisions and crafting of the physical solution, the stronger the commitment to program success.

L. See Below

M. See Below

N. No Input

O. See Below

P. See Below

Q. See Below

a. What would you recommend we include in these "TO-BE" physical architectures?

A. The "TO-BE" physical architectures should only include the functionality required to integrate it into a hardware, software and/or operating environment congruent with USSTRATCOM's Master Architecture. Thus, the contractor will have the Legacy Technical Blueprint and the required parameters (DOD/Industry standards} necessary to make the transition in a timely manner. It is tempting to include requirements modifications during the reengineering process. Since legacy systems will not have been thoroughly replaced with migrating systems/operations, the "TO-BE" architectures should concentrate on very high levels of transitions, considering requirements to validate legacy-to-migration functionality. If requirements are included to the "TO-BE" architectures, and the legacy (stovepipe) system does not currently perform that operation, there could be problems with implementation and potentially disastrous consequences which may affect missions. Need: (1) Mission (2) Resources (3) Locations (4) Command Relationships (5) Legacy to Migration Systems Architecture and Milestone Schedule

B. Some key issues that must be considered include: (1) Fault tolerance/survivability (2) Security (3) Concurrency with legacy system (4) Migration vs replacement tradeoffs (5) Support for heavy-duty decision aids and models as well as standard MIS functions (6) Compatibility with national and theater C4ISR systems and standards.

C. "To Be" physical architecture should include mission-related functional and communications requirements, functional and technical reference policy or guidance documents, a description and schematic of the proposed architecture, size projections, descriptions of the required data to be processed, and milestones for the accomplishment of the future architecture.

D. See 4 above.

E. Ensure seamless operation with other DoD and government initiatives, including GCCS and DMS. Rely on COTS (and to a lesser extent, GOTS) instead of STRATCOM-unique hardware and software. This avoids costly development costs and provides flexibility to upgrade both hardware and software based on rapidly improving commercial vendor upgrades. Use high-end commercial hardware/software that relies on distributed rather an mainframe architecture. Use servers and desktops, with maximum use of parallel processing for complex calculations. Pentium or Pentium-Pro systems or, where necessary, SUN workstations are two possibilities. Look for ease of upgrade.

F. See 4 above.

G. No Input.

H. It should include everything you plan to do prior to contract award.

I. TO-Be architectures should represent conceptual objectives derived from Government studies which should be provided for conceptual information only. The Government should expect the CESAR contractor team to develop specific technical solutions for Government review and approval. Formal Government input to the CESAR prime contractor should be in the form of requirements, goals, objectives and guidance. Otherwise, the Government assumes responsibility for a particular design or architecture. The TO-Be architecture should include key applications, LANs, external interfaces and significant systems/hardware that will likely be existing in 1999. Assuming the AS-IS architectures are sound the TO-BE guidelines are clear, an experienced integration engineering organization should have minimal difficulty in evolving a well structured plan to systematically migrate the numerous systems into a well coordinated integration architecture. Based on the in-depth/hands-on experience with some of the USSTRATCOM programs, our team will come to this program with a full comprehension of what exists today in a majority of the programs within each of the four primary domains. Our team will also have an existing clear vision of the direction USSTRATCOM wants to take in each of these domains as CESAR evolves to full maturity. We would suggest the Government provide a listing of the desired standards to be complied with, a basic description of the desired performance and a list of desired outside/peripheral systems that must be supported/interfaced with. In our approach to systems engineering and systems integration, the physical/hardware selections and decisions are optimized through the use of open systems engineering processes with the physical elements and environments being determined late or last in the engineering process. In this way, the customer can get the most purchasing power for the funds invested and the most current technology to support the program needs.

J. If the Government intends for the contractor to have responsibility for the system, the Government should not be defining "TO-BE" physical architectures. The CESAR prime contractor should take the best available "AS-IS" architecture and make his recommendation for the "TO-BE." If the Government defines the "TO BE", the prime contractor is forced into a solution path which may or may not be in the best long term interest of USSTRATCOM. If the Government defines the "TO-Be" situation, what precautions will be taken to prevent undo influence by current contractors, vendors, and government personnel (military and civil service) between now and the award of the CESAR contract? We believe the CESAR prime should be the developer of the "TO-BE" architecture, however, if the government wishes to proceed with developing a "TO-BE" architecture, we recommend that, as a minimum, the following be included in that "TO-BE" architecture: (1) Conceptual View (Operations Concept) (2) Development Views over Time (Showing Topology - Object and Layering) (3) Process View (Showing System Dynamics) (4) Physical Views (Showing Hardware and Software, Networks, Etc.)

K. See 4 above.

L. "To-Be" physical architectures should include the shortfalls between present/programmed capabilities and those necessary to accomplish mission objectives.

M. We feel use of Web-based approaches will play a big role in the development of systems within the next few years. These architectures simplify development, licensing and interoperability. Currently we are exploring Netscape, WWW tools, JAVA and object oriented environments. The concentration of tools to manage these facilities are new and demand resources and security features that are functionally and technically important. The on-the-fly execution of JAVA and the "lifetime" of the static data retrieved from data servers if of great interest. The use of collaboration tools is emerging in our Intranets, but policy management issues must be understood. Proliferation of data, easy to retrieve from Internet-Intelink systems will again beg the issue of intelligent or expert EIS technology to correctly and quickly identify salient data.

N. No Input

O. We recommend that "To-Be" physical architectures be developed for each of the four mission areas: Office Automation, Command & Control, Strategic War Planning and STRATCOM Specific Intelligence Support (ref chart on p. 18 of the CESAR handout presented at the 19 June 1996 industry briefing). Each of these architectures should identify the components of the COE required for interoperability across the mission areas, the basic shared data components of the Enterprise Data Base, and the Common Services required to support each of the mission areas along with identification of each of the specific legacy systems and their migration paths in support of each of the mission areas.

P. Achieve a common architectural structure by using the TBMCS architecture incorporating a common data structure and APIs to allow legacy programs to run in a "plug and play" manner. In the provided architectures provide specific products used including release versions, hardware specifications and interface documentation.

Q. Obviously, fully detailed AS-IS system architectures would be beneficial. All TO-BE system architectures of established migration systems, e.g. NPES, SWPS, CCSS would be vital. Required building codes (standards, compliance SAIs, regulations) would be helpful.

b. How complicated will it be for the contractor to develop solutions?

A. We expect to be solution oriented and we'll have corporate knowledge of emerging technologies. It should not be difficult to develop solutions to well stated requirements. The impending hurdles will involve certification and accreditation requirements of the government. The solutions can be easy but whether USSTRACOM, DIA, DISA, AIA and the other DOD controlling authorities will accept how the function is accomplished or the hardware it is being performed on is another question.

B. For the TO-BE (as presented), the solution set is moderately complex but manageable.

C. The CESAR program is very complex. However, our personnel have the broad technical knowledge, skills, and (most importantly) experience in similar programs required for this program. The Government will simplify the process by (1) requiring cooperation among contractors as described in the answer to question 3 and (2) actively participating in all CESAR-related IPTs.

D. See 4 above.

E. It depends on the requirements. There are many commercial alternatives on the street today, but whether they are capable of meeting the requirements of the range of mission areas, from office automation to the complexities of SWPS, must rely on thorough requirements analysis (which should be preceded by process analysis). Software development and maintenance will probably be much more complex than the hardware aspect.

F. See 4 above.

G. No Input.

H. When requirements are known, the ease of development and implementation of a solution is significantly enhanced.

I. The prime contractor selected for CESAR should be experienced and proficient in system integration and development of complex operational architectures. Such a prime should have no major difficulty translating USSTRATCOM requirements, goals, interfaces and existing equipment into technical solutions for review and approval. Members of our planned team, in many cases, are already involved in this type process. Our approach to teaming and the in-depth experience the team will bring to bear on CESAR will assist in reducing the evolution and complexity of programmatic solutions. A team led by or made up primarily of non-USSTRATCOM-experienced contractor support will have a hard time sorting through the numerous stove pipe systems and legacy code to evolve effective solutions. In this case program risk will be high.

J. This is a hard question to answer straight away because of the large number of dependencies and/or constraints which may be placed on the CESAR-prime such as: (1) Quality and depth of the "TO-BE" architecture to be provided by the government. (2) Availability of the End-User personnel for consultation. (This does not normally tend to be very good.) (3) Government management's support of Government-contractor teamwork. (4) Ability to establish an iterative, evaluation of design process. However, if the Government can effectively deal with the above dependencies/constraints it should not be difficult to provide solutions provided the CESAR prime contractor has the freedom to accomplish the job at hand.

K. See 4 above.

L. The degree of complexity in developing solutions cannot be determined in a vacuum, rather it must be overlaid against requirements.

M. Contractors will find it difficult to build integrated solutions without large test beds or access to common lab facilities that correctly mirror the ever more complex combination of CINC, Base-level, and DOD components. Correct interaction of the objects is the next major hurdle after interoperability is achieved. Knowing how the environment will be affected by the proliferation of browsers, servers, and tools is necessary to prevent catastrophic or minor failures.

N. No Input

O. As has already been recognized, the CESAR effort addresses a broad spectrum of requirements and encompasses a complex set of functional activities. Development of innovative solutions in a complex environment is always a challenging task, however, a carefully selected contractor team with a proven track record in standards based open systems development and integration, working as a member of an Integrated Product Team (IPT) with the Government and with the applications developers, will be able to meet the challenge. The key to success will be the ability to effect a teamworks approach throughout the total lifecycle from requirements definition through operations, maintenance and sustainment of the systems.

P. Require more detailed information to respond meaningfully.

Q. Depends on the scope and breadth of the requirements and how synergistic the relationship is among and between integrator team members. Most of the problems are complex which implies complexity in solutions. The issue of solution development may not be as critical as obtaining solution approval from the Government. Historically, the Government has exhibited a tendency to micromanage Contractor solution development and, whether intentionally or unintentionally, steering it toward the "preferred" solution by not approving otherwise acceptable solutions.

c. Is the scope too broad? Would it be difficult to manage?

A. Each candidate hardware, software and/or operation may have different goals and therefore, different management, maintenance or reengineering strategies. Some areas may have sufficient documentation, version descriptions, and specifications, but legacy and unique one of a kind subsystems may not have been documented accurately. In those instances, the scope should be tailored to fit the environment, which suggests again to break the CESAR into manageable parts.

B. The scope is manageable, because there is a single organization who owns the system. This is compared to something like TBMCS with its much larger, more diverse user base and wider variety of scenarios and missions it must support.

C. A contract of this scope requires extensive management and oversight to ensure a coordinated effort among contractors/subcontractors and tasks. Management will be complex and labor intensive. Management of this program must focus on IPTs developing specific products.

D. See 4 above.

E. The scope is certainly broad, but probably necessary if STRATCOM is going to get out from under "the house that SAC built." They key will be to have a strong systems integrator to ensure that mistakes from the past (e.g., software driving the process rather than the other way around) are avoided and that the large number of contract team members, both new and old, are kept in the loop.

F. See 4 above.

G. No Input.

H. No.

I. The scope of CESAR is aggressive but reasonable. Carving up small pieces of the C2 and War Plans would result in the Government continually playing the role of integrator and coordinator with no contractor accountability for the system. It is important, just as it is in normal military organizations, that there be clear lines of responsibility and authority. It would be inefficient to have one contract maintaining/evolving part of a command center and another doing the same for other parts. In this case the Government ends up with all of the responsibility. For CESAR to succeed, a single responsible focal point for all USSTRATCOM C2 and War Planning systems is necessary. The responsible contractor must have a formal contractual superior/subordinate relationship with major supporting contractors. The projected scope and difficulty are only issues for those companies that have limited knowledge of the four CESAR domains and have little or no experience in managing large contractor teams and large complex programs. For companies from outside the USSTRATCOM contracting domain, a program of this nature with the high level of specializations and uniqueness, would be extremely difficult to comprehend much less successfully manage and modernize. The resulting risks would be unacceptable. In our case, this is what our company is all about. First off, we are a highly experienced USSTRATCOM contractor with years of innovative support in some of the domains. Secondly, we have an in-depth track record of managing large programs and large program teams. On a program we now have a team of five contractor companies and we are also a prime integrating contractor with a team of eleven sub-contractors. For us, the scope is not too broad and the management would not be difficult.

J. The scope is appropriate for the maturity of the user and existing software systems. Given the normal automated system's life-cycle and where SWPS is in that life-cycle, it is time that the SWPS system undergo reengineering. This will not be an easy or simple job, therefore, it will require exceptional management attention, formal reviews, etc. However, the problem is not insurmountable.

K. See 4 above.

L. The scope and management difficulty are dependent on (a) and (b) above.

M. Numerous techniques and tools are available to manage and identify to be architectures and migration strategies. In GCCS we have used a set of tools to "segment" components and we used this technique to restrict participation and implementation of how systems that must meet interoperability to goals.

N. No Input

O. While the scope of CESAR is indeed broad, we do not view it to be excessive. Management of the effort will require a prime contractor with proven past performance in managing a large team of subcontractors, experience in working through associate contractor agreements to integrate battle management systems developed under independent contracts into a cohesive system of systems, and a corporate culture that encompasses continuous process improvement and work with the customer in an integrated product team environment.

P. Require more detailed information to respond meaningfully.

Q. The scope appears to be too broad. A company of any size would have difficulty managing a contract of this size and scope. Many tasks and functions would be ongoing, and managing all deliverables in a timely and quality manner would be a complex responsibility (e.g. AFMAS).

d. What type of architectural questions would you see addressing?

A. Because architectural decisions are make early in the life cycle and to some extent, within a decentralized environment, it is difficult to ascertain which architecture will become the USSTRATCOM lead architecture for any given domain. In fact, many on-going changes were based on domain architectures without consideration of other interoperable requirements i.e., MIPS-toMIDB; NPES-to-NPES-II, CCPDS-R use of UNAS, CMAH Tech Control/Fibite Software etc. The most recurring questions will be: (1) Commonality: The most important and most misunderstood area. (2) Interoperability: With whom and to what extent? (3) Security: Standard requirements to safely protect all kinds of data whether SCI, SIOP or Collateral. (4) Migration: The assumed key to architectural achievement is summed-up in migration timelines. How soon can it be implemented: if it takes too long, then the solution is outdated before it's implemented. (5) Benchmarks: The idea of metrics and measurable benchmarks are coupled with architectures.

B. See reply under 4a. Above.

C. In addition to refining segments of the "To Be" physical architecture, we would address non-physical issues such as the common operating environment, data standardization, enterprise-wide common database, interoperability requirements and migration strategy as it impacts architecture design and milestones.

D. See 4 above.

E. (1) Seamless cut over (especially with SWPS) from old to new systems (2) Integration with migration systems (3) Interoperability with theater systems (4) Multi-level security (5) Integration of fixed, ground-mobile and airborne systems (6) Transition of planning functions from airborne to ground-mobile systems (7) Interoperability with NMCS.

F. See 4 above.

G. Regarding architectural solutions, it is imperative prospective bidders understand the nature of and have access to any legacy software, whether GOTS or COTS, to evaluate the likelihood for success in migrating to an open solution.

H. The question appears to be ambiguous.

I. The fundamental criteria for architecture development are: standards compliance, extensibility, openness, flexibility, elimination of redundancy, ease of upgrades (software distribution), reliability, security, response time required and data integrity. The fundamental direction of the USSTRATCOM architecture is fairly well established. The key challenges for the CESAR team will be to establish a sound migration and integration plan to achieve the target architecture and to evolve and maintain the architecture as standards, mission and technologies evolve without impacting on-going operations. An important principle is to achieve increasing levels of overall architecture capability with out back tracking and getting into technological dead ends. For the program to be a success. We would rely on proven architectural processes that have been employed on past programs. Our team would conduct, in a coordinated effort with Government reviews, an analysis of each of the targeted C2, War Plans, Business and Intel programs. That analysis would provide a foundation from which to evolve a recommended phased course of action to bring each program up to a common level of open system standards and then to integrate them into a single way of doing business. The primary question is how do we get from where the systems are today to where the J6 Vision wants them to be. The best architectural support possible will be the extensive domain knowledge of each of our teammates/subs and the strong integration leadership of our firm.

J. The top level questions are addressed by 4.a.J. above. These are the questions one would need answers to before proceeding. But other questions which should also be addressed are as follows: (1) Develop appropriate, documented assumptions. (2) Develop a basis for assessing performance through modeling/simulation. (3) Implement a single responsible organization for System-wide and Network Administration. (4) Start the institution of the DII COE in the strategic community. (5) Continue Database reengineering.

K. See 4 above.

L. Scope, feasibility, achievability, security, and the rapid evolution of information technology.

M. What tools and techniques are evolving to provide information security in the interoperable environment? Are the protocols and transmission standards for the CESAR in line with advancements that industry will fund and propagate? What is the life-span for applications system components in the C2 and WP environments? What is available from other DOD programs?

N. No Input

O. The architecture for the STRATCOM Computing Environment must address a wide range of technical and functional issues including items such as LAN throughput, a standard security architecture that provides multi-level security between the office automation, war planning, C2 and intelligence mission areas, standardized application of common services, insulation of functional applications from changes in the service layers through he effective use of middleware and applications program interfaces, software reuse, technology refreshment, network management and administration, effective use of COTS solutions in a secure computing environment, data normalization in an enterprise data base, adherence to industry and Government standards, requirements traceability to services and functional capabilities, allocation of functions to specific hardware and software configuration items, interface definitions and control, and configuration management. All of these issues must be addressed in the context of an affordable evolutionary (vice revolutionary) migration to a fully open systems environment that is compliant with the Defense Information Infrastructure (DII) goals and specifications.

P. Reliability, availability, security, throughput, HMI, database, standards compliance, technology insertion and scalability are all constraints to meeting operational requirements. Balancing these factors with the reengineering of the legacy programs constitute the main focus of the CESAR program.

Q. Technical: Integration of migration systems, standardization (applying style guides, TAFIM, GCCS DII, etc.), interface interoperability, security (e.g. MLS, or multiple levels), data management, communications management. Management: Integration of migration systems, interoperability with other services, commands, CTFs, etc. Management oversight of current and developed systems (e.g. systems management, systems administration, database administration, etc.). Significant effort to manage numerous subcontractors' performance responsibilities.

5. What key standards and technical issues are involved in implementing a seamless, highly-integrated system?

A. See below for key standards. Technical issues involved in implementing a seamless, highly-integrated system: (1) Standard Operating System (2) Application Program Cross-Platform Operation (3) Proprietary Applications (4) Controlling Authority Authentication and Accreditation Guidelines (5) Approved Hardware List Application (i.e., NPES, MIDB, etc.) (6) Multi-Level Security Constraints (7) Compliance Criteria (i.e., DII-COE, Multi-level security)

APPLICABLE STANDARDS/REFERENCES

Standards must result from the C4ISR Framework Architecture

MIL-STD-973 Configuration Management Procedures

EIA/IS-649 National Standard (Replacing MIL-STD-973)

MIL-STD-498 Software Specific Configuration Management (Replaced MIL-STD-2167)

DoDI 5000.2 Air Force Supplement 1, Defense Acquisition Policies and Procedures

AFPD 33-1 C4 Systems

AFPD 33-2 C4 Systems Security

AFI 33-101 C4 Systems Management Guidance and Responsibilities

AFI 33-102 C4 Capabilities Planning Process

AFI 33-103 C4 Systems Requirements Development and Processing

Component Standard STD/Group Standard Number/Series

User Interface MotifTM OSF OSF/MotifTM(20)

Programming Languages Ada FIPS FIPS 119(1)

" " C FIPS FIPS 069-1(4)

Database SQL FIPS FIPS 127 (27)

Network OSI FIPS FIPS 146 (GOSIP) (21)

" TCP/IP MIL-STD MIL-STD 1777, 8 (18,31)

Operating System POSIX FIPS FIPS 151 (IEEE-P1003) (23)

Physical 1472 MIL-STD MIL-STD 1472D (12)

Ethernet CSMA/DC IEEE-802.3 ANSI/IEEE IEEE-802.3 (15)

Baseband Ethernet 10BASE2 ANSI/IEEE IEEE-802.3a (19)

Broadband Ethernet 10BRAOAD36 ANSI/IEEE IEEE-802.3b (3)

Ethernet Attachment 1BASE5 ANSI/IEEE IEEE-802.3e (22)

Token Bus IEEE-802.4 ANSI/IEEE IEEE-802.4 (29)

Token Ring IEEE-802.5 ANSI/IEEE IEEE-802.5 (30)

Fiber Interface FDDI ANSI ANSI X.3T9.5 (8)

Serial Interface RS-232D EIA EIA RS-232-D(16)

Serial Interface RS-449 EIA EIA RS-449 (10)

Balanced Digital Data RS-422 EIA EIA RS-422-A (6)

Unbalanced Digital Data RS-423 EIA EIA RS-423-A (7)

Terminal Interface X.25 CCITT CCITT-X.25 (17)

Command Response 1553 MIL-STD MIL-STD-1553B (5)

Equipment Control GPIB ANSI/IEEE IEEE-488.1 (13)

Broadband Backbone ANSI/IEEE IEEE-9-802.7

B. Key standards include: C4I interoperability, a part of which is DII COE compatibility. Other commercial standards, including COTS product choices. Data definition and standardization. USI. System/Software development.

C. Highly integrated systems required data standardizations and a standard development process that emphasizes open system solutions. The technical issues that impact these standards include an accurate "As-Is" model, a complete "To-Be" model, and security. Success will depend upon active user involvement in IPTs.

D. Your choice of an operating system must not only be governed by security capability, but you should also be careful to choose one that will grow with technology, preferably one that is widely used in the commercial world. This way, you can benefit by technology advances that result from a wide range of users.

E. This research would be a prerequisite for the CESAR contract and would be part of the requirements analysis phase of the work plan. A key is understanding and integrating both government-unique standards and commercial standards. Government-unique standards should be used only when absolutely necessary in order to avoid cost growth and stove-pipe solutions. Whenever possible, open system, plug-and-play commercial solutions should be the choice.

F. The requirements should emphasize openness and commonality with industry standards. Minimize the use of military and DoD standards when possible.

G. This vendor believes critical success factors include reduction in life-cycle cost, improvement in system administration and maintain cost, the introduction of rigorous (SEI-3 or greater) software engineering practices, and elimination of proprietary legacy technologies.

H. We have developed a product which has been widely accepted by industry as an open systems architecture of choice.

I. The key standards and technical issues are related to the fundamental architecture criteria as discussed in the previous paragraph. Standards-based solutions are required to reduce dependence on particular vendors or technologies and to facilitate upgrades. The issue of extensibility is important because an architecture that cannot grow to efficiently meet evolving mission requirements will become unsatisfactory. Flexibility is important so that the architecture can be quickly reconfigured due to component failure, changing mission, or unexpected requirements without fundamental architecture changes. Elimination of redundancy, except where operationally required, will reduce maintenance costs and improve reliability. Ease of upgrades can be a critical issue for large client server systems. Upgrades to client COTS and application software can be a maintenance nightmare. There are strategies for partitioning between client and server and use of distribution tools that can reduce this maintenance risk. Reliability, data integrity and security are key criteria for most C4I systems for obvious reasons. To date, it would appear that the standards provided by GCCS and DII provide for that basic foundation. Based on those targeted standards, additional elements may need to be added because of the unique nature of the programs involved. As new elements are added, they should be required to be complimentary to the base standards. The primary technical issue is to determine the configuration that will make up the common operating environment. Each existing domain has selected a specific way of developing and operating and within those domains many of the programs have been built in a stove pipe fashion. In addition, many of the programs have long standing legacy aspects that will require new thinking and potentially deep cultural changes on behalf of the existing program managers and engineers. The evolutionary process will be founded on an analysis of each candidate program. The results of the analysis will support the recommendations regarding the most obvious configuration to migrate to, the phasing of the program migrations, the full set of standards to be adopted and the program modernization's that will be required. In this process, it will be an important factor to not only understand the existing system hardware and software configurations but to also have detailed information on each of the existing support contracts so that in addition to evolving a sound engineering approach, the CESAR team can marry that approach to a series of suggested sound business decisions. This is important to insure effective management of the existing systems and a cost effective evolutionary process to the new systems based on the milestones and decision points/option points and end times of the existing contract vehicles. USSTRATCOM may be better served by allowing some program contracts to run full cycle while others may initiate their modernization's and integration at earlier option points.

J. Key standards for the CESAR are covered by the DII COE. All other Government called out standards during a normal information systems procurement will need to continue to be included. As to technical issues, we believe key issues in the areas of interface integrity and data cohesion within processes exist and should be addressed.

K. Among the key standards and architectural guidelines that provide the foundation for a seamless, integrated systems are: Open Systems standards. Flexibility of the system architecture to respond to and evolve as standards advance (e.g., DCE vs. CORBA). Adoption of data format and transmission standards. Compliance with the Global Command & Control System (GCCS) and Defense Information Infrastructure (DII) Common Operating Environments (COE) or Standard Operating Environments (SOE). Adoption of standards in relevant enabling technology areas (e.g., storage, security, communications standards) that are not addressed in the COEs/SOEs. It is extremely important to note that adherence to Open System standards is a necessary but not sufficient requirement to ensure ease of system interoperability and evolution. Data formats, communications methods, and application interfaces often have a much higher impact on the ability of two applications to work together and should be emphasized in any CESAR procurement.

L. Common data, database structure, common programming languages, and computers and communications.

M. There are three standards of primary interest. They are the Technical Architecture Framework for Information Management (TAFIM), the DoD Joint Technical Architecture, and the DII COE Integration and Runtime Specification. There are other standards of importance as well, such as the User Interface Specifications for the DII and Information Security Standards as they evolve in C4ISR Test Bed.

N. The key standards involved are the communications standards to be used on the various networks. TCP/IP standards should be used as a base for all communications. These can be used with a variety of transport layer protocols to form the backbone of the sub networks and should allow for easy transition between different transport layer protocols. The primary technical issues are: (1) The integration of existing networks into the new system. The first step is to write communications APIs that will convert existing protocols to the TCP/IP standard. (2) The use of separate networks or firewalls to achieve security separation between networks of different classifications and the certification and accreditation of the firewalls. (3) The conversion formats for data, e.g. word processing formats, between systems that are now separate. This could involve the use of several standards such as rich text format (RTF) or other similar "standard" formats.

O. The key standards and issues involved in implementing a seamless, highly integrated system for STRATCOM revolve around the concept of an open systems architecture. The DOD Technical Architecture Framework for Information Management (TAFIM) embodies many of the standards that must be followed. Other standards and architectural guidance documents such as the Department of Defense Intelligence Information Systems (DODIIS) Reference Model, and the DII being implemented through programs such as the Global Command and Control System (GCCS) and the Global Combat Support Systems (GCSS) coupled with the appropriate security guidelines produced by the National Computer Security Center and the Defense Intelligence Agency provide the framework for interface definition and control, COTS integration and data normalization necessary to achieve interoperability across a common operating environment.

P. Guidelines proclaimed in the DII, GCSS, DMS and GCCS form the basis for new programs to be interoperable and integrated. Data element standardization is critical in integrated systems.

Q. The contractor must be able to understand, comply with, and implement current standards. These standards are now fairly well described. The Defense Infrastructure (DII) Common Operating Environment (COE) Integration and Runtime Specification (I&RTS) should be adhered to for all developing systems. The Technical Architecture For Information Management (TAFIM) should be adhered to. Migration system guidance in conjunction with the Federal Information Publications Standards (FIPS) should be adhered to. MIL-STD-498's follow-on commercial standard should be adhered to for software development.

6. What are the critical success factors for this program, and why? What can the Government do to ensure success?

A. All end products must satisfy the end users needs and must have traceability to the user's needs. Efficient management of multiple contract participants will require a process for rapid and definitive arbitration of opposing opinions. Costs and costing must be clearly defined and in as much as is possible directly related to quality. Effective and efficient communications between all government and contractor organizations is essential (fax, telephone, internet, etc.). Success Factors: Single COTR (Central Government Authority). The decentralized management and control of systems as currently arranged at USSTRATCOM not only support single system enhancements and stove piping but creates conflicting architectures. Easily understood and manageable scope of work. The scope should be concise and easily interpreted. The best way to achieve this is to keep it to one or two subject areas, i.e., software, hardware, etc. The broader the scope the lower the probability of success. Identifiable and measurable criteria. If the scope is tightly written, the criteria becomes much easier to identify. It's best to address software criteria for NPES, Command Center, etc., than to address software requirements for an innumerable general list of systems that are considerably different in makeup, design, and functionality.

B. Critical success factors include: Corporate (USSTRATCOM) commitment to the program. Other attempts at automation or automation improvement (ACCESS, for example) did not have the top to bottom organizational backing, and hence were operational failures. Another aspect of corporate commitment is the recognition that a seamless system my require some minor power bases be dissolved. Oftentimes, seams represent one organization's information control point. A true Customer (USSTRATCOM)-Contractor team. The large majority of contractors supporting STRATCOM are of superior technical competency and have as their goal to provide the Government with a high value solution. Contract scope and terms and program organization and structure must have the Government and Contractor sitting on the same side of the table. Adequate funding and schedule for the desired system. The converse to this is that expectations must be in line with program constraints.

C. The success of a broad-scoped program such as this is most dependent on management processes that focus on both the business issues driving the program as well as the development of technologically sound and highly flexible solutions. Mature management process statistics show that large contracts fail because of schedule and cost overruns. The Government can evaluate the maturity and quality of these processes using an evaluation method such as SCE or SDCE in addition to past performance analysis. In addition, the Government must implement the following success factors: True implementation of the IPT paradigm-seamless integration of a variety of computer systems requires an overarching system IPT. Mature development processes and methods supported by state-of-the-practice tools -- seamlessness and interoperability is required -- the contract is large enough to require multiple contractors or multiple teams on a contract -- there is potential for geographically dispersed engineering teams -- common presentation is required for Government IPT members to evaluate and assess the evolving system. Active Government participation in outside programs that influence the CESAR program-STRATCOM can lobby to ensure that outside programs enhance the evolving CESAR architecture.

D. First, establish an atmosphere wherein the government manages the prime, and the prime manages the technical effort and the subs. Next is the detailed explanation of just what the contractor is to do. Your draft technical requirements document will be critical. You must have in it a clear statement of the requirements--not how the contractor is to do the work, but what he is to do. This is the essence of the new Statement of Objectives that will be used by the government. Then, the turnover to the contractor must be transparent to the day-to-day operation of STRATCOM. Plan for a period of performance where the contractor is working along side the current operators in selected critical areas, and phase into the contractor-only status gradually. Those areas should be spelled out in the RFP, so that the contractor can bid to that requirement.

E. The critical success factors are successful system operation, in a timely manner, and without undue disruption to other STRATCOM contracts and operations. The Government should clearly define the functional requirements for CESAR in performance terms and solicit COTS-based alternatives for demonstration/validation and subsequent enhancement to full CESAR required capabilities. Another key factor is for the contractor to understand the needs of the various customers to be serviced. This will necessitate close coordination and integration with government users, especially during the initial requirements analysis.

F. Selecting a prime contractor that understands the current mission at STRATCOM. Establish realistic schedules and goals.

G. No Input

H. Simply put it is partnership with the contractor. The question is far too reaching to be specific, but the contractor can not be successful without your complete cooperation and actions as a partner. Five critical factors for the customer: (1) You must have accepted that there is a need for a basic change in the operation of the business (2) There needs to be an Executive Sponsor of the change - a person that will lead the organization into the change and overcome all of the internal opposition to change (3) The customer has to understand their current operational performance, configuration, existing contracts and the terms and conditions of such relating to change (4) The customer has to have a vision of where they want to be in the next 5-10 years (5) The customer has to be ready to enter into a partnership with a contractor as a single entity and work for win/win solutions. Eight factors for the contractor: (1) Experience in managing large I/T operations (2) A deep technical bench (3) Financial security (4) An excellent reputation with customers (5) Global service support (6) Knowledge of your operation (7) Flexible contracting arrangements (8) A partnership with you to satisfy your mission requirements. The removal of the Government from the delivery process all together through the use of agreements with the prime contractor is a key contracting requirement. That is, the Government needs to baseline current and document future demands. The Government should then allow the prime to select the architecture that delivers the defined service level. Further, the Government should give the prime the discretion to change the components as necessary to maintain a cost effective system. This is a different operational scenario than what has been portrayed to date. It will place the responsibility for the I/T service on the prime, leaving the Government with the critical responsibility of providing strategic direction for the development and use of the application systems.

I. The primary success factor is support of the USSTRATCOM user. This can be measured in terms of responsiveness to requirements, reliability, security and ease of use. Other supporting success factors include cost savings compared to the current cost of existing systems, maintaining an established schedule for providing capabilities/savings and being able to rapidly respond to new mission requirements. The Government can ensure success by carefully selecting the CESAR prime contractor based upon proven capabilities and ability to readily accept responsibility within the USSTRATCOM environment. The most difficult decision the Government will make will be to determine which offer provides the best value and least risk to the program. A majority of the work that is managed at USSTRATCOM today is unique to this command. The existing programs have been undergoing continuous modernization since the early days of SAC. A relatively small contractor group has specialized over the years to provide this unique functional support. Understanding and functional competence of these unique applications cannot be readily acquired outside of the USSTRATCOM domain. As a result, the critical success factors are: (1) A contractor team with a history of successfully supporting USSTRATCOM objectives. (2) A large, experienced contractor team that has both team depth and combined corporate breadth to support all of the specialties of the CESAR vision. (3) Clearly established lines of authority and responsibility for the prime contractor. (4) Integration of contractor and Government organizations to form a CESAR team that will work side by side in an air of total cooperation and dedication. (5) A strong programmatic communications exchange between all players to insure thorough coordination and a continuous focus on the objectives. The expansion of the IPT process to carry out this coordination and resource optimization. (6) A proven prime development and integration contractor with the experience necessary to manage the team and the knowledge of the targeted programs necessary to make effective modernization recommendations and to carry them out. (7) Complete user/customer involvement. (8) Determination of a clear set of program standards, goals and objectives.

J. The prime contractors ability to serve as the integrator is the most critical factor for success. In the Industry Day briefing on 6/19/96, Government representatives expressed their desire that USSTRATCOM become a "user." If that is indeed the goal, let the prime contractor be the integrator, and leave the details of system design and implementation to the prime with the government participating in formal reviews, test and acceptance, etc., to provide the prime the needed feedback. Critical Success Factors which are self-evident: (1) Selection of a "Prime" contractor who is good at team building. (2) Prime with established processes and procedures for software development (independently rated and approved at the SEI Maturity Level III as a minimum), subcontract management, and risk management. (3) Prime and subcontractor team members with good past performances. (4) Prime and subcontractor team with locally available committed resources who are technically competent. (5) Prime with experience at coordinating activities between non-collocated developmental sites. (6) Prime and subcontractor team who understands the legacy systems and what the Government wants as an end product or system. (7) Prime and subcontractor team who understands the processes being implemented at both the physical and conceptual architectural levels. (8) Finally, a Prime and subcontractor team who truly understands the USSTRATCOM organizational mission. Government actions to establish a cooperative atmosphere, to promote frequent, open dialog with the contractor integration team, and to participate in team building with the contractor, will go a long way toward ensuring a successful program.

K. The following describe some critical success factors for the CESAR Program: (1) Establishment of a highly Integrated Customer/Prime Contractor/Associate Contractor team is key to the success of the "TO-BE" architecture and migration--joint decision and joint buy-in leading to joint success. (2) A migration characterized by zero impact to operations and production requirements, resulting in a satisfied customer and end user community. (3) Human factors, training and simplicity, adaptability to changing requirements, maintainability of availability are all important elements to be included in any award or performance penalty criteria. (4) Optimal use of system and human resources to meet CESAR operational needs, representing optimal utilization of program budget. (5) An open, flexible, and scaleable CESAR architecture that supports the latest hardware, software, and communications technology both today and into the future, precluding delivery of a system that is already obsolete at initial delivery or becomes obsolete during the term of the contract. One of the primary methods for the Government to ensure achievement of the above success factors is through an effective contract incentive structure. The success of the integrated team can be further facilitated by proprietary disclosure agreements among the contractors; establishment of a non-compete, OCI clause for the Prime; and Government-mandated IPTs.

L. The critical success factor for this, or any, program is customer satisfaction. To ensure success, the Government should: (1) Clearly state requirements. (2) Award the contract to a company with integrity that has performed well in the past. (3) Motivate the contractor with awards/incentive fees. (4) Provide adequate funding. (5) Measure contract success periodically (PMRs). In addition, the government must have a strategy to incorporate changing requirements, new hardware, and new software methodologies into the procurement process. Otherwise you end up with a system built to outdated requirements that neither meet current requirements nor use the best technology.

M. Among the critical success factors for CESAR, based on the information available to us at this time, we find: (1) Proven capability and experience in providing a seamless interoperable and integrated environment. (2) Knowledge and experience in GCCS and the DII COE. (3) Ability to test prototypes in a secure environment. (4) Experience and capability in Multi-Level Secure systems. (5) Ability to provide and manage a large team which includes SB & SDB companies. (6) Ability to provide risk-free transition into and out of the contract. (7) Ability to rapidly react to changing Technological and DOD shifts in program funding. (8) Ability to understand the implications of high-end simulations and graphics in a secure environment.

N. No input

O. Critical factors for success in an effort such as CESAR include management of transition risk; requirements definition, stability and tracking; organization and management of IPTs; and the sustainment of functional capability through migration from legacy systems. Careful selection of a contractor team, based on proven past performance and low risk software engineering processes, coupled with open post award dialogue and cooperation between Government and the contractor in an IPT relationship are the prime ingredients for success.

P. In transitioning any operational program into a new environment it is necessary to involve the user in defining and refining the requirements. An IPT with government and industry needs to agree on a schedule with specific requirements for each release and a cooperative effort needs to be inplace to solve problems rather than document problems. The government must be involved in relationships with the application contractors and the CESAR prime contractor to resolve relational conflicts.

Q. Critical success factors are: (1) Program Office. A well established, experienced, well-managed program office is essential to the success of this program. (2) Funding. You get what you pay for and deserve what you get! (3) Contractor Experience includes the following: (a) Systems Engineering and Integration Experience. The integration task is large. Without excellent systems engineering and integration experience to bear, the program will fail or will be implemented poorly. (b) Area Domain Experience, e.g., Strategic War Planning, Command and Control, Local Area Networks, Intelligence. The complexity of these domains requires experience to bear on the problem. Otherwise, the ramp-up of the vendor could take months to years and many wrong paths will be pursued. (c) Technical Systems Experience. Experience in the technical environment is critical to ensure a smooth transition. (d) Contract Management Experience. The ability to manage a large contract is a critical success factor. The ability to successfully work with other contractors and to manage subcontracts is critical. The Government should do the following to ensure success: (1) Write an achievable Request for Proposal (RFP with clearly defined objectives. (2) Establish and maintain an experienced Systems Program Office. (3) Ensure the program managers engage the participation of both prime and subcontract management in program direction and review functions and processes. (4) Fund appropriately. (5) Provide financial incentives for performance, otherwise you'll get what you pay for. Be sure to apply cost realism tests during BAFO. (6) Provide direct communication channels between subcontractors and the Government Program and Contracting Offices.

7. We envision the contractors of this effort to be an integral part of our Integrated Product Teams (IPTs). What do you foresee as your role and responsibility on these teams?

A. The CESAR Integrated Product Team (IPT) must have quality people with authority, possibly report to a qualified Flag Officer/Corporate Officer "Graybeard" Steering Group. The awarding contractors will not only have to be an integral part of the government PAT but also part of the woven fabric of the organizational structure. The contractor representation would present architectural ideas and solutions using advancing technology. Their participation would also solidify a change process that is communicated across each community i.e., command center, ground, airborne, mobile etc. Ensure that the proposed architecture and resulting systems/capabilities support the warfighter, comply with the C4ISR Framework Architecture, move STRATCOM toward the goal of a DII/COE, and permit Technology Refreshment. Help the IPT's develop SOOs so that tailored products are developed to meet a specific requirement.

B. The contractor's IPT role and responsibility should match its role on the program. We should be prepared to chair appropriate IPTs. As the integrator, the success of each product is our responsibility, thus the direction of the IPT should be ours also. The current IPTs are oriented along SIOP product lines. The CESAR contractor(s) should have roles on the IPTs similar to the ones now held by your application contractors. The goals of the CESAR contract will mean that there will more interdependencies between the IPTs as currently structured: Seamlessness means that the system cannot be separated to the same extent that the products are. The Contractor will need IPTs to structure system development. Initially these will constitute a separate set. Over time we may recommend a reorganization of all IPTs so that the distinction between a SIOP product/function and its tools disappears.

C. The CESAR contractor should be an active and equal participant in all IPTs, providing technical information and recommendations as required. Also, the contractor should chair the system IPT.

D. The prime, and certain subs, should certainly be full members of selected IPTs, and they should be working members, not just advisors or assistants. They should be integrated into the team structure and fill roles applicable to their specialty.

E. The CESAR contractor (team) offerors must clearly understand the role and charter of the IPT. The successful contractor (team) must be easily inserted into the existing IPT and immediately function as a responsive team member to ensure STRATCOM acquisition goals. Demonstrated performance of the offeror on an IPT should be included as an evaluation factor in the RFP. The role of the contractor is to thoroughly understand both the requirements and processes used by the customer and then to research and recommend alternative solutions (options) for consideration by USSTRATCOM. Once the acquisition decision is made by USSTRATCOM, the contractor must procure the system hardware and software, debug before cut-over and work with the customer to schedule installation and periodic system maintenance/testing in order to minimize disruption.

F. It is essential that contractors supporting this effort be actively involved on established IPTs. This will enhance system effectiveness and will facilitate communications and team-work between the Government and contractor team. The IPTs are implemented by assigning individuals from the functional support organizations with the desired experience and expertise. These teams are given the authority and responsibility to conduct any tasks. From this point on, these empowered team operate autonomously within the confines of the contract to complete their tasks.

G. No input.

H. As a member, it is our responsibility to define the current operational environment. We will also identify required changes to the environment to accept new or modified applications.

I. The key role of the IPTs will be to facilitate development of requirements, Government review and approval of architectures and designs to assure user satisfaction. IPTs will constitute the primary means of Government control and oversight of the process on a day-to-day basis. The contractor will participate in discussions, present technological solutions and other data as requested and provide administrative support such as agendas and minutes. Our approach to the successful integration and modernization of large programs such as this is highly dependent on the use of and participation in IPTs in every aspect of the evolution. IPTs form the backbone of a low-risk engineering process and our team would expect to be integral members of the Government IPT process. In addition, due to the wide ranging scale and scope of this program, we would also implement an internal level of IPTs in order to maximize the team's wide ranging engineering expertise. This internal effort would, in turn, enhance the effectiveness of the Governments formal IPT process.

J. What is the real role of the Government's Integrated Product Teams? If they are to continue to act in the same manner as today, the Government will continue to assume the integration role! The prime contractor should have the role as lead for most IPTs. The prime contractor should receive requirements and inputs from the user communities via the IPTs and coordinate the appropriate responses. If the development-related IPTs operate without the prime contractor as the lead, the Government really continues as the integrator. Having said that, we firmly believe that the government would want and should chair the CONOPS IPT with the CESAR Prime and all Applications developers (other Primes) as participants. As a contractor we already participate on several IPTs and recommend that the contractors be included on all IPTs to foster that cultural change desired by the Government leadership. Today, we see the role is advisory in nature. This needs to be rethought if the Government is ever going to relinquish the integration role. However, whatever the government decides as an appropriate role of the CESAR contractor with regards the IPTs, we are ready to support you.

K. We believe that IPTs are most effective when customers, contractors and suppliers are equally empowered members of the team. Our past experience with IPTs suggests that problems occur when there is a lack of definition in the decision making authority among the participants. As the CESAR prime contractor, we would actively seek membership on USSTRATCOM IPTs and invite active customer and applications contractor participation in our own program specific IPTs which we would tailor as much as possible to mirror your IPT structure.

L. The contractor should be the technical lead and be responsible for IPT Scheduling, Preparation, Hosting, Reporting, Follow-up Actions (Approved by the Government).

M. We agree with the level of importance the government has placed on IPTs. We recommend that IPTs be staffed with both government and contractor personnel with roles and responsibilities (technical as well as leadership) assigned based on the tasks to be performed, the resource requirements to successfully perform the tasks and the skill mix within the IPT membership. The IPT focus should be on achieving the goals and objectives of the IPT and not on organizational boundaries. On other contracts we have accomplished the movement of applications and COTS through the CM, integration, test and deployment cycle in an average of three weeks which could only be accomplished through IPTs.

N. The contractors main role in the IPT process is to bring the system development expertise to the IPT and use that experience to the IPT in its decision making. The contractors should provide the technical lead experience for the IPT in the processing of developing the IPT products.

O. We are accustomed to working in an IPT relationship with the Government, supporting programs in that fashion for more than 30 years. Our role varies depending on the nature of each specific task. In some cases, we lead the technical effort (and the IPT) with Government personnel working side-by-side with our contractor team members, in other cases we provide technical advise and specialized skills to tasks led by Government personnel. In none of the cases, do Government personnel work under the direct supervision of the contractor. Final responsibility to meet contract delivery requirements rests with the contractor. We envision the same sort of relationship, proven successful over a 30+ year period, for the CESAR effort.

P. The CESAR prime contractor will serve as the technical expert in a function similar to a SETA contractor. Additionally he will perform studies and analyses on incorporating future capabilities and will serve as the acquisition agent for procuring required services. Configuration management and control will also be the responsibility of the CESAR prime contractor. Recommendation and enforcement of standards among application developers will be a part of the prime's IPT role. In short, the prime's role will be to be the technical arbitrator on what is possible and how it should be done.

Q. No input

8. Do you view contractor provided demonstrations as a feasible part of the selection process? How does the Government reduce contractor incurred costs and provide better feedback on how we evaluate these demonstrations?

A. The government should consider not entertaining contractor provided demonstrations. There are two reasons for this: (1) There is very little to be gained from demonstrations. Contractors will demonstrate, reluctantly, for the work, however, only a modest effort will be put forth. The risk with being left with an unacceptable costly solution and a number of contingency personnel hires will be important factor. (2) Fly-off can be very costly for the government in terms of partial funding and supporting demonstration and testing periods. There should be a definable item to measure and considering the systems involved--that could be a difficult choice. Moreover, fly-off can be potentially disastrous, because unless the government's technical evaluators are very good, solutions that perform satisfactorily in test may not function the same in its actual environment. Therefore, once accepted, the bill will be paid in getting it to work correctly.

B. Architecture demonstrations are extremely expensive. On TBMCS the bidders estimated they spent $2M each on the demos. And for these demos the bidders were given legacy software as a starting point. Nonintegrated demos don't necessarily prove anything; they merely provide proposal elaboration. If the Government requires a demonstration they may be limiting the bidders to those with deep pockets, not necessarily those with the best technical approaches.

C. As an example of contractor technical capability, a demonstration can be a valuable evaluation tool. However, the demonstration must represent a technical solution for well-defined requirements to provide a common baseline for evaluation. If requirements are not well defined, each contractor's demonstration will represent little more than a "marketing pitch". In addition, scoping the problem will serve to control costs.

D. Yes, demonstrations should be helpful to the government. The government could ask for demonstrations of "in-being" systems, or limit the amount of B&P funds allowable for the demo to control contractor costs.

E. Use of a contractor demonstration on a broad scale would be time-prohibitive for the government evaluators and costly for the contractors due to the large scope of this contract. An alternative approach would be submission of the offeror's existing software (COTS) and technical approach for evaluation as to suitability for enhancement to CESAR functionality. Only those offerors whose products were determined to be acceptable would be offered the opportunity for demonstration. This could be done on a briefing versus written basis to minimize contractor cost.

F. Contractor demonstrations would probably not be feasible on a program like this because of the broad scope and O&M aspects of the program. Very specific requirements to produce comparable end-products for demonstration would be required. This could prove to be very expensive.

G. This vendor does not believe a LTD or other types of benchmarking will add any value to the CESAR evaluation model. Existing industry benchmarks are little more than scripted "drag races" as the benchmarks bear little resemblance to actual work being done in the workplace. LTDs or proposed solutions, while potentially helpful, typically prove astronomically expensive to a prime and would drastically reduce the number of bidders given the scope of this contract.

H. We recommend that any demonstrations be limited to critical elements of the solution that the Government has reason to believe will not work as planned. Demonstrations are very reassuring, but expensive. Small demonstrations can be time consuming, cost thousands of dollars, and require significant labor.

I. Demonstrations can be an effective part of the selection process, however there are pitfalls. Sometimes an offeror can quickly implement a demonstration with little technical substance that looks impressive. It may have little to do with the actual technical capabilities. Particularly since many demonstrations are prepared by organizations in remote cities who will probably not be involved in the actual CESAR work. The local contractor community has been successfully demonstrating their capabilities for years. Demonstrations, while interesting and sometimes informative, do not provide a solid level of understanding of the inherent management capabilities of the prime company, nor do they provide a measurable level of the quality and productivity that a contractor delivers on a daily basis. Rather than an emphasis on demonstrations, it is suggested that past performance be the primary deciding factor. Past performance can be indicative of sound management practices, strong engineering operations, cost effective productions and the effective mitigation of programmatic risks. Demonstrations are costly and the contractor can never fully recover the total cost of conducting one. In addition, the cost may be a deterrent to the participation of some contractor teams. It would appear that the funds would be better spent building a strong team or applying them to the solution rather than to a demonstration that may not be applicable in the long run. The evaluations of demonstrations is at best subjective unless the contractor teams are demonstrating the same capability but with different tools and processes. In this case, the tools and processes are being evaluated and not the contractor's real engineering and team management abilities. As a result the demonstrations, for the most part, will not provide an in-depth understanding of the contractors total domain knowledge, ability to effectively support on-going programs and operations, the ability to consistently produce high quality deliverables nor the ability to consistently meet or beat program schedules with cost effective solutions. By basing a Government decision on an isolated demonstration the Government assumes a very high programmatic risk on a program that is essential to the defense of the country.

J. It is unclear to us that demonstrations required in past procurements correlated strongly to contract performance. Demonstrations, by their nature require development of unique software, prototype subsystems, and "show and tell" software. While these may be used to demonstrate understanding of the problem, they tend to be loosely related to the contractor's ability to deliver production quality products and integration. Also, demonstrations may be a desirable part of the selection process from the Government's standpoint, but are very time consuming and expensive for the contractor to set up. In some fashion, these costs must be passed back on the Government in terms or higher labor rates, higher general and administrative fees, greater percentages of operating budgets for Bid and Proposal (B&P), etc. If the Government wants demonstrations, and to keep the contractors' cost low, then the Government would have to establish an unclassified hardware and software suite for them. However, this presents a false sense of acceptance to both the Government, the contractor and potential vendors. We need not remind you of some of these problems. Likewise, the long-term integrated, seamless solutions the government desires may not fit the hardware and software suite established for the tests and/or demonstration. Another concern, short-term solutions would probably fill the need for demonstration purposes, but may lead the Government into an award for the short-term and not the long-term. We feel that demonstrations do play a very important part in the architectural development phase after contract award and recommend that they be used then. Demonstrations are really only useful if a specific objective/problem has been identified and then a measurable demonstration can be used to determine how effective the demonstrated solution(s) is/are in meeting/solving that objective/problem. Otherwise, demonstrations can become the mechanism whereby the Government - Contractor team gets into the mode of chasing technology for technology's sake, wasting precious resources on both our parts.

K. With the shrinking Defense budget and little opportunity to recover from faulty source selection decisions, it is essential that the Customer have maximum insight into "what and who he's buying". It is difficult to make a well-informed decision based upon only the voluminous paper contained in the submitted proposal documents. Demonstrations and oral presentations are becoming more common mechanisms to support the source selection decision, and are not only feasible, but are a highly effective techniques for judging the technical know how of the contractor, the viability of the proposed solution, and the interpersonal rapport with and among the contractor team. The requirements and associated evaluation criteria for any demonstration should be clearly stated in the RFP for a X-1 acquisition or in the statement of work for a X-2-1 acquisition. We highly recommend a two-phase acquisition for CESAR, with a migration demonstration as one of the deliverables during the planning phase. This approach reduces initial contractor-incurred costs prior to the first down select. During the fly-off phase the contractor is greatly incentivized to make the requisite investment required for the win. The contractors' business case is further enhanced because the Government provides some funding for the fly-off phase. The fly-off enables the Government to "feel and touch" the contractor's migration strategy and likelihood of success.

L. Contractor demonstrations in some situations may be a feasible part of the selection process. The technique of providing sample problems and limited response times is especially effective in determining how well a bidding team can come together and solve an unexpected problem or resolve an issue. Costs can be reduced through use of state-of-the-art management tools such as Video Teleconferencing (VTC), etc. It's the next best thing to sitting face-to face. VTC could be used for competitive demonstrations/evaluations and providing feedback on how demonstrations were evaluated by the Government.

M. Absolutely; demonstrations are feasible for CESAR and should be part of the selection process. In a fly-off the Government can reduce contractor incurred costs by selecting appropriate demonstrations that fit well into CESAR's overall development plan; scheduling each demo realistically and most importantly, clearly defining the scope and goal of each demonstration. Demonstrations normally illustrate a notional view of what a system will consist of. The degree of realism that is achievable is a function of a contractor's ability to invest (cost share) and the clarity of the Government's goals. Typically, the government looks either for functional performance or for process in a demonstration. In both cases, the demonstration should show the Government how well the contractor is able, within the parameters of the competition, to meet the performance specifications of the objective system. The degree of fidelity to system functional requirements and performance specifications will be determined by how well the government defines its requirements and how many of them it wants to see demonstrated. Because early and clear establishment of the demonstration's scope is vital to the process, it should not be done in isolation. The scope should be determined by the government with extensive contractor input. This entire process requires close collaboration between all parties. Contractor input will help insure that achievable and affordable goals are set for each demonstration. It will also reinforce the objective of creating a true partnership between the Government and its CESAR contractors. Selecting a contractor with similar, related experience who can capitalize on acquired knowledge, laboratories and skills further reduce costs in development of these demonstrations. Demonstration and evaluation provided through SIPRNET/Internet access can reduce some costs.

N. No input

O. Live test demonstrations are very feasible as part of the selection process. While demonstration preparation costs can be high, the can be offset by reduced requirements for written proposal materials. Oral presentations combined with demonstrations can offer the opportunity for the Government to view firsthand the proposed approach of each bidder, and eliminate the substitution of proposal "vaporware" for real substance. To be meaningful and ensure fair evaluation, the acquisition documentation (RFP and associated TRDs, SOOs, etc.) should provide clear guidance on the objectives, scope, and desired results of the presentations and demonstrations.

P. This is a tough choice because of the scope of the CESAR program. Elements of the proposed acquisition lend themselves to a demonstration, but the central focus of CESAR is to rearchitect and reengineer the existing system. This aspect of the program is more process oriented than demonstration friendly. For this part SEI evaluations, ISO process certification, and related experience may mean more than a demonstration. Security issues associated with this modernization requires that any demonstration planned be carefully constructed to avoid special access requirements. The Government must decide what they want the demonstration to show and decide if the results would provide a meaningful discriminator on a company's ability to perform the contract.

Q. No Input.

9. What avenues do you suggest to ensure open communication with potential bidders from pre-RFP through the award debriefing (electronically and face-to-face)?

A. All possible avenues of communication betwixt and between all potential bidders and government contracting as well as end user personnel must be kept open. The domains considered for inclusions are somewhat closed entities to many offerors, and as such, little can be gained, mutually, through an electronically provided source. Consideration should be given to conducting "Functional" Information Forums over the course of the next several months. This could be set up as a full day presentation, where potential offerors are advised of an Open Forum and the Functional Branch i.e., SWPS, CCPDS-R, CMAH, NPES, OA LAN Office, etc., would in-turn based on an agenda. Functional Area Managers could provide further presentation of their systems, operation and inter-organization coordination processes, and include a question and answer period. A 55CONS representative should be present to ensure contract integrity is maintained. Additionally, a library providing all of the important system level documentation, specifications and processes should be established on Offutt AFB, to be available at anytime, for offerors to visit on a reservation basis but without restriction.

B. The Government is best served by encouraging dialog, rather than limiting it. The playing field is not level and can never be made level. Providing feedback, but not direction, is the best way for the Government to get the very best proposals and technical approaches. The Government must have in place the procedures for protecting proprietary information that may be presented. ESC's HERBB has worked very well. An evolutionary development of the RFP gives the Government the best chance to produce a tight spec for the procurement and gives the contractors the best chance to write a proposal that gives the Government all the information it needs to award to the best-value contractor.

C. We recommend an electronic bulletin board that includes up-to-date information on RFP-related activities and status information. In addition, we recommend periodic industry conferences designed to review Government requirements, discuss Government acquisition strategy and award schedule, and answer questions from potential bidders.

D. Use the WWW to distribute information and use the classic solicitation conference system to allow face-to-face exchange of information.

E. Use of an electronic bulletin board system to provide access to CESAR-related information and documentation may prove to be the most efficient and cost effective method for information flow both from and to the Government. Another avenue might be video-conference updates on a periodic (e.g., quarterly) basis. The current industry day scheduled for September should be supplemented by a pre-SOW meeting to discuss government and contractor concerns and questions.

F. Permit face-to-face meetings/communication as long as possible, at least until the formal RFP is released. The WEB page is a great idea and industry day briefings etc., would be of great value.

G. This vendor believes regular face-to face briefings are the most desirable form of communications. Such discussions minimize the potential for misunderstandings. Electronic bulletin boards, while helpful, frequently suffer from inadequate bandwidth, lack currency, and are not available (Hanscom's BBS is a typical example). The government should consider establishing a web site for this program. Also, the government should consider holding CR/DR conferences as well as providing assurance that any questions submitted by a certain date will in fact be answered.

H. Internet, e-mail, telecoms and collaborative software products are the least expensive. Face to face meetings are expensive and should be saved for extensive reviews and last for minimum of one solid day with a fixed and well defined agenda. Face to face meetings would also be appropriate where there is a need to provide general direction to multiple contractors concurrently.

I. It is our opinion that the USSTRATCOM CESAR community is doing a fine job in the way they are conducting the acquisition of this program. Sufficient information concerning CESAR objectives and scope has been made available. As additional contractual/bidding information is formulated, it should be provided in the form of either a hard copy or community brief. The planned documentation concerning the AS-IS architectures should be sufficient.

J. We support any and all opportunities to participate in information exchanges as long as they are done in an open forum. Establishing an E-mail connection between potential vendors is an easy first step in facilitating rapid communications but may not be useful as a bidders library. Not all documentation needed for the contractor's bidders library will be available in electronic media since some Government purchases have been for user licenses only, without documentation. With this constraint, the Government would be better off providing an open face-to-face bidders library. Current contractors with specific technical knowledge will have some advantages, but there isn't a way around that problem.

K. We believe that frequent, open forum bidders conferences, a classified reading library, an unclassified Internet bidders reading library, and a web site will be to the mutual benefit of USSTRATCOM and interested contractors.

L. Avenues to ensure open communications include: Video Teleconferencing (VTC), Teleconference, Web Site (include E-mail & documents), Phone, Fax, Mail, Face-to-face meetings/briefings; also, provide ombudsman and quick turn around on questions.

M. Open communications between potential bidders and the Government should be encouraged throughout the process. Exchanges should be both frequent and continuous in individual as well as group sessions. Electronic media i.e. a homepage should be used but must be updated often. We also recommend that a video teleconference for all interested parties be scheduled on a regular basis.

N. No input

O. The Air Force Standard Systems Center recently conducted an acquisition similar in scope and requirements to the proposed CESAR effort. That procurement known as AF-GCSS, can serve as an excellent model for communication between the Government and potential offerors throughout the acquisition life-cycle. Contractors were invited to participate in the definition of the RFP sections L and M criteria, the Government established an electronic bulletin board to disseminate procurement information, offerors were afforded multiple opportunities for face-to-face discussions with Government representatives, multiple draft RFPs were produced to generate feedback from the bidders, a comprehensive library was established, and potential offerors were afforded the opportunity to provide input to, and comment on, the SOO, the award fee plan, the CLIN structure, and the cost evaluation model.

P. It is key that a library be established as soon as possible. A procedure that has worked in the past is to establish a two hour block every two weeks for a prime contractor to be able to have one-on-one discussions on issues concerning the program. This provides an opportunity to bring up issues, get answers to questions, and propose ideas in an environment where discussions can occur. An electronic bulletin board is required to keep industry aware of program status and to convey information related to new library documents, acquisition strategies, etc.

Q. No input

10. Commercial Off the Shelf (COTS) and NonDevelopmental Items (NDI) - What guidelines should the Government use in defining and evaluating these items? What problems do you foresee with COTS/NDI support in integration?

A. The Government should provide a hierarchy of priorities to be used to select hardware and software, e.g. 1 - COTS/NDI, 2 - MOD-COTS/NDI, 3 - Design and Development. Before contractor launches a subsystem acquisition and integration he should rigorously search industry and DOD for existing/soon to be released products that satisfy the need. Each acquisition should be preceded by a well documented Life Cycle Cost analysis which fully explores the availability of COTS and NDI, and analytically projects Life Cycle costs of the alternatives. All acquisitions must fall within the parameters of DII, as it has been incorporated into the architecture of the CESAR. These products need to be as complete and free of planned obsolescence as possible (minimize closed ended architectures). Also, all proprietary and intellectual rights must be passed to the government (patents, copyrights, etc.) Suppliers of COTS and NDI must be able to clearly demonstrate how the government will be able to support these items through their life cycle and at what cost. There should not be any special COTS/NDI guidelines required unless specific makes, models and designs are necessary. As example, there are specific hardware integrations included with systems such as GNT, MIDB and others that are either pre-approved by the controlling authority, or interface features dictates various types and sizes. Integration of COTS is the preferred method and shouldn't present any problems, however, NDI could pose problems without specific engineering and testbed evaluations.

B. One of the biggest problems in selecting COTS products is the tradeoff of maturity for capability. Many promising products are the result of "low budget" developments. They are often high on promised functionality but low on robustness. Besides hands-on evaluation the Government-Contractor team should set guidelines related to a products installed base (size, duration); field-identified defects/defect rate; help desk response. A different integration process is needed with NDI/COTS, since this constitutes a true black box integration.

C. We have extensive experience in the integration of COTS and NDI software into DOD IC4I systems. In our experience, successful COTS/NDI evaluations begin with well-defined requirements (i.e., based on mission needs, not products). We then identify COTS/NDI products as candidates for integration based on the following criteria: Does the COTS/NDI item satisfy system requirements? Does the COTS/NDI item comply with our architectural requirements (i.e., open standards, layering of services, etc.)? Is the product viable (performance and quality considerations)? Next, we demonstrate viable COTS/NDI components to verify that the product meets the user's expectations. During integration, COTS/NDI components are treated in the same manner as developed components. They are entered into our configuration management system, integrated and tested in both standalone mode and with other system components as required. These steps validate the viability of the selected COTS/NDI products. This careful evaluation of the product minimizes COTS/GOTS integration problems.

D. Encourage the use of COTS/NDI, but make the contractor responsible for the overall performance.

E. In the software environment, the use of COTS products may offer a viable alternative to new development. For hardware, however, COTS products and Non-Developmental Items (NDI) may prove to be a challenge for logistics support. The RFP should clearly require that full or short form Provisioning Technical Documentation be provided with all hardware, including COTS and NDI products. In either case, use of COTS and NDI puts STRATCOM in much the same dilemma as the company information systems (IS) manager trying to evaluate commercial market products. There are many technical journals that rate products. Further, the size of commercial companies is based in large part on public perception of quality, utility and cost-effectiveness of the company's line. Both of these factors should be major evaluation factors. A survey of methods used by IS managers might also prove helpful in determining evaluation criteria.

F. The integration of COTS and NDI is the way of the future on Government procurements, especially with the recent deactivation of many military and DoD standards relating to custom design. The contractor team should be given the responsibility to select the best solution for a given requirement. The use of COTS and NDI can become a problem for long-term supportability and when customization becomes necessary.

G. The government should ensure that any proper COTS/NDI technologies have a proven record in mission critical application environments. The government should keep in mind that by definition, COTS products will be constantly modernized to stay competitive in the commercial sector. As such, the government should not encumber itself by insisting on a specific product release or platform. The selected prime should be able to demonstrate how new technologies will be inserted.

H. Again, the Government seems to be developing the solution, and therefore in the end the Government will have to assume responsibility for the performance of the system. This may not be the Government's intent or interest in the long run. We are not implying the Government should not provide general direction and approval, but selection should be the responsibility of the contractor.

I. COTS/NDI software and hardware should be proposed by the contractor for use in CESAR. Some of the evaluation factors are standards compliance, cost, functionality with respect to the requirements, ease of use, and ease and cost of maintenance. The CESAR contractor should recommend upgrades/changes to the COTS/NDI baseline as appropriate for Government approval. Even though it is not necessarily a simple task, COTS/NDI integration is a common requirement in today's environment and any competent integrator should be experienced in this area.

J. This process is desperately in need of formalization. The Government needs to publish the Standards for COTS and NDI it intends to follow. We believe that the following should be considered when investigating COTS and NDI products: (1) Product maturity (2) User base (3) Manufacturers stability and commitment to the product (4) Adherence to established, accepted Standards (Government or Industry). We have been able to overcome integration support issues with a wide-variety of vendors selling COTS and manufacturing NDI products, therefore, we do not see any overwhelming problems in this area for the program under discussion.

K. COTS and NDI-intensive system solutions are very common today due to the merits of reduced costs, enhanced maintainability, and ease of future technology insertion. It is critical to carefully assess and comparatively evaluate available products before making a decision. Be cautious of high-pressure marketing and slick brochures. There are a variety of methods for conducting comparative evaluations to select COTS products. These methods typically consist of: defining and prioritizing or weighting the requirements that the product must satisfy; selection (or soliciting) a set of commercial products that meet some or all of these requirements; evaluating the functional capabilities of the products against the requirements, leading to a set of product ratings or scores; operationally demonstrate or benchmark the top rated products; and finally select the "best fit" product. NDI products or capabilities offer the potential for substantial cost savings but are generally more difficult to evaluate because they are not commercially available. It is important that the Government confirm the existence, availability, and viability of any NDI offering from a contractor. This could be accomplished via a demonstration during the fly-off phase. It is important to get a commitment from the vendor or offeror that any COTS or NDI product will be available and maintained for the life of the contract. To avoid costly surprises later, the Government should be very specific with regard to their expectation of a COTS offering in terms of maintenance, upgrades, and licenses. One of the major challenges facing the Government and contractors today is how to cost the design, development, integration, test, and operations & maintenance of COTS/NDI-intensive systems. The traditional lines-of-code based cost estimating methods do not accurately depict the level of effort required to integrate COTS products or reuse NDI. In evaluating contractor proposals for CESAR, look for cost/labor estimation methods that are tailored specifically for COTS/NDI solutions. Lastly, a demonstration of proposed COTS/NDI during the fly-off phase is critical to confirm the real functional capabilities, performance characteristics, and viability of the proposed offerings.

L. The Government should require that COTS and NDI used on this contract be developed by American firms. COTS is being used in a Battle Management C3 (BMC3) project and it seems to be working fine.

M. COI/NDI use and selection is directly related to the successful implementation of the DII COE. The Government should minimize software development and maximize use of COTS/GOTS. It can do this by relying on the contractor to narrow the field of COTS possibilities so that the Government can select the best candidates. In turn, the Government should narrow the GOTS field and allow its contracting partner to evaluate and recommend the best GOTS solution. We anticipate the following problems with COTS/NDI: Single log-on security; Enterprise licensing; Backwards compatibility in COTS/GOTSs components; Reliance on COE Kernel services; Compatibility of RPCs and interprocess communications; and Redundant capability between components.

N. No input

O. COTS/NDI evaluation guidelines should be driven by the requirements - that is, how closely does the product come to satisfying the desired functionality, ease of integration, built in security features, product stability/history, and the stability of the vendor. Potential problems always exist in the areas of COTS license management/configuration control, interface definition and control, version control, and cross version compatibility with other products. Security is frequently an issue in integrating COTS products since many of them have little or no built in security features, and the integrating contractor does not have access to the COTS source code to develop those features (and even if source code is made available, modification of COTS software is generally a bad idea, since any modifications made are not likely to be supported by the vendor, now will they necessarily be present in the future version releases).

P. In hardware products, meeting open systems standards is critical so that if a given product no longer is available, a substitute can be procured. In software some provision needs to be made so that the source code would be available in case a vendor goes out of business. Maintaining current licenses is a factor in the supportability issues associated with COTS integration contracts. As new releases are integrated there may be new problems that require fixes. O&M costs must include the costs associated with license renewals.

Q No input

11. Which acquisition strategy would be best suited for this type effort? Traditional (X-1) or Fly-Off (X-2-1)? (Refer to Information for Industry Background paper).

A. X-1 seems to be the logical and most cost effective for both the government and the contractors. Although the X-2-1 approach may look good on paper, the CESAR program is not suited for this strategy for the following reasons: It is doubtful in our minds that the government can provide a "prototype" CESAR-type subsystem or sample task which can be equally developed by two contractor teams and evaluated in a "fly-off." It is also doubtful that the source selection team can make any better judgment on the results than it can on technical proposals prepared in the traditional manner. The functional subject matter skills necessary to do justice to a prototype development are few and far between -- and are currently employed by the "application" contractors discussed above in question 3. Fierce competition is envisioned for those few subject matter experts. Since only one team will win, half of these experts will be out of work (or viewed as the enemy" after selection to one CESAR contract team. The X-2-1 strategy will be very expensive for the two selected contractor teams. Although the government states it will limit the "fly-off" tasks to $1M each, it would not be unreasonable to envision each contractor team adding several million of B&P to the effort. Might consider a dual or tri award indefinite quantity CPAF task order contract. In fact, the CESAR contract effort is simply a USSTRATCOM reorganization of functions, and necessitates the utmost care and attention to details. Therefore, combining functions within a Commercial Systems Operational Function, should be approached with caution. The first step is the most difficult, and steps to follow are easier because more details are known. Thus, a Multiple Award (X-1, X-2, X-3) is recommended.

B. The traditional acquisition strategy is the best. The fly-off does not give the Government a better evaluation considering the extra money spent and can have a negative impact on overall staffing. What is demonstrated in a fly-off does not clearly represent what will happen in a long-term contract. In a fly-off, both incumbency and front-end investment can have a significant positive influence on initial performance which may not be sustainable over the length of the follow-on contract. A traditional approach levels the playing field by requiring high marks in demonstrated past performance, evaluated integration/software engineering ability, and documented solutions. During the fly-off, the incumbent will most likely have to draw away key people from his current effort. Considering the Omaha hiring environment, he will most likely have trouble keeping adequate staff levels, possibly affecting ability to perform. Any challenger will have to dip into his own strained local pool or import staff (an expensive proposition). At award time, a winning challenger would look first to the imported or transferred staff (usually key people) before considering the incumbent's key people. Valuable resources that the Government would like to keep might end up left out of the picture. A traditional approach facilitates the smooth migration of incumbent staff to the new contractor.

C. We believe that the traditional acquisition strategy is best suited to large-scale acquisitions. A fly-off will increase the cost and complexity of the acquisition process (see our response to question 13). The rapid planning phase will inhibit careful consideration of integration issues (COTS/NDI and GOTS systems) and lock the contractor into a technical solution that may not support smooth evolution as requirements are refined. The fly-off may accelerate the loss of subcontractor skills and specialties as the planning phase is brought to an abrupt end and the performance phase begins. The rapid planning phase may force contractors to overlook important business issues in favor of completing a technical evaluation. A fly-off complicates IPT involvement, which is the most effective mechanism for gaining Government insight into AS-IS and TO-BE architectures. Overall, a fly-off acquisition approach presents greater risks to the success of this broadly scoped program than a traditional approach that allows for a well planned, coordinated execution of the CESAR architecture planning and design phase.

D. Prefer X-l, traditional.

E. Schedule permitting, the fly-off acquisition strategy would provide lower technical and performance risk to the Government. With only about $1M per fly-off offeror, however, the fly-off would be limited to a short duration at a very limited number of sites. The advantages to the Government are the capability to select competing alternative technical approaches and to select the best-of-breed.

F. I would suggest traditional (X-1) for the following reasons: (1) This is not an R&D system(s) development effort where the winner can be readily distinguishable. (2) Fly-off would result in a heavy expenditure of company(s) B&P funds, resulting in a situation described in Question 8.

G. This vendor believes the traditional (X-1) acquisition strategy is the most logical model to follow for CESAR. A multiple award vehicles is a logical model for a commodities type deployment contract. However, this vendor believes such a vehicle for a system integration contract makes little since as there is no guarantee for sustained business.

H. Since the final system performance requirements are unknown and since it will probably take a significant investment on the part of the Government to develop, we understand the Government's desire to have a fly off. The fly off would also insure all of the data that is required to develop a complete solution is available to the contractors. Standard procurement procedures would probably result in the prime contractors making more assumptions about the solution, which equates to risks.

I. The traditional acquisition makes the most sense for CESAR. Due to the nature of the CESAR work (integration) it is not possible for two contractors to both integrate and maintain the same systems. It is not feasible for the Government to provide two equal baselines for use by the contractors to execute integration and maintenance for a fly-off due to cost. The fly-off would become little more than a paperwork process that would only exercise a small portion of the necessary CESAR skills. Normally, paper solutions almost never turn out to be the final design. It would essentially delay the CESAR implementation by a year and the Government would gain little more than a more elaborate proposal that is partially funded by the Government. Additionally, the major risk for this program is contractor program management and the ability of the contractor to provide high quality deliveries in a high production process that meets or beats both schedules and costs. A fly-off will not directly address these aspects. Refer to paragraph 12 for additional rationale concerning demonstrations that also applies to the fly-off concept. With a lightning bolt initiative where the contractor teams are working from a Statement of Objectives (SOO), each will be formulating their individual concept of a Statement of Work (SOW). It is most certain that those SOWs will look nothing alike. Comparing those SOWs will be an apples and oranges process. If the fly-off is based on the two best SOWs, which are most likely going to be different, the final results may be more like an apples to bananas subjective evaluation. With a program as critical as this one is to our national defense, it would appear that a strong objective evaluation would be in the best interest of USSTRATCOM and CESAR vs a subjective evaluation. Other factors to consider include a years delay in initiating the actual program, the cost to the government, the investment by the contractor teams which may reduce the playing field due to tight funding constraints (this could actually promote a buy-in by a large but minimally qualified team), the need for the government to provide two oversight/ management teams for the duration, the throwing away of half of the work at the end, the possibility that the awarded demo may only be partially useable, and if the Fly-Off operation is not clearly objective there exists a real potential for protests that could further add to the program costs and schedules. A program of this scale and magnitude will attract some very aggressive corporations.

J. We support a traditional approach to the CESAR procurement for a variety of reasons: (1) Local Government inexperience in conducting a fly-off likely to result in a lengthy protest. (2) Local Government personnel do not have any set procedures or actual methodology whereby they would be able to provide the necessary information to the two competitors such that one does not gain an unfair advantage over the other. To accomplish a fly-off, the government needs a plan on how to execute a fly-off. (3) The Government would have to dedicate a team of people for the entire period of the fly-off to insure that the competitors' system analysts had their questions answered, customer interviews scheduled and accomplished, etc. With the various other Government functionals and guarantee that requested government written responses be provided to them in a timely manner. (Note: A considerable amount of work by Government functionals (end-users) is a necessary part of this type of System Engineering job. From time-to-time this will likely pose a large disruption to end-user work schedules.) (4) During the period of the Fly-Off, Government employees will likely suspend making critical decisions which could/would introduce additional costs and schedule risk for every current software developer. Why? Government employees would be unsure of which competing architecture would win. (5) The winning design will likely not be materially better than if the procurement were done in the traditional way. However, it would cost the Government $2.0 million, take an additional year (or more) to accomplish, thus stretching out the program. (6) The Government wants contractor participation in the Integrated Product Teams (IPTs) yet this is fundamentally inconsistent with a fly-off approach; the Government IPTs can not participate with competing contractors during a fly-off, therefore, this could necessarily lead to solutions which can not respond to current issues known to the Government. This situation does not exploit Government's knowledge and delays real IPT participation until after the fly off. (7) The fly-off encourages bidders to invest their own funds which leads to internal pressures within the company to protest if their efforts fail to win the business. (8) A fly-off also severely skews the evaluation process. In a move to present the "least cost" case for their proposal, the prime contractors will have to try and make their solutions compatible with the existing "stove-pipe" systems already in place. These proposals may not represent either the best long-term nor the overall least cost solution. In other words, the least cost proposal may not end up being the least cost nor best solution.

K. As previously discussed, the Fly-Off (X-2-1) type of acquisition offers the Government the greatest potential for judging the full range of capabilities of the bidders and their proposed solutions. The protracted fly-off competition enables the bidders to gain a more in-depth understanding of the CESAR requirements and environment, and thus offer a more informed, targeted solution. The Government further benefits by getting to know and interact with the offerors and by gaining detailed visibility into the competing proposed approaches.

L. A fly-off (X-2-1) strategy is not practical in an acquisition that is not precisely defined, measured, and evaluated. The Government would be spending double the money for a year over a traditional (X-1) strategy. Setting up the standards to measure two competitive contractors would be difficult. Monitoring and measuring both contractors fairly would be difficult. Selecting one to continue after one year could lead to the "mother of all protests" on a contract this large. A properly developed RFP would alleviate the need for a fly-off.

M. Although there are strong arguments on both sides, circumstances favor the X-2-1 approach. STRATCOM is about to embark on a program which will result in radical changes in the way the command does business, yet the changeover must be nondisruptive. This suggests that deliberate planning is key to success. The proposed Phase I funding level would permit two small contractor teams to develop a comprehensive solution over an extended period of time. The contractors would be incentivized to give importance to an approach that would offer a seamless transition as well as meeting the technical challenge, rather than one that was put together within the resource constraints imposed by a single step procurement schedule.

N. The fly-off approach will be effective only if the evaluation of the management processes are evaluated at least as heavily as the technical performance. Unless the decision makers have experience with the fly-off approach it has a higher risk for the government than the traditional, known procedures.

O. We do not recommend the "fly-off" approach. Fly-offs typically require a major investment by the bidders, result in dollars being spent by the Government (funding the losing solution) that have no payoff in moving toward the final solution, and may actually interfere with the Government's ability to communicate useful information to the contenders to ensure the efforts stay on track with evolving Government requirements during the fly-off phase. A more traditional (i.e. single award) approach, or a multiple award ID/IQ vehicle, offer the Government the opportunity to evaluate proposed solutions based on demos and presentations, and can offer great flexibility in the post-award period to ensure requirements are met.

P. The traditional (X-1) approach is best for this acquisition. In a competitive Fly-Off phase the IPT relationship would be flawed because of the still competitive nature of the phase, security billets would be a problem in that the time required to get them would delay the process and the loser of the Fly-Off would have excess cleared personnel, costs for the industry would be greatly increased, and finally, this is a process oriented program. What, specifically do you expect to have at the end of a fly-off? TBMCS, solving a similar problem for the tactical forces went with the X-1 approach and could provide the advantages and rationale for their approach.

Q. No input

12. Proposal/Evaluation Criteria - (Fly-Off and Traditional Contract award approach) What do you recommend as the proposal evaluation factors, are they weighted, how should they be verified, and what is their relative importance? How do we measure and evaluate "past performance"? What are the critical components and processes that each bidder must address in their proposals?

Please answer the above questions for both types of contract award approaches.

A. Technical Approach/Methodology (50%), Task Specific - 30%, Past Performance - 20%. Management (25%), Contract Specific - 10%, Past Performance - 15%. Cost (25%), Contract Specific - 20%, Past Performance - 5%. Some of the evaluation factors would be: (1) Technical Understanding (2) Systems Configuration Management Knowledge (3) Requirements Process Knowledge (4) Legacy/Migration System Knowledge (5) Past Performance.

B. For the fly-off, we recommend evaluating Performance Ratings in each category of work and the Cost in man-hours for performance. For the traditional, we think the tried and true factors of Understanding the Problem, Thoroughness of Approach, Reasonableness of Approach, and Cost of Approach are appropriate. Some key issues include: System engineering, development integration and maintenance processes. Objective Architecture. Architecture Evolution. System operations approach. Program Management and team management. Transition to Awardee. Technology choices and technology evolution. Past performance should be measured by relevance in size and scope. The Government should be looking for not only absolute performance but growth through adversity.

C. The success of this program is most dependent on management and technical processes. The Government can evaluate the maturity and quality of these processes through an SEI SCE or AFMC SDCE and analysis of past performance. We recommend that the Government establish a minimum SEI CMM maturity level and evaluate the proposal based on proposed technical and management processes and past performance. In addition, we recommend that evaluation criteria be weighted in the following order (most to least): Past performance - weighted most heavily as evidence of the successful implementation of quality processes. Management Processes - effective management processes are key to the success of a large scale program such as CESAR and essential to effective risk management. Technical processes - standard technical processes will help establish effective control and predictable results over a multitude of technical tasks.

D. From most to least important: Past performance, Technical performance, Management, and Cost. Same for both approaches, with fly-off results being a part of "Technical performance."

E. For either the fly-off or traditional approach, criteria may include (in weighted order) current demonstrated capabilities, ease of modification to CESAR functionalities, life cycle cost (acquisition, personnel, and logistics) and past performance. The past performance criteria should be evaluated on successful experience as an integrator in a large-scale hardware/software system. Since there are very unique aspects to C2 and strategic war planning that will be critical to integration of technical solutions with As-Is and To-Be architectures, another factor should be team expertise in STRATCOM-specific aspects of these two critical mission areas.

F. Past performance on similar Government contracts - Review of CPARS. Technical solution offered. Experience with current STRATCOM programs.

G. No Input.

H. For an effort as critical as this, we suggest that the following be considered as the primary factors: (1) Past experience in outsourcing operations (2) Financial security and resources (3) Technical ability and access to those resources especially to perform in critical periods (4) World wide presence (5) Flexible contracting agreements (6) Access to technology and ability to test and use the latest technology in solving new customer issues.

I. The CESAR technical domains are unique to USSTRATCOM. The systems and programs that make up those domains are also unique. Additionally, the programs within each domain are unique from each other. Many have evolved for very specific reasons over the past history of SAC and USSTRATCOM. While much progress has been made in the evolutionary growth of these program, many of them are still using proprietary components and technology and are considered to be stand alone stove pipe systems. In spite of that, for the most part, they work well and provide an excellent operational capability within their realm of applications. In light of this, the biggest problem the Government faces is being able to select a contractor team that will provide the highest level of productivity and quality at the lowest risk and cost. Due to the long history and complex migration path that produced these program, it is recommended that a strong emphasis be placed on maintaining the USSTRATCOM industrial base. To lose or degrade that combined level of expertise would be potentially damaging to the future of USSTRATCOM and CESAR. The contractor team that will be able to provide that capability will be one that has a comprehensive understanding of the existing domain environments and has already demonstrated to USSTRATCOM their ability to support and produce via technical capabilities and management expertise. Because of the complex nature of this environment and the importance of this program to national interest, the going-in risks associated with this program are high and need to be managed and mitigated. As a result, the emphasis of this competition should be to focus on those aspects that mitigate and reduce the risks to acceptable levels. The most prominent element that will address this is past performance of contractors currently on programs within the USSTRATCOM contract environment. The emphasis should be on proven past performance. Within past performance, the performance of the prime contractor needs to be focused on two elements: (1) program/team management and (2) integration capabilities. Without a strong performance indicator in these areas by the prime contractor the program could be in serious trouble. Past performance by the subcontractors should be focused on their ability to perform within their area of functional expertise that relates to their position on the team and their role within CESAR. For the most part, those contractors are well known by both the Government contracting offices and the program offices. The difference for this program will be how the various teams form and the combinations of expertise that each brings to the table. Cost is an important factor. However, a cost shoot out between teams would be detrimental to the program as might related to labor rates. Current contractor support rates for USSTRATCOM are low. These existing rates make it very difficult for the local contractor community to maintain the high level of expertise required of the USSTRATCOM programs. It is an acknowledged fact that there is a constant flow of good talent from the contractor community to the Omaha civil business sector. Realistic labor rates should be anticipated/expected and those that appear low should be suspect and potentially considered a risk to maintaining the proper level of talent and expertise. However, innovative costing measures resulting from smart business practices, functional consolidations, efficient engineering operations and the use of tools and processes that provide cost effective high volume-high quality production should be taken into account. Management at the prime contractor level of large programs and large teams is especially important to the success of the program. Subcontractor management abilities as related to their areas of expertise and their CESAR areas of responsibility are also important. Technical abilities and capabilities should be focused on three different aspects of CESAR. The first has to do with daily maintenance and support of the numerous programs that are currently within each of the four CESAR domains. This will require a strong past performance set of credentials. The second focus should be related to the teams ability to provide evolutionary modernization upgrades to those same programs. Again, clear past performance should be the measurement. The third focus would be directed primarily at the prime contractor. It has to do with the technical ability to integrate numerous operational programs/systems from a complex environment into seamless operation with out interference with the mission. This also should be based on proven past performance. In summary, we would recommend the following elements for selection criteria: (1) Past Performance (40%) - Management Experience (Large Teams, Large Programs, Multiple Program Integrations) - Domain Experience/Performance. (2) Technical Ability/Approach (30%) - Domain Knowledge - Domain Experience - Domain Performance - Proven Integration Skills/Processes (3) Management Plan/Risk Plan (20%) (4) Cost (10%) - Labor Rates - Innovative Cost Saving Considerations. This should not be a two step or lowest cost awarded contract.

J. The following proposal evaluation factors should be considered for both procurement methods under consideration: (1) Understanding the Problem - 20% - Verified by Government Technical Evaluation (2) Technical Approach - 20% - Verified by Government Technical Evaluation (3) Cost - 25% - Verified by Independent Government Assessment (4) Management Approach - 10% - Verified by Government Technical Evaluation (5) Relevant Past Performance - 25% - Verified by Independent Government Assessment. (Also, see the answer to Question 6.

K. The following table contains sample evaluation criteria, components or topic areas within each, and associated weights for both a fly-off and traditional acquisition approach for CESAR.

TRADITIONAL PROPOSAL                    FLY-OFF PROPOSAL                         

                                                                                 

    EVALUATION            WEIGHT/            EVALUATION             WEIGHT/        
     CRITERIA            IMPORTANCE            CRITERIA            IMPORTANCE      

                                                                                   

Management           1.  400 points       Technical             1.  400 points     

- Corp Commitment                         - Understanding of                       
                                          problem                                  

- Staffing Plan                           - Design Approach                        

- Risk Management                         - Demonstration/                         

- Staff                                     Prototype Plan                         
Qualifications                                                                     

- Migration Plan                          - Applicable                             
                                          Corporate                                

- Project                                   Technology                             
Organization                                                                       

- Key Personnel                           - SEI CMM Status                         

- Program Controls                                                                 

- Mgmt Process                                                                     

- Subcontract                                                                      
Management                                                                         

- Security Plan                                                                    

Past Performance    2.  350 points       Management           2.  450 points       

- Applicable                             - Corp Commitment                         
Contract                                                                           

  Performance                             - Staff & Key                            
                                         Personnel                                 

  (scope,                                   Qualifications                         
complexity)                                                                        

- Scope or                               - Project                                 
Schedule                                 Organization                              

   Change                                - Program Controls                        

- Award Fee                              - Migration                               
History                                  Planning                                  

- Terminations                              Process                                

- Customer                               - Security Plan                           
References                                                                         

Technical             3.  250 points      Past Performance     3.  150 points      

- Understanding of                        - Applicable                             
problem                                   Contract                                 

- SEI CMM Status                             Performance                           

- Technical Approach                      - Scope or Schedule                      

- Technical Design                           Change                                

- Technical                               - Customer                               
Insertion                                 References                               

   Strategy                                                                        

- Configuration Mgmt                                                               

- Quality Assurance                                                                

Cost                4.  Best Value       Cost                 4.  Best Value       

- Cost Realism                           - Cost Completeness                       

- Cost                                   - Cost Completeness                       
Completeness                                                                       

- Discounted Life                        - Discounted Life                         
Cycle                                    Cycle                                     

  Cost                                      Cost                                   

Note that the above table lists sample topic areas within each of the major evaluation factors, which should then appear as topics within a submitted proposal. The criteria for the traditional proposal should also be used for the first phase of a fly-off.

L. Proposal/Evaluation factors in order of preference include: (1) Past Performance (Verified by recent performance on similar contracts) (2) Technical (Verified by responses to RFP, statement of work, demonstrations, problems, etc.) (3) Management - Key Personnel (Verified by resumes of Program Manager, Project/Task Managers, and all senior personnel. Both the company CEO and each individual should sign a statement of intention to support work on that specific Contract, if it is awarded to them) - Procedures (Verified by inspection and rating of SEI CMM level desired or required) (3) Cost (Verified to be straight-forward and devoid of tricky costing schemes). There are contracts being done like CESAR and the Government would benefit by contacting these organizations for lessons learned. For example, CBIRS is a contract being competed out of Los Angeles that involves a fly-off between two contractors.

M. X-1 Approach. (Evaluation Factors, what should they be) The Government should consider all of the traditional factors, i.e., Technical Approach, Management, Key Personnel, Past Performance, Corporate Capabilities, and Cost. (Evaluation Factors, weighting and relative importance) Technical Approach should be the single most important criterion. The rest (except for cost) should be treated collectively and ranked second, and should serve to establish that the contractor has the ability to assemble and manage a large team and successfully execute a project. Cost should be third in importance. Cost should be reasonable and should be considered in the Government's Best Value model, but should not be an overriding consideration in this type procurement. A suggested weighting would be Technical 60%, other non-cost 30%, and Cost 10%. (Measurement of past performance) Until such time as the Government develops an equitable system of determining past performance that yields meaningful discrimination among contractors, past performance should be a "go, no-go" gate. The absence of severe sanctions such as debarment or cure letters should give the Government confidence that performance as at worst satisfactory. Attempts at further refinement become extremely subjective and, at best, arrive at a judgment on a contractor team that no longer exists and may have little relevance to the procurement at hand. (Critical components and processes) The contractor should demonstrate experience in projects of similar size and technical content and a set of proven tools and methodologies that are applicable to the procurement under evaluation. X-2-1 Approach. (Evaluation Factors, what should they be) The Government should consider all of the traditional factors, i.e., Technical Approach, Management, Key Personnel, Past Performance, Corporate Capabilities, and Cost. (Evaluation Factors, weighting and relative importance) Technical Approach should remain the single most important criterion, but its relative importance should be decreased in Phase I of a fly-off. A suggested weighting would be Technical 40 % and other non-cost 60%. (Measurement of past performance) See above response. (Critical components and processes) See above response.

N. No input

O. We do not expect that the basic evaluation factors would change appreciably regardless of the acquisition approach. The evaluation should be driven by the requirements, not the contracting strategy. For either approach, risk mitigation is paramount. Past performance on efforts of similar size and scope, and the proven maturity of the offeror's engineering processes are the best measures of potential risk. Past performance can be evaluated by examining the scope of existing contracts and interviewing (in person, or by questionnaire) the customers for those efforts. Information on cost performance and award fee history can be provided by the bidders in their proposals, and can be easily verified by the Government. Since the CESAR effort is broad in scope, it is likely that the bidders will consist of fairly large teams with multiple companies represented. The experience, performance, and proposed approach of the prime contractor in managing large teams of subcontractors should be a major evaluation item. Other important processes are software engineering processes in each of the key process areas, as defined by the Software Engineering Institute's Capabilities Maturity Model, the offeror's risk management processes, and cost/schedule performance management processes.

P. I would suggest you host a Government/Industry working group to establish evaluation criteria and evaluation factors once the scope of the program is established. Until the scope is established and promulgated, it is premature to address the subjects of this question.

Q. Evaluation Critical should include a meaningful Small Business Subcontracting Plan. This should include all Small Businesses not just SDB and Women-owned businesses. The Small Business Subcontracting Plan must include high technical support (i.e., system engineering services, software maintenance services, hardware maintenance services, and configuration management services) as well as Hardware/Software purchasing/manufacturing throughout the life of the contract. The Small Business Subcontracting Plan must include the basic work element as well as future task orders.

13. Given a Traditional approach, the Government would expect a detailed technical, management and cost proposal for the planning and performance phases from the contractor. With a fly-off approach, the government would expect a fly-off proposal (planning phase), contractor references, and contractor qualifications. What is the estimated amount of time and cost it would take to prepare a proposal in each of the two scenarios?

A. With the traditional approach, it could be anticipated that each technical proposal would cost $3+ million. If four teams submit proposals, total B&P could exceed $10M. With the fly-off approach six or more teams would attempt to qualify, each spending $1 million. When two are selected, each could expect to spend $4-5 million above the government's $1 million funding line. Total B&P expended could be $14 million. Although preparation of the technical proposals under the traditional approach is more complex than the "immediate down-select" phase of the fly-off, the fly-off approach would take longer to reach the final conclusion due to the one year fly-off competition. Not only would the total funds expended in the fly-off be higher, USSTRATCOM would not realize CESAR benefits until further down the time line. The Streamlined-Traditional approach would take approximately 45 days from notice and $35,000 - $75,000 to accomplish the proposal. The Fly-Off approach would take approximately 180-270 days to complete from end of demonstration, and $500,000 - $1,500 000 to accomplish the proposal.

B. The cost for the fly-off approach would run from $1.5M to $2.5M and include both proposal document costs and company matching investments. The traditional approach would be in the $750K range. For a 1-step procurement, 45 days from the formal RFP release is needed. In reality the bidder will have been working against the draft for several months. All told it would require about 75 staff months. If we assume that in a 2-step the Government essentially requires the same information, but in different phases, then the total effort would be about 100 staff months. Of course, if the Government expects a more detailed proposal in the 2-step, then the effort is probably closer to 150 staff months.

C. For a traditional approach, dedicated proposal personnel support the proposal effort from the time the Government announces an intent to procure services throughout final proposal submission (often several months of effort). Once a draft RFP is released, technical personnel augment the proposal team. Once a final RFP is released, a proposal team comprised of technical and non-technical support and proposal personnel are dedicated to the proposal effort until proposal submittal, usually 45 days. For a fly-off approach, dedicated proposal personnel will support the proposal effort from the time the Government announces an intent to procure services through the initial proposal submission. In addition, these personnel will monitor activities during the planning phase and will again dedicate their efforts to the final proposal. Technical and support personnel will participate in the planning process for both proposals. In our experience, a fly-off easily doubles or triples proposal costs due to the necessity of submitting two responsible proposals. Proposal activity extends from the initial procurement to the fly-off award with extended strategy planning bridging the entire process.

D. No estimate at this time.

E. It is recommended that the fly-off proposal requirements include priced options for the production of CESAR products and the production of the products. This will preclude development buy-in by a contractor, with high production costs after down-selection to the production offeror. Obviously, the fly-off option will be a less costly method to prepare a proposal, but not by much since the planning phase will, by necessity, include detailed requirements analysis, alternatives development and analysis, a maintenance plan, and detailed cost proposals.

F. Traditional - $1.5M - Cost (Prime) 6 months. Fly-Off - $1.5-2.0M (Prime) 6 months plus the time required for fly-off.

G. No Input.

H. Without the scope of the procurement, the direction for the new systems, and the degree that we are pushing technology, there is no way to assess the time. The Government should however keep in mind that it currently takes the Government and the vendors about a year to develop a solution, document it in a proposal, and go through several rounds of questions/answers and system changes to reach an agreeable solution and contract after the RFP has been finalized. There is no reason to believe this time would be reduced under either scenario.

I. Traditional: 30 day proposal turn-around. Suggested page limits: Technical - 100, Management -30, Related Past Performance - 30, Cost - None; but contents must be related to and support cost. Fly-off: Same.

J. The cost for preparing a proposal regardless of type is dependent on a number of factors such as team member' support, location of proposal production, availability of engineering staff for proposal support, previous knowledge of customer, strategy, etc. Therefore, we are unable to provide a cost estimate at this time. However, we believe that 45 days for the fly-off (plus 1+ years for the X-2 to X-1 down select) and 90 days for the traditional method should be more than sufficient.

K. For a traditional, single phase acquisition for CESAR, the optimum proposal response period is estimated at 90 days, assuming that a complete, accurate, and concise RFP is released by the Government, precluding lengthy extensions. The estimated cost for preparation and submittal of a traditional proposal for CESAR is in the several million dollar range. For the Fly-off, two phase acquisition approach, the proposal response period is estimated at 45 days, with an estimated cost in several hundred thousand dollar range; however, the post-award fly-off phase would also involve a substantial additional investment by both the Government and competing contractors.

L. The estimated time to prepare a proposal would be 45-60 days for either with the cost dependent on requirements and the technical complexity of the SOW or SOO.

M. A proposal should take no more than 30 days in either event, provided that the time is not lost in obtaining clarification of RFP requirements. The costs should not be significantly different. Most contractors spend bid and proposal money in proportion to the anticipated reward. A contractor might be willing to spend more under one type of procurement if it was perceived that the probability of a win would be greater than on the other type.

N. No input.

O. Given the complexities and broad scope of the CESAR effort, it is difficult to envision a reasonable proposal cycle of less than 60 days for written submissions. If there are live demonstrations, Software Capabilities Evaluations, Oral Presentations, etc., that period may need to be longer in order to ensure the Government is receiving the best that each offeror can provide. Streamlined acquisitions are desirable, but "rushed" solutions seldom hit the mark. The specific scope of the CESAR effort is still too fluid, and the acquisition strategy is not well enough defined to give a more precise answer on the length of time or dollars required to effectively respond to the CESAR solicitation.

P. Preparing winning proposals takes a lot of preproposal effort that would be common to pursuing either approach. While the actual writing of the fly-off proposal would be simpler the total cost including the preproposal activity would not be significantly less. Actual cost estimates need more information as to size, required plans that would need to be submitted and costing detail required.

Q. No input

14. The potential fly-off will be for a six to twelve months performance period with a budget of no more than $1M per each of the two fly-off contractors. Knowing that we expect a design solution and services plan that would be implemented immediately at the conclusion of a fly-off period, do you foresee problems with this approach? What adjustments would you make to either the length of the fly-off performance period, money apportioned, or product expectations? What advantages/disadvantages do you foresee with the fly-off approach (focus on areas of risk)?

A. The two contractor teams chosen for the fly-off will have a very difficult task managing their Government funded $1M, their corporate B&P contribution, the secrecy and proprietary information controls necessary to protect their fly-off solutions. Due to the classification of the reference material and the availability of legacy subsystem expertise, the fly-off work will almost certainly have to be done in the Bellevue/Omaha area. Once the winner is announced, a 60 day minimum ramp up time line would be required to begin performing tasks on the CESAR contract. Loss of composite innovative solutions due to the inability to effectively combine the best part of two ideas submitted by separate contractors. The government will of course own the rights, but it may not have the necessary know how. Problems will be: Reduced Bidders List, High Contractor Risk and Fragmented Application.

B. No input.

C. We believe that the fly-off approach presents several potential problems (as described in our answers to questions 11 and 13). The most important of these risks involve inadequate requirements definition. Large scale system integration requires a significant financial investment and inadequate fly-off planning will need correction through ECPs and contract extensions. Therefore, a fly-off has the potential to greatly increase the overall cost of the program.

D. Large, well established contractor may buy in to fly-off approach, at the immediate expense of smaller contractors, and at the ultimate expense of STRATCOM.

E. Unless the number of fly-off sites is extremely limited, the $1M fly-off ceiling would preclude a six to twelve month demonstration and validation period. It would be prudent to establish the desired fly-off period, designate the fly-off operational sites and number of users or systems, and have the fly-off as a not-to-exceed price in the contract. $1M for the fly-off approach is likely to limit the planning and analysis phase of this contract. This may prove penny-wise and pound-foolish. Thorough up-front analysis and planning is crucial to the success of this contract. To arbitrarily limit the planning phase by time and funding is not prudent.

F. No comment

G. No Input.

H. It is impossible to address the period or the amount of funds, without details of the target systems; however, a fly off operation will give the vendors direct access to the current system and all of the information and personnel that are required to develop a complete and accurate system solution. It would also allow for the interaction between the government and the vendors as the solution is developed and thus reduce the time the government requires to select the solution. Care must be taken on the part of the Government to keep the vendor data secret and to treat each vendor equitably, so there is no chance of protest. We think the period and the amounts indicated are reasonable. Again, demos may be required, but we suggest they be performed after the preferred vendor is selected as part of the final performance evaluation.

I. As we have stated in other sections of this paper, we believe that a fly-off concept for CESAR is a high risk operation with potential impacts to the mission of USSTRATCOM. However, if the Government elects to conduct a fly-off, we would recommend funding levels of between $2M and $3M as being more realistic for a one year effort. In past cases such as this, fly-off teams have invested even more. The major problem with a paper fly-off of this nature is the potential lack of objectivity and the limited evaluation of the most critical success factors related to this program. Specifically they include: domain experience and knowledge, ability to manage, modernize and integrate a wide range of complex systems and sub-systems without an impact to on-going operations, the ability to reduce program risks to a minimum and the ability to effectively manage a large contractor support team. Proven experience at accomplishing these activities with innovative cost and schedule reductions is essential. We do not see any serious issues with immediately implementing a design solution or service plan at the end of the fly-off performance period.

J. Implementation would require a startup phase which could be extensive. The competing contractors will not put staff in place for the implementation until after award due to the uncertainty and cost. Therefore, if their existing staff needs augmentation, the startup phase could take some time while staffing up occurs. The length of the Fly-Off is an issue. A Fly-Off is a lengthy process if done right. The actual costs of doing a Fly-Off having useful products will exceed the $1.0 million. The money appropriated should be increased by 2 to 3 times to adequately fund and insure a quality product. Product expectations are for a thoroughly detailed 7 year executable plan. This may not be realistic given the rapidly changing technology in the communications-computer field, evolving USSTRATCOM mission, etc. The Fly-Off introduces risk for the contractor and the Government: (1) Increased likelihood of protests due to local Government inexperience in performing this type procurement: (2) A virtual work stoppage as decision-makers await the outcome. Likely, a waste of a year (or more) determining an architecture which you could have had in hand upon contract award using the traditional procurement method; (3) The winning design will not be materially better in detail than what one could expect from the traditional procurement method due to the inability of the government to fully participate in an IPT environment with uncertainty surrounding future technological developments, future requirements from an evolving command mission, etc. (4) Cost the Government at a minimum $2 million. (5) To preclude a series of protests, the Government must have its requirements and missions defined explicitly. With this much background work done, the Government might just as well run a "traditional" procurement not funding anyone's proposal, applying those funds toward the new contract. (6) With a Fly-Off the original two down-selected contractor teams will not be positioned to drive associate contractors (other Primes - who are already doing work). The Government cannot begin implementing the Fly-Off winner's work until after the downselect to a single Prime. (7) There is the potential for loss of not only one of the CESAR Prime's work, but also the loss due to another year (or more) of all the other Prime software developer's work.

K. It is recommended that the fly-off period be for a period of twelve months to assure adequate time for capturing and delivering all necessary background information and legacy system access to the competing contractors, and allowing them time to effectively analyze the inputs and respond with a comprehensive design and plan. The major risk area for either acquisition approach is the availability, accuracy, and completeness of the "AS-IS" architecture and associated operations information. Frequently, the system documentation and operations procedures for mature systems are not current and reside in the heads or informal notebooks of the operators and maintainers. It would no doubt be prudent to assess the state of the system documentation today and take any necessary measures to upgrade that information to a relatively current level. It is important to note that the acquisition of necessary documentation or system access to support the CESAR acquisition will become far more difficult and risky if the application contractors are among the CESAR competitors.

L. Answers to these questions raise the following issues: Fly-off approach may be time and money wasted. A design solution could possibly start sooner with a traditional approach. A fly-off may not lead to a better design solution than what is proposed in a traditional approach. I can't think of a way to quantify the evaluation of information services' design solution and plan. I think subjective judgments would lead to protests. It is much different than having a fly-off between competing physical systems. It would be necessary to coordinate/meet with the government giving status briefings and receiving guidance to keep both participants' effort on track. Assuming access to classified information would be necessary, a period of six to twelve months may eliminate offerors employing leading industry experts who may not currently hold a security clearance. How can you ensure that both competitors receive no special treatment when guidance is given? How do you measure their progress? Would their progressive PMRs clearly indicate a winner or would it become a subjective judgment by a Government Source Selection authority? Would any of the "design solutions and services plan" developed by the contractors be implemented during the fly-off? Will there by any actual programming written and implemented? How can any company confidently identify individuals who will be available to do the implementation work one year downstream, or do you plan to ask for resumes/individual qualification of employees doing the implementation work?

M. With a fly-off, the Government is, in effect, buying a study on a firm fixed price basis. There are thresholds of time and money below which the results will be less than satisfactory. Likewise, there are resource levels above which the quality of the product will not be significantly improved. A period of six months is probably adequate, and a year probably borders on being excessive. Realistically, a million dollars would probably fund a team of six relatively senior scientists and engineers for six months. This team would have the necessary skills and experience to produce an implementable plan in the allotted time.

N. Risk: The fly-off will probably require a Traditional phase to reduce the number of competitors to two. Otherwise the fly-off costs will rise dramatically. The Traditional phase will then require additional contractor bidding costs. If you reduce the scope of the traditional phase then it becomes much harder to find the discriminators between contractors because the proposal will have to be reduced in scope. Risk: Implementation immediately at the end of the conclusion of the fly-off is a high contractor risk. By its nature the fly-off is of limited scope. Some amount of time must be provided at the end of the fly-off for the winning contractor to purchase materials, add additional personnel (difficult in the current job market), train the new personnel and move into the work facilities. This would entail a significant delay in the start of work on the upgrades and integration. Recommendation: Make the fly-off period in the six month range. Advantage: The government gets to see the contractor perform over a period of time and gets insights into the contractor's management processes and metrics as well as the contractor's technical competence.

O. As stated in response number 11, we do not favor the "fly-off" approach. Funding of $1M over a 6-12 month period is unlikely, in our opinion, to provide the Government with a meaningful design solution that can be fairly evaluated, nor will it favor the close coordination and cooperation between contractor and customer that is so critical in the design phase of any project. Our experience has shown that "fly-off" designs are usually discarded at the conclusion of the fly-off, and a fresh start is required. When this occurs, the money invested in the fly-off is essentially wasted, since little progress toward the real solution has been made. It is also our experience that fly-offs require substantially more investment on the part of the bidders than do conventional acquisition approaches. While fly-offs may serve a very useful purpose in selecting a vendor to manufacture a piece of hardware or a weapon system, they may be of lesser utility in selecting an integration contractor.

P. See response to question ll. In addition, a fly-off will delay the start of transition by the length of the fly-off period. $1M split between multiple team members over 6 months to 1 year doesn't generate critical mass.

Q. One disadvantage of the concept of a fly-off approach for planning purposes is that during the CESAR Fly-Off (planning) Phase, preexisting incumbent contractors currently performing on the programs/contracts to be consolidated under the CESAR contract will, presumably, continue their performance on the current contracts through the remainder of the contracted period of performance (or until the inception of CESAR Phase Two) while at the same time cooperating with the two fly-off competitor contractors' CESAR planning activities. In reality, the likelihood is that each and every one of the incumbent contractors will also be a team member on one or the other of the fly-off contractors--and will therefore will be naturally partial to, more cooperative and helpful to the fly-off competitor with which they are teamed. Such partiality will seriously bias the competition for Phase Two (implementation) in favor of the fly-off contractor with the most incumbents on their team.

15. In light of a potential fly-off approach, (knowing that immediately after the fly-off the continued contractor will be expected to begin implementing their solutions), what data requirements do you feel would be necessary? (The intent is to keep data to a minimum.)

A. If the fly-off tasks are hypothetical examples, the winner would have to shift gears quickly to get back into the "real world." If the fly-off tasks are real world, the winner would have to have complete access to other contractor data (potentially company private) to begin performing that task. The use of proprietary information and claims to intellectual property rights would have to be rigidly controlled. Perhaps treat the fly off as a contest between prospective primes with the loser having the option to team as a sub with the winner.

B. In light of the statement to keep the data to a minimum, most information for SWPS and OA LAN will be available in the Phase III documents currently in development, particularly the Architecture Master Plan. We would also need these documents for the C2 and INTELL as well as the current and proposed manning levels for the services areas of the contract.

C. A contractor will require any documentation that describes validated user and system requirements (As-Is and To-Be models). In addition, the contractor will require any available technical documentation on USSTRATCOM C4I systems (such as run-time interface documents and existing software code).

D. No estimate here.

E. Data requirements should be defined to the extent that data is required to assess the successful CESAR offeror. Full contract data for hardware and software products should then be a part of the options for production.

F. No comment

G. No Input.

H. All data required to understand the current and future operations must be made available to insure two viable solutions. Restrictions on the availability of data would be counter productive and lead the vendors to seek changes to the contract upon award if assumptions are off the mark.

I. There would be no additional data requirements. Our team will come to this program with a solid foundation of systems, program and domain knowledge and experience along with clearly established/proven management capabilities. We will be bringing a low risk approach to CESAR.

J. Regardless of the procurement approach, data requirements will be substantial. The Government will have to make public its current data processes, database structures, application designs, code, etc. Just as a winning Prime, any and all unsuccessful bidders (or non-selected bidders in a Fly-Off) have an equal right to all data requirements and directions the applications developers are currently taking. Without full and open disclosure of the current SWPS hardware configurations, commercial software purchased, applications being undertaken, database structures to include record formats, data definitions, etc., some contractor will likely claim that they do not have a level playing field. Therefore, we recommend that all reasonable requests for government non-proprietary data be honored, a library established, and the Freedom of Information Act rules be strictly followed.

K. It is assumed that this question is intended to clarify the data that is essential to commence the fly-off phase, which is at a minimum as follows: "AS-IS" architectures including hardware, software and communications; mission or business process requirements; soft copy of "AS-IS" applications software source code; any "TO-BE" architecture findings; system data flows; internal and external interfaces; definitions of data holdings and data volumes; key operations requirements and products produced; concept of operations; system usage characteristics; and key performance requirements or goals.

L. The government should provide as much data as necessary to permit offerors to develop a comprehensive design that can be smoothly transitioned into the implementation phase.

M. The design team would require data that is descriptive of the systems in being, the functionality to be migrated to the integrated system, anticipated system usage data, and projections for system enhancements and add-ons.

N. No input.

O. It is difficult to understand how a design fly-off can be fairly and thoroughly evaluated unless a full suite of design documentation is developed. If data is "kept to a minimum", what will be evaluated to downselect to a single offeror? At a minimum, architectural designs, top level system design documents, requirements documents, and interface definition documents would seem to be necessary.

P. Until the scope of the contract is determined, data requirements cannot be identified.

Q. No input

16. We expect continuity between the potential fly-off and implementation phases. We expect personnel qualifications in the performance phase to be as good or better than those personnel working the planning phase, and we expect some (most?) portion of the work to be done locally. Please comment on these assumptions (let us know what is realistic).

A. Continuity between the fly-off and implementation phases would largely be dependent on the fly-off task provided by the government. If it a hypothetical task, a longer time to do a mental switch in thinking would be required than if it were a real world task. It is unreasonable to expect that the same people will be used in the performance phase as were used in the planning phase. The contractor's have different goals. The planning phase goal is to win at all costs, the performance phase is to make money. Agree that much of the work will have to be done locally. The more the tasks are tied to the functional requirements of the communities, particularly the SWPS portion of CESAR, the higher the necessity to perform the work locally. Accredited SCIF space in Bellevue will be renting at a definite premium unless the government can provide two equal SCIFs at Offutt AFB during the fly-off period. The only assumption to be criticized is the local performance of work hang up. Preference would be a globally distributed interlaced network of resources for optimum product fitness for use by the end user. Most operators could care less if it was made in Timbuktu or Chicago; does the product work right when they want it and if not can it be serviced yesterday? The second part of that question is not necessarily dependent on local performance. Insert a key personnel clause in the RFP to ensure personnel qualifications in the performance phase are as good or better than those personnel working the planning phase. In addition, there must be some trust on the part of the Government that the winning contractor team will put their best and most qualified people on the job. The assumption that some (most?) portion of the work should be done locally is valid and should be expected so the Government (STRATCOM/J6) can monitor progress, participate in the DT&E, and ensure requirements are being satisfied.

B. We would expect the same experience levels to work the fly-off and the actual program if the fly-off effort reflects the actual work to be performed. All work should be done locally.

C. We believe these are realistic expectations. We expect most technical personnel involved in the planning phase will participate in the performance phase, providing continuity and expertise to an expanded performance staff. We also expect most of the work to be done locally if the prime contractor and/or major subcontractors have a large local presence. However, a prime contractor located elsewhere may choose to perform a large portion of the work at the contractor's primary location.

D. Your continuity concept is reasonable; personnel will not be same; and, there should be no problems in doing a lot of the work locally.

E. Use of a Substitution of Key Personnel clause in the contract would ensure continuity of the skills between development and production. "Locally" needs to be clearly defined and there must be a realistic rationale for designation of a local area in which development and/or production must be accomplished. Restrictions on the sites for either development or production would constrain competition and may form the basis for protest. Local interaction will prove crucial, especially in the requirements and process-definition phases of the planning work, but there are many areas of the work which should not be locale-restricted.

F. If the award criteria becomes a "rate-shoot-out personnel qualifications may be less than what is required. It is not unreasonable for the government to expect most of the work to be done locally. The selected contractor team should have a large, stable base of qualified employees to support all phases of the program.

G. No Input.

H. If you are buying people, your assumptions are reasonable. For the major portion of this contract, you should be buying the delivery of services at agreed to performance levels. In this environment, there is no need to address people. The customer should be interested in performance and leave the skills requirements to the vendors.

I. Realism, from our business practices perspective and for all of the reasons cited earlier, would be to conduct a traditional competition. It is also realistic to understand that in most past fly-off competitions, companies go to their most powerful and capable engineering houses to develop the best package possible. With the competing companies investing from $2-5M each in the outcome, they will want do to what ever is necessary for the fly-off. However, if the Government elects to conduct a fly-off and the rules create a level playing field, we consider your expectations to be acceptable and reasonable.

J. Is it realistic to expect that those working a planning phase to be the same as those working the performance phase? The government must realize that the implementation phase would likely require a different skill mix than the planning phase. For sure, it is not realistic to expect a better qualified staff. The key to success is to have the right talent for the right tasks. It is a waste to have top systems analysts/designers doing software coding functions. Likewise, there's no sense trying to get a coder to synthesize systems architectures. We understand that the Government does not want a less qualified staff designing a system and then having better qualified staff implement the design. However, we believe that the government should best leave this up to the CESAR prime and concentrate on seeing that the prime meets schedules and cost projections, etc. Staffing is an issue only if the prime can not find the proper personnel with the proper skill sets to accomplish the job as bid meeting schedule expectations. Also, where the work is performed should not, for most tasks, be a material concern either. Given that local work is likely to result in lower costs, the competing contractors will have to deal with this during the competition to be competitive. However, he will need to have the flexibility to obtain the services he needs wherever they are located. Whatever the prime decides as to where the work is to be performed, he should handle this in his proposal through cost trade offs remaining as competitive as possible.

K. These assumptions are valid. The key personnel in the fly off phase should be contractually committed to the first year of the performance phase. Substitutions, where they are necessary, must be at least equally qualified. The performance phase, however, will require significantly greater numbers of people with a broader range of skills and depth of knowledge. It is reasonable to require that a large percentage of the work be done locally, however, it may be more cost effective to do portions of the work at other locations. For example competing vendor products may be evaluated at out of the area contractor test facilities vice replicating these facilities in the Omaha area.

L. It is realistic to expect continuity between the design/planning (fly-off) phase and the implementation phase. However, it may take different skill sets, education and experience to design compared with what it may take to implement solutions. Key to this contract's success will be cooperation, coordination, and comprehension between the Government and the contractor. This would be difficult to achieve unless most of the work is accomplished locally. The Government's assumptions on corporate continuity are realistic and necessary, however personnel qualifications between those who design the system and those who implement and maintain the system may be significantly different.

M. We would expect that most of the work would be done locally, but it would be unrealistic to suggest that it would be 100%. Personnel employed in the implementation phase would be as well qualified for their jobs as those in the design phase, however it would be noted that the skill sets required for the two phases are not identical.

N. The expectation that the personnel qualifications in the performance phase of the contract will be as good or better than those personnel working the planning phase is equivalent to assuming that the winning company has used less than its best management and technical staff in the fly-off phase. This is an unrealistic assumption. A more realistic expectation is that the management and technical staff will put sound processes in place and train the personnel working the performance phase of the contract. It is also realistic to expect a core of those management and technical personnel to remain on the project through project completion. The expectation that the work be performed locally is realistic but it is unrealistic to assume that the full implementation will start immediately after contract award. Staffing in a tight job market and the necessity to move people into the area will cause implementation delays as will training of new personnel. This is one area where using small local subcontractors could be involved in the contract, provided that the subcontractors were integrated into the prime contractor's work force and not kept separate.

O. It is reasonable and realistic to expect that most of the work can be done locally, and that there be no degradation in the quality of the staff from fly-off to continued performance. It may be unrealistic to expect an offeror to make a significant commitment to secure local facilities and transfer staff to the local area during a relatively short fly-off period. Bidders who do not already have a major presence in the Omaha area may not be able to maintain their fly-off staff for continued performance - another risk in pursuing this acquisition approach.

P. See response to question 11. Without a fly-off you will eliminate the personnel problem. Even with a fly-off the types of personnel required for the implementation phase may be different. It is obvious that parts of this contract must be done locally, but until the scope is defined the optimum location of specific tasks cannot be determined.

Q. No input

17. Are you aware of a similar contract as this? (used Lightning Bolt, SOO vs. SOW, two-stage downselect, infrastructure, complexity, etc.)

A. IC4I, TBMCS, DEIS are all examples of similar acquisitions albeit not a two-stage downselect. SOF/ATS was a two-stage downselect as was PEACE SHIELD. Considering the magnitude, expected cost and potentially disastrous outcome, why not use the Defense Enterprises Integrated Services (DEIS) contracts for this effort? These contracts are in being because they: Save considerable funding. Removes the contracting burden from USSTRATCOM (55CONS). Already include the integration multiplicity.

B. We have experience in at least two procurements that use Lightning Bolt initiatives and a SOO. CCPL was modified Lightning Bolt in that an SOO was provided, but rather than responding with a SOW, the bidders respond with a Proposal Plan. This is due to the nature of the contract which is essentially a task order type contract rather than a development and management contract. The Allocation, Quality Review & Analysis Software Support Contract was a Lightning Bolt initiative with the issuance of an SOO and a contractor developed SOW response. Our most recent experience with two-stage downselect was SOFPARS. We were one of the initial selectees, however, the procurement was canceled after the first downselect. Companies in a downselect are gambling with a great deal of their own resources. Similar contracts in terms of infrastructure and complexity include our company's contract to manage the Armstrong Medical Research Lab in Dayton, to provide brigade level training and exercise battle staff simulation to the Army at BCTCP school and widely geographically dispersed operations, and to develop, manage, and operate a complex interactive air defense simulation system (TACSSF/nee IFFN).

C. Several large-scale contracts have been acquired under the Lightening Bolt initiatives. For example, the IC4I contract is a large-scale, open-ended contract for Integration for Command, Control, Communications, Computers and Intelligence (IC4I) systems. The Global Combat Support System-Base Level System Modernization acquisition procured by SSG will provide a common infrastructure throughout the Air Force at the base level. Further, CECOM out of Fort Monmouth has recently combined 250 contracts under 26 umbrella vehicles.

D. Yes. Use of Lightning Bolt Initiatives, SOO vs. SOW, two-stage downselect, and contracting for program support are common. The complexity of the project at this point does not seem to be a major problem, but one that can be worked.

E. The current RFP for the DoD-wide Standard Procurement System is a fly-off approach for a COTS-based procurement Automated Information System. The Contracting Officer is Mr. Ashley Eanes, Naval Information Systems Management Center, (703) 428-1004.

F. Yes - Two-stage down-selects become rate/price "shoot-outs" - you'll get what you pay for (e.g., Network Support Program - AFSPCMD/50th Space Wing)!

G. No Input.

H. We are not aware of any contract similar to this one in the Government.

I. This has got to a first and as a result it possibly could be taking on additional levels of risk which may be another reason for more seriously considering a traditional competition. The complexity of the systems, sub-systems and domains of CESAR coupled with the seriousness of the USSTRATCOM mission might indicate that a more conservative/traditional approach be evaluated.

J. We do not know of any examples of a "Fly-Off" approach similar to that being suggested for CESAR being used in a highly data processing oriented program. For a development effort with a tangible product, such as military vehicle, army tank, or jet aircraft, this approach has some merit. The Government goes into this type procurement with a defined, core set of requirements which must be met. This is not the case with CESAR. To some extent, all existing applications, databases, and hardware suites are entering arguments into a CESAR solution. CESAR will require development of an explicit set of requirements before it can use a Fly-Off methodology. Without an explicit set of requirements as a basis, the Government will have a hard time developing good evaluation criteria upon which to base any award.

K. The Air Force is acquiring the Global Transportation Network, the Theater Battle Management Core System, and the Base Level Systems Modernization programs in a similar manner, the NASA EOSDIS Core System is another example of a contract which has at least some elements which are similar to the CESAR contract.

L. See Item 12, however, we are not aware of similar contract (used Lightning Bolt, SOO vs. SOW, two stage downselect, infrastructure, complexity, etc.) being used to obtain Information Services Support.

M. The Air Force is currently in the proposal evaluation phase for the Global Combat Support System Base Level System Modernization (GCSS-AF (BLSM-II) contract. The GCSS-AF (BLSM-II) Program is being procured via the "Lightening Bolt" procurement initiative. The RFP for the GCSS-AF (BLSM-II) program included several pages that defined the Statement of Objectives (SOO) with the contractor required to provide a Statement of Work (SOW) with their proposal submission. The GCSS-AF (BLSM-II) RFP also included as part of the proposal evaluation an eight hour contractor demonstration of the proposed Management and Technical approach. The IMDS procurement confronted by ESD is another example of a two-stage down select which incorporated a demonstrated capability at the end of phase II. The DEIS contracts have emphasized the DII infrastructure.

N. We are not aware of any similar contracts.

O. The acquisition with which we are familiar that most closely resembles CESAR is the AF-GCSS procurement being managed by the Air Force Standard Systems Center at Gunter AFB, AL. It has used a Lightening Bolt approach with a TRD and SOO vice a SOW, is acquiring a complex Information Systems Infrastructure/Common Operating Environment compliant with the DII, and is modernizing host legacy systems and migrating them to an open systems architecture. It did not, however use a two stage downselect approach.

P. The closest program in objectives and scope is the TBMCS program awarded last year by ESC. However, only limited use of the Lightning Bolt initiatives were used and it was not a two stage source selection. GCSS and IMDS have many of the same elements and employed the Lightning Bolt initiatives.

Q. No input

18. What type of contract would you recommend (fixed price, cost reimbursement, incentive, award fee, other), and why? What parts of the CESAR Contract would you recommend be incentivized, and what criteria would be used for the incentive? How would you structure a good incentive plan/award fee? What would be examples of poor incentive plans/award fees?

A. CPAF. Involves architecture development and systems integration--both involve risk and uncertainty. Make the award fee criteria at least 50-70% quantitative based, e.g., 14 of 19 products delivered at least one week early, or, Fixed Plus Cost Reimbursement.

B. Recommend CPAF. We have shot at high fee without the risk of FFP.

C. If requirements are well defined, a fixed price contract will provide the services needed at the best possible price. If requirements are not well defined, a cost plus contract will ensure that the contractor thoroughly identifies and analyzes potential requirements in preparation for system integration. We recommend a contract based on cost-plus for the requirements definition work and fixed fee for development and implementation work. We also recommend an incentive program based on a set-aside pool of funds and preferential selection among multiple award contractors. First, each contractor/subcontractor will receive a monetary award from the award pool for on-time, in-budget performance on each task order. These funds will be provided with the understanding that award fees will flow down to the lowest level of practitioners of the task order - incentivizing excellence at individual levels. In addition, contractors who consistently perform within time and budget constraints will be preferred for additional work over contractors that do not perform within the required constraints.

D. Cost Plus Fixed Fee (CPFF). You need a contract like this to be focused on delivery of the product.

E. A mixed pricing type contract would seem appropriate due to the substantial risk of performance being placed on the contractor. For the development phase, a cost-plus-fixed-fee would appropriately share the development risk between the Government and the contractor. For the fly-off period, a fixed price type arrangement will constrain the cost to the Government. For the production options, a firm fixed price is appropriate as contract risk is then at a reasonable level. Incentives for ahead-of-schedule delivery would also be beneficial.

F. The core contract should be CPAF. Task orders/add-ons could be any other, such as CPAF, incentive, FFP etc.

G. This vendor supports a fixed price contract for COTS products. The terms and conditions for COTS products are as stated in FAR 52.212-4, Contract Terms and Conditions - Commercial items.

H. For the "fly off" scenario, there are two phases of the contract effort. The requirements gathering phase and the operation or delivery phase. We recommend the requirements gathering period be cost plus and the total or partial assumption of the Information Technology service be fixed price. Award Fee criteria should be based upon the vendors ability to synthesize a system that is within the allowable development budget and the estimated operational costs are within budget. This will however require tradeoffs of needs, desires, and technology available.

I. Recommend a task order type contract with a Basic, multi-year (7-10 years) effort with CPFF, T&M, Incentive/Award Fee CLINs. There should be a 2-3 year performance period after the end of the Basic contract period. This would accommodate tasking that was put on contract near the end of the Basic period. Incentive should be used for time-critical deliverables or when contractor cost, performance and schedule are critical. Our experience with an Award Fee Plan has been positive. Much of that plan could be directly tailored for application to CESAR. We can provide a copy of the plan, if desired. The RFP should request a vision for completing the Objectives of the CESAR SOO, a technical and management plan for accomplishing those objectives, substantiation that the proposing company/team can accomplish the objectives, and an identification of risks and plan to mitigate the risks. Task scope, schedule, and a ROM cost (using estimated hours and contract bid rates) for delivery orders would be worked out with the contractor in advance of formal start of the delivery order.

J. We recommend that the CESAR contract be a hybrid with integration services being firm-fixed price and hardware/software procurements being cost plus fixed fee. We believe that this type contract could be further incentivized by establishing criteria such as meeting or beating budget or cost projections, schedules, and S/SDB goals, etc.

K. The Fly-Off portion of the contract should be Firm Fixed Price. During the implementation phase all of the engineering and associated support and documentation services should be on a Cost Plus Award Fee basis. Certain tasks such as acquisition support, V&V or T&E support to external applications programs should be procured on an ID/IQ CP/AF basis as required. With respect to the hardware and COTS/NDI software, the government could buy such items through existing omnibus vehicles. However, the government should also consider allowing the CESAR contractor to offer such items and associated maintenance on a fixed price lease with option to buy basis. Firm Fixed Price rates for these items could be offered for the life of the contract. Rates for items required for system growth and/or technology refreshment would be included. The almost universal attributes that result in a poor incentive plan include: criteria which focus on delivery of labor hours rather an end products, and fee structures which are too low to impact performance and motivation. Good incentive plans are based on cost, schedule and technical performance with heavy weight on factors most important to USSTRATCOM. We favor putting a majority of the fee at risk, that is, a very low base fee with a large award fee contingent upon meeting stated customer expectations. Service interruption credits penalizing the contractor for poor maintenance or system availability could also be included to motivate the contractor to perform.

L. Cost plus fixed fee/award fee (CPFF/AF) places the risk on the Government to know what they are doing, but has more flexibility, and room for change in requirements. Cost plus award fee has an advantage in assuring the Government will have the best support over a long term (7-10 years) especially with the exploding salaries being paid to Information Technology (IT) experts. If the CESAR prime contractor gets caught in an upward wage spiral over the next 10 years, the quality of support will drop because the best people cannot be afforded at yesteryear's wages. Upward spiraling wages can be passed along to the customer (Government) and the quality of support can be maintained without the contractor going broke trying to provide the quality, or providing the Government whatever quality the contract can support. We feel that incentives for the large prime contractors to retain Small Business companies throughout the life of the contract should be included. Parts of the CESAR to incentivize include: Contract Performance, Contractor Team/Personnel stability, Retaining Small Businesses. Criteria should include: Technical Performance, e.g. deliverables, milestones; Management, e.g., budget. An incentive/award fee plan should be structured to provide Government assessment resulting from quarterly PMR coupled with contractor provided data describing noteworthy achievements that merit an award fee.

M. It would be in the best interest of both the Government and the contractor if the primary contract vehicle were an ID/IQ contract that permitted fixed price, cost reimbursement, and other types of Task/Delivery Orders. In this manner, the appropriate contract type could be selected to match the nature of the work being performed and the surrounding circumstances. This gives the Contracting Officer maximum flexibility which, if properly applied, is effective in incentivizing the contractor to meet program objectives effectively and efficiently. Incentives that encourage S/W development vs. procurement of COTS should be avoided.

N. No input

O. We would expect to see a flexible vehicle with a basic CLIN for the Architectural design and COE development/sustainment and an ID/IQ component consisting of a variety of CLINS for different structures (e.g. CPAF, CPFF, FFP, LOE, T&M, Hardware Procurement, ODCs, etc.) to cover future requirements.

P. We would recommend a cost plus incentive fee/award fee (CPIF/AF) contract for the development and integration effort with cost reimbursable CLINs for the COTS products. Because of the development/integration nature of this effort, a cost plus contract allows the government greater flexibility. The contractor could be incentivized on the integration portion of the contract while the award fee could apply to specific technical and management milestones. A successful incentive plan/award fee would have specific established schedule milestones and performance criteria while a poor one would not include specific targets.

Q. We would recommend that CESAR be awarded as a Cost-Plus-Award Fee. The maximum fees payable under this contract would not exceed 10 percent. The contract should include a 2 percent incentive for contractors who meet the Small Business goals of their Subcontracting Plan. (The Subcontracting Plan must include Small Businesses as well as SDB and Women-owned businesses.) This 2 percent incentive would be considered a part of the maximum fees under the contract. The 2 percent incentive plan would not apply when the subcontracting plan submitted is a plant, or commercial products (hardware/software) products plan. The combined Base Fee, Award Fee and 2 percent incentive shall not exceed the maximum fee limitation of 10 percent. Make award based on other than low cost--consider value.

19. We are considering a seven year contract. Please comment.

A. Recommend a three year base contract with two, two year options. Also recommend that the Government be required to notify the incumbent of their intention to exercise an option at least one year prior to the scheduled date of the option. The longer the base period and options, the better and more efficient the contractor teams planning and performance.

B. The length seems a minimum amount of time for least disruption to operations for a program of this magnitude and criticality.

C. A long term contract exacerbates program risks discussed in our response to Question 1. In addition, a long term contract poses a risk of diminishing input of new data and fresh ideas. We recommend a short term contract (one basic year, two to four option years) to prevent complacency and mitigate program risks.

D. Seven years is a good choice.

E. A seven year contract will achieve intense competition but will also increase the risk for protest. Unsuccessful offerors will lose a large part of the previously segmented STRATCOM business and may not have a broad enough business base to remain economically feasible. This may decrease long term competition but could provide significant economies of scale for the successful fly-off offerors to pass on to the Government. A diverse teaming arrangement composed of required small business involvement may help mitigate the risk of protest.

F. Based on the anticipated investment by industry and the Government, a ten (10) year program would be best.

G. No input.

H. Seven years is reasonable. We would not recommend anything shorter. Some similar types of contracts are for ten years of operation.

I. Given the CESAR vision, the number of programs involved and the current state of those programs, it is recommended that a minimum of seven years be considered. Understanding that the mission of USSTRATCOM is about to take on a new significant role in Joint Operations and that CESAR will probably be the contract of choice to implement CESAR capabilities into that evolving mission, we recommend a seven year contract with at least 3 one year options. A 7 year contract with 3 one year options is appropriate for an effort of this magnitude if a Basic+Options+Delivery Order contract is implemented. We would also suggest consideration for an extended three year performance period after the Basic Contract period expires that could be used to manage task orders that might come at the end of the Basic Contract performance period. This would add flexibility to the Government's management options.

J. Seven years is sufficient. Regardless of the length of the contract, there must be provisions to allow for changes in the hardware and software products and configurations over the period of the contract. The rate of technological changes is accelerating. Without provisions for changes and or upgrades, equipment on contract will be outdated very quickly, in some cases less than one year from contract award.

K. Although we believe a ten year contract is optimal, seven years is a reasonable period of time to complete this complex system integration, migration program. At the completion of the contract period, it appears likely that transition to a lower cost life cycle support contract would be appropriate. We recommend that planning and executing that transition be a deliverable under the CESAR contract.

L. The longer a contract can run without being recompeted, the greater the cost savings to the Government and the contractors. On the other hand, it is difficult to write a contract that will serve the customer properly over a period of 7-10 years, considering the rapid expansion of Information Technology.

M. Our feeling is that the period of performance of the contract should be sufficiently long so that the effort is not disrupted by renegotiation or recompetition. These activities result in lost productivity on the part of the Government and contractor. A contract period of seven years will enhance continuity of effort. However, a program of subcontractor/vendor refreshments may be needed to keep pace with Technological advance.

N. No input

O. Long term contracts carry with them both advantages and disadvantages. A seven year effort can provide term stability, and sufficient time to effect migration under single vehicle; however, it may not be long enough to sustain ordering and COTS refresh. Because of that, some agencies have moved toward contracts with CLINS that are effective for as long as 15 years. The use of option years and multiple long term awards can mitigate the risk of being locked into an unsatisfactory long term relationship that is difficult to sever.

P. A seven year contract is acceptable.

Q. No input

20. How would you recommend the Schedule, Section B, be structured? For example: CLIN to requirement, (SOO-IMP-SOW) traceability.

A. Contract Line Item Number (CLIN) to requirement.

B. That seems reasonable.

C. The SOW, written by the contractor, will vary with each proposal and cannot provide the consistency required for evaluation of cost information. The SOO is usually too general to provide the granularity needed to define cost fields. We recommend that the Government level the playing field by structuring Section B by labor category, providing specific labor hours for each base and option year of the contract or by using a costing model.

D. No comment on this area.

E. No input

F. CLIN structure, yearly and options.

G. Schedule B should be structured based on vendor manufacturing part numbers instead of CLINs. By using a CLINs structure, the government can only receive the benefit of new technology if the technology replaces an existing CLIN. A contractor part/model numbers approach (instead of CLINS) means the Government can obtain the benefit of new design breakthroughs, technological updates, or advances in equipment within a short period of time after it becomes commercially available.

H. The requirement portion of the contract could be a CLIN structure to organize the costs, versus the activities, but the total or partial assumption of Information Technology services stage of the contract would have the performance levels as the requirements and the price of the entire operation would be fixed and therefore just one CLIN for simplicity. Required system alterations could also be handled on a CLIN basis.

I. Consistent with Questions 18 and 19, Schedule B should allow for the Basic Contract period, Delivery Orders, and one-year Options. It should be CLIN-to-Delivery Order-requirement.

J. Map the CLIN items to the requirement. A statement of objectives (SOO) under the Lightening Bolt initiatives leaves interpretation of the requirement and subsequent CLINs to the contractor further indicating that the CLINs must be mapped to the requirements as outlined in the SOO. Also, without having a predefined set of requirements, how would the Government evaluate one proposal from another to determine the Best Value?

K. We have chosen to respond to questions 20, 21 and 23 together. Recently, ESC at Hanscom AFB has begun to develop RFPs jointly with industry. The technique was used effectively with the CCPL effort and is now underway on the RPS solicitation. It appears that the size and complexity of the CESAR effort would benefit from early collaboration between USSTRATCOM and interested bidders. We recommend that a three to four day conference be scheduled in Omaha as soon as possible after requirements are relatively firm, possibly in the December, January time frame.

L. None.

M. We suggest a review Section B of the DEIS II contract (DCA100-96-D-0051) recently awarded by DISA.

N. No input

O. A Schedule B that provides traceability of CLINs to Requirements will provide a good framework for managing and accounting for the effort while ensuring that the vehicle has the scope and flexibility required to handle all requirements.

P. There should be a one-for-one mapping of the SOW paragraphs to the Section B CLINs.

Q. No input

21. What Special Contract Requirements clauses would you recommend? (Technology Enhancement, Economic Price Adjustment, Award Fee, Incentives, etc.) Furnish samples, if available.

A. Recommend Technology Enhancement and Value Engineering Change Proposal (VECP) clauses. Furthermore, recommend award fee plan which targets Technology and VECP activity. Allow economic price adjustment for all firm-fixed priced non labor CLINs.

B. If lengthy contract (7 years) recommend EPA clause. Plus, for Award Fee, recommend roll-over of unearned fee to subsequent periods, and anticipatory billing of award fee.

C. We recommend a technology enhancement program that will (1) allow the contractor to monitor and report on viable new technology and (2) provide a mechanism to ensure refreshment to the development environment technology. We also recommend an Award Fee and Incentive program as addressed in our response to question 19.

D. An Award Fee will give you a tool with which to incentivize the contractor to make your priorities his priorities. It is dynamic, allowing for changing priorities or requirements.

E. No input

F. Economic Price Adjustment (EPA) especially if it becomes a 7-10 year program.

G. The government should reserve the right to require the contractor offer hardware and software components available to the contractor's commercial customers under this contract. In this way, the government seeks to ensure that it can obtain the benefits of new design breakthroughs, technological updates, or advances for equipment currently on the contract. When requested, the contractor shall provide a refreshment proposal including the components as indicated above. In the event that the contractor is no longer able to provided the products proposed (because they are no longer being manufactured, for example), the contractors may, with the government's approval, remove the products from the contract. Any new technology shall be considered if the contractor submits a proposal outlining the new technology. Included in this proposal shall be pricing data (i.e., current published commercial price list/GSA price) and other technical information as listed below. With the receipt of a proposal from the contract, or the Government shall have the right to approve this proposal and to unilaterally modify the contract to provided for ordering of the new technology. The criteria for acceptance of the new technology proposal is as follows: List price pages, TR write describing the proposed technology, Contractor part/model number, Contractor description, GSA/Commercial catalog unit price, Delivery period.

H. Technology upgrades and economic adjustments are part of our normal recommendations. We will have additional recommendations as we more clearly understand the detailed needs of the project.

I. Refer to the answer to question #18.

J. The CESAR contract will need some form of Technology Enhancement Clause and an Economic Price Adjustment for labor related items. One usually thinks of incentives as motivational in nature, however, if improperly applied or used in a threatening way without specific measurable objectives, they can result in just the opposite result. We recommend that incentives be specifically tied to measurable goals which are important to the government for overall program success (Note: Several examples of Technology Enhancement Clauses will be sent under separate cover.)

K. Refer to the answer to question #20.

L. Refer to the answer to question #18.

M. We suggest a review Section H of the DEIS II contract (DCA100-96-D-0051) recently awarded by DISA.

N. No input

O. Technology refreshment should be a major consideration in a broad scope, long term effort like CESAR. Because of the varied nature of the requirements, a diverse CLIN structure is called for. Incentive in the form of award fees can be advantageous if properly structured, however, some types of work are best suited for FFP or other types of contracts, and they should be available in the contract structure to allow the Government the maximum flexibility without having to resort to contract modifications.

P. Special Contract Requirements clauses recommended are Award Fee and Special Award Fee (to incentive innovative ideas).

Q. No input

22. Are you aware of any DOD mission-critical, TOP SECRET SIOP/ESI or SCI computer shops that are contracted out? If so, how were the following security issues resolved that limit contractors:

A. Contact DISA on the DEIS Contract or the Software Technical Center at Hill AFB UT. However, if the contractor(s) will become an integral part of the USSTRATCOM team, then restrictions listed should not be a concern. Contractor personnel will be cleared and should have access to areas necessary to fulfill the contract.

B. We are not aware of any DOD classified computer shops that are outsourced, however the CIA outsources it computer operations.

C. Our personnel support and have supported a variety of SCI missions. In our experience, the contractors' personnel were provided privileges similar to active duty military personnel.

D. Can not comment in this document.

E. Not sure that "contracted out" is the correct terminology for this example, but we know that Air Force Studies and Analysis works closely with a contractor on a large, nuclear connectivity model called STRATCAM. Not aware of how they handle security issues, but below (see a. through d.) are some independent ideas.

F. Escorted access is generally provided by the Government or authorized contractor representative for non-cleared persons needing temporary access to a secured areas. Generally, all employees needing routine, continuing assess to a secure area are issued the appropriate clearances for unescorted access. Temporary access to "root" software processes and super-user privileges have been granted to us during installations and testing on the DMSP program.

G. No Input.

H. No.

I. Yes, however, due to the sensitive nature of the programs we cannot discuss in this paper. We are involved in special access programs

J. We are aware of support contracts by other integration contractors where government support is provided for Top Secret facilities at the Defense Intelligence Agency and the Central Intelligence Agency. No, these are not TS/SIOP. They are TS/SCI! We know of no reason why contractors could not be performing any and all duties related to security items. Contractors are subject to identical security background checks and investigations as are the Government civilian and military. Although not direct employees of the government, the contract employee is subject to the same criminal punishments for unauthorized disclosure of classified material.

K. The security issues above have been addressed on several occasions, however we are precluded from releasing the specifics.

L. No.

M. We are not aware of any instance where this has been done, however access to production environments and unescorted access to restricted areas is not uncommon, and we know of no reason why super-user or root privileges could not be granted in a sensitive environment. Proprietary issues can be resolved by non-disclosure agreements between the contractors involved.

N. No input

O. We are not aware of any efforts in the DOD mission critical TS SIOP/ESI environment that have been completely outsourced. Access to production environments, restricted areas and super-user privileges, however, are commonplace and appropriate safeguards can be implemented to ensure that they are controlled and not abused. Proprietary information issues can be easily dealt with through associate contractor and non-disclosure agreements.

P. Not aware of any similar contracted activities.

Q. No input

a. Access to production environments.

A. See Above

B. No Input

C. Our personnel were provided a user account in the production environment based on job description.

D. See Above

E. Use COTS/GOTS to the maximum extent possible. If use of COTS/GOTS software is not possible, separate software development location and customer operations areas. Have customer visit software production facility versus contractor visiting customer or meet on neutral ground.

F. See Above

G. No Input.

H. See Above

I. See Above

J. See Above

K. See Above

L. See Above

M. See Above

N. See Above

O. See Above

P. See Above

Q. See Above

b. Unescorted access to restricted areas.

A. See Above

B. No Input

C. We provided cleared personnel who were granted unescorted access to restricted areas based on individual "need to know".

D. See Above

E. Restrict unescorted access to a small number of contract personnel. Allow unescorted access only during normal duty hours. Prohibit unescorted restricted area access.

F. See Above

G. No Input.

H. See Above

I. See Above

J. See Above

K. See Above

L. See Above

M. See Above

N. See Above

O. See Above

P. See Above

Q. See Above

c. From having super-user or root privileges.

A. See Above

B. Each of the above issues were resolved by issuing "staff like" clearances to contractor personnel.

C. Our personnel were provided super-user or root privileges based on job description.

D. See Above

E. Restrict access on a project-by-project basis. This can be done through use of password protection that only works on certain files/databases.

F. See Above

G. No Input.

H. See Above

I. See Above

J. See Above

K. See Above

L. See Above

M. See Above

N. See Above

O. See Above

P. See Above

Q. See Above

d. Providing network support for WAN's in other government contractor's facilities (proprietary information issues).

A. See Above

B. Resolved through the use of an isolated computer, air guards and firewalls. Nothing allowed outside of controlled data.

C. Our personnel were provided a user account on the Government network based on job description. Each user or user group received only those accesses required to perform network support. Personnel did not have access to proprietary information.

D. See Above

E. Require legally-binding non-disclosure statements between contractors.

F. See Above

G. No Input.

H. See Above

I. See Above

J. See Above

K. See Above

L. See Above

M. See Above

N. See Above

O. See Above

P. See Above

Q. See Above

23. When should an Industry Day be scheduled and what should be on the agenda?

A. The proposal, as we understand it, is to have another Industry Day in September. Recommend a robust agenda, with the results of this questionnaire to be briefed in detail along with the Government's opinions/decisions on each of these questions. For planning purposes, schedule an Industry day at least quarterly until RFP release (Jan 97, Apr 97, Jul 97).

B. The Industry Day should be scheduled after the release of the DRFP and at least two weeks prior to release of the formal RFP. A suggested agenda includes: Program Overview, Program Changes, Responses to DRFP questions/comments, Acquisition Strategy, and Round Table Discussion. The last part could be done in a format similar to what Mr. Gilligan set up for TBMCS, where computers were utilized to allow anonymous comments.

C. The Government should schedule frequent industry days. The agendas should include reviews of the Government's requirements, discussions of the Government's acquisition strategy and award schedule, and question and answer periods.

D. Prior to finalizing the Acquisition Strategy.

E. The use of a frequently updated and robust bulletin board will alleviate the need for an early Industry Day. This meeting should be scheduled when the CESAR requirements are reasonably well defined, the acquisition strategy has been approved, and the Draft RFP is ready to be released for comment. At least 45 days should be allowed for the comment period on the Draft RFP prior to the scheduled release of the final RFP.

F. Government should probably have an Industry Day following the compilation of the results of this questionnaire, followed by another approximately one month after release of a draft RFP.

G. No Input.

H. Scheduled after you further define the acquisition.

I. We would suggest that as soon as the Government settles on the type of contract and the contract content and scope that a conference be held to clearly define the competition to the potential bidders. Additional information concerning the four domains and their related systems and programs isn't considered necessary. The focus should be on the contract and the bidding process.

J. We recommend that another Industry Day be scheduled as soon as information becomes available which might cause a change in the government's timelines, intentions, requirements, etc.

K. Refer to the answer to question #20.

L. If Industry Day can be adjusted to accommodate CESAR, the best schedule would have Industry Day 30-40 days after a draft RFP has been out and companies have responded back with comments. Then the Government could address contractor's comments on the draft RFP.

M. Schedule an Industry Day after CESAR's scope has been determined but before the draft RFP is released. This will allow for industry/government dialogue that will improve the draft RFP.

N. No input

O. Another Industry day would be appropriate soon after the C4I study described at the 19 June briefing is completed. This meeting should focus on providing the results of the study to the potential CESAR offerors.

P. Agenda items should include: Detailed discussion of security issues, anticipated scope of effort, if you go with a fly-off approach a discussion of what you expect to accomplish in the fly-off, procedures for future interchanges, status of library. Industry day should be scheduled as soon as meaningful information is available.

Q. No input

24. What types of information would you want to see in an electronic bidders library? How could we make documents and information more accessible?

A. The temptation will be to provide a large quantity of information due to the convenience and capabilities of the Internet. The danger with a large body of information such as this is that some of it will be inaccurate. Recommend a technical review team be appointed to review, edit, and approve all information before it is released for distribution to the electronic library.

B. Deliverables from the current contract, ACA's from the current contract, vendor lists, formal meeting notes, and current procurement documents. An FTP site would help tremendously as most, if not all, the bidders are online.

C. We would like to see the following information: Any documentation that describes validated user and system requirements (such as mission need statements and objective architectures). Any available technical documentation on USSTRATCOM C4I systems (such as run-time interface documents). The latest version of RFP as well as amendments and answers to RFP-related questions.

D. Draft technical documents, as soon as possible, even though they are changed an/or refined in the future.

E. The development bulletin board or a robust homepage on the INTERNET will provide access to all interested parties. Password protection could also be used if the resources are available to issue and monitor the passwords. Hot key on the homepage could send the user over the INTERNET to the source of the documentation in many cases, reducing the need for massive availability of data on the web site. Available information could include: current organizational structure, a detailed inventory of information systems including hardware, software, and communications components, process and data models, etc. Since there are a larger number of potential bidders in the Washington D.C. area, a separate bidders library should be established in this area.

F. All procurement related documents, including all referenced documents. Current specifications and documents describing the current systems at STRATCOM.

G. No Input.

H. All of the technical data describing the current system and the target system.

I. We would like to have available basic descriptions of the existing targeted programs (the AS-IS Architectures) to include information concerning the current supporting contractors and information specific to their contract vehicles such as type, value and schedule. We would also like to be able to access a list of contractor companies that are interested in CESAR in order to both facilitate our teaming arrangements and competitive analysis. This information usually leads to very competitive proposals that are in the best interest of the Government. In support of that, we welcome the opportunity to have all interested companies apprised of our interest in this program along with our points of contact.

J. A large percentage of the required documentation supporting SWPS is either not Government owned or not in an electronic format. Unless the Government purchases the many electronic documents from its application and database vendors, these documents will not be available except in paper (hardcopy) format. The Government has not traditionally required documents to be available electronically in its CDRLs from most of its contracts. However, we support the concept. It is important to provide a comprehensive electronic library and this concept should be evaluated for cost feasibility.

K. The library should contain at least: the information described in the answer to question 15, the RFP and DRFP, frequently asked questions, mission area plans and appropriate general references.

L. The electronic bidders library, as a minimum, should contain current information about the RFP, schedule, Questions and Answers, bidders list, and unclassified tech library documents.

M. We would like to see all (or as much as possible) unclassified information posted electronically. We would also like all classified information made available at more than one locations i.e. Offutt AFB. This will provide remotely located contractors easier access to information required during the pre-bidding and proposal development process.

N. No Input

O. Draft solicitation materials, applicable CONOPS, system documentation, vision statements, system descriptions, and functional descriptions should be made available on-line through a bulletin board or World Wide Web page, to the extent permitted by security considerations. Classified or sensitive documents should be made available through a consolidated library.

P. Information that describes the functional tasks that must be provided under this contract. It is important that bidders understand the current processes involved so that a viable transition plan can be developed. Descriptions of the current contracts, that normally would be acquired under the freedom of information act, associated with the applications programs and the other efforts currently under contract that will be included in CESAR or will require an associate contractor relationship. Programmatic information such as funding profiles, key milestone delivery dates, and priority requirements would permit the contractor to optimize a delivery schedule to meet operational requirements. MNS and ORD documents will help the contractor understand the operational objectives and enable us to be more responsive in our proposals.

Q. No. Input

25. Are there any other areas that we should seek your comments on to insure success of this endeavor?

A. The scope does not appear to include a concept or mission-oriented role for the contractor. It would appear that the function(s) for the contractor include management, development, implementation, and production. Where is the concept function for meeting mission objectives? Who has the breath of knowledge to integrate warfighter requirements into the overall architecture of the CESAR? Who understands WWMCCS, NPES, GCCS, SWPS, national policy guidance and the target structure, along with command and control, weapon systems, crew training, and exercise administration/evaluation? Who can call on the resources, experience, and research capabilities to tie together a seamless strategic planning and warfighting system? Recommend that a concept role be established for the overall contractor and/or team member (under USSTRATCOM guidance) to handle conceptual/architectural considerations in the design and fielding of the CESAR. Direct research, development, and coordination of the overall concept of CESAR integration/communication/ operating functions will directly benefit in the final, end product. Did not see a great deal of concern for the design, development and building in of information warfare defensive and offensive functions, given the current and foreseeable wave, maybe we ought to start paying more attention now instead of possibly being forced to retrofit tomorrow. Define organizational and software goals and directions. Form an integrated reengineering team. Define a Capability Maturity Model. Select a standard set of software metrics. Analyze the legacy software. Define a reengineering implementation process. Develop or update a standard testbed or validation suite.

B. This questionnaire is an excellent idea that will, hopefully, be responded to and then repeated before the draft RFP is released. At that time we should have more comments.

C. No input

D. One additional area where you might find good return is in resource allocation decision making. That is, require the contractor to provide expertise in the automation of this decision making process, to assist in the allocation of scarce resources across broad requirements, or to select the most effective solution among various alternatives.

E. Since the scope is so broad, in addition to the requirements analysis envisioned as part of the contractor tasks, recommend a business process reengineering analysis as a first step, at least as a pilot project to explore efficiencies. Process reengineering is critical and should be done before architecture design. Without a thorough analysis of the goals and objectives of the organization and of the existing organizational and operational environment (the "what you want to do" aspect), before analyzing the "how to do" aspect, there is a risk of wasting money. This preliminary analysis should focus on identification of redundant, no-value added, and/or cost ineffective processes, information requirements, and performance standards which must be met in the future environment. This analysis will also provide the opportunity to identify and resolve interoperability issues, define future standards, and identify potential process improvements to be incorporated into the future architecture. Using SWPS for example, without such an analysis this large effort will only upgrade the same modules which have existed for over 30 years. Surely the significant changes to the strategic environment, such as the need for more rapid SIOP planning, mobile ICBMs, rapid retargeting of emerging targets, and integration of theater planning should have impacted the planning process by introducing many inefficiencies. Evaluation of potential improvements should include thorough analyses of executable options and determine the relative economic value of suggested improvements to ensure decision-making is based on the best business base. This will focus efforts on executable options and avoid unexpected hidden costs during implementation. Following evaluation of alternative solutions, thorough planning is required to define dependencies among reengineering initiatives, establish priorities for implementation, and provide the foundation form implementation scheduling and budgeting.

F. No.

G. This vendor suggests you look at successful migration efforts. We would welcome your visit to our facilities to demonstrate how we migrated off mainframe systems to a pure client-server environment. This would provide you with an opportunity evaluate how we did it and how we assessed critical success factors for such a move.

H. Are you currently considering a Service Level Agreement, Outsourcing type of contract? We would be happy to review this type of operation with you and show you how your mission would be better served by a complete Outsourcing arrangement.

I. No Input

J. None

K. CESAR may present unusual problems if SIOP and/or SCI material is required to develop the proposal. First, it will put a significant burden on the contractors to produce and deliver such a proposal. Second, it would place a significant burden on the government evaluation team. We believe that a meaningful unclassified proposal can be prepared for evaluation purposes.

L. A concern of the Government should be the cycle time of computer technology vs. the cycle time of a contract. The cycle time for a generation of computers is months while the cycle time for a contract is years. A way is needed to integrate flexibility into contracting that will allow the contract to evolve with changing technology.

M. No, your questionnaire was very comprehensive and we are grateful for the opportunity to respond.

N. No input

O. A continued open dialogue between Government and Industry is the best way to ensure that CESAR requirements are understood, and ultimately satisfied.

P. I'm sure there are and they will emerge as more information becomes available on the scope of CESAR and as the acquisition strategy evolves.

NOTE: Responses to this questionnaire were due NLT 1 July 96 to 55 CONS/LGCXS, 901 SAC Blvd STE 1E16, Offutt AFB, Ne. 68113-5640. Due to time limitations, specific industry responses were not solicited during the 19 Jun 96 session. During the discussion period, the Government entertained inquiries pertaining to clarification of the questions. Clarification could also have been requested prior to 19 Jun through the 55 CONS/LGCXS, 901 SAC Blvd STE 1E16, Offutt AFB, Ne. 68113-5640.