The Architecture Design and Evaluation Process
The Architecture Design and Evaluation Process
The Aerospace Corporation conducts architecture trade studies to assess options and solutions to meet its customers’ space system program requirements.
First published May 2013, Crosslink® magazine
In the early stages of developing a new space system program, U.S. government decision makers need to be assured that they are making the best acquisition choices, while also handling uncertainties such as cost, schedule, technology, and integration risks. Architecture trade studies are performed during this front-end, formative stage of a program. By examining the large trade space of alternative solutions and improving the understanding of the myriad of available technical and program options, decision makers can then use these study results to determine how to allocate their limited budgets and identify the options that are most likely to succeed. The Aerospace Corporation regularly conducts these studies for its customers, who use the findings to better understand the options, benefits, costs, and risks of the alternatives.
Aerospace has been conducting concept and architecture studies since its earliest days, and the general principles remain the same. The steps involved in this process facilitate the transfer of knowledge from long-time employees to the newest generation of engineers and scientists, and ensures consistency and access to study team members, tools, and techniques, so that they can be applied in a repeatable and successful manner across the corporation.
This decision support framework establishes a traceable structured process for making defensible decisions and recommendations. Aerospace’s technical expertise spans the gamut of disciplines and perspectives. Trade studies typically include Aerospace experts in system requirements, conceptual design, technology readiness assessment, system performance analysis, utility analysis, and acquisition planning. The principal job of the architecture trade study lead is to elicit and coordinate the findings of these experts.
The Architecture Study Process
Architecture studies can be performed at the earliest stages of or prior to a program’s formal inception. In some cases, they can be performed before a program’s requirements are specified and validated. After all, requirements should not be dictated in a vacuum, but rather should be based on what is feasible and consistent with the available budget and other constraints. Architecture studies provide a better understanding of the relationships among requirements, cost, and capabilities, thereby facilitating a more informed decision regarding the best trade-offs among these factors.
Once a program is given the authority to proceed into the materiel solution analysis phase, a formal analysis of alternatives is conducted. This is a general systematic approach to developing and comparing alternatives and can be applied to any trade or concept exploration study. The “Analysis of Alternatives Handbook,” published in July 2008 by the Air Force’s Office of Aerospace Studies, outlines the recommended process. A prestudy is often performed to refine the trade space of alternatives, flesh out the analysis process, prepare the study team members, and establish roles and responsibilities. Aerospace’s architecture trade study process is compatible with the guidelines in this handbook, and adds some of the details of how these studies are implemented at Aerospace.
Experienced study leads understand that each architecture trade study differs due to the various constraints of individual customers, time, resources, and other stakeholder interests. Therefore, the methodology outlined here serves as a guide for best practices that can be tailored to specific situations. Other uses of the architecture study process include assisting with the concept characterization and technical design, developmental planning, or preanalysis of alternatives, which are also performed at the front end of an Air Force, National Reconnaissance Office, or civil program formulation.
Aerospace has a six-step architecture design and evaluation process, which is the foundation for conducting an architecture study. These steps include problem definition, implementation and operational context, alternatives, system definition, analysis and evaluation, and integration and summarization. The steps do not need to be performed consecutively, nor does each need to be completed before the next one begins, but none should be skipped. The study team’s understanding of the problems and trade space will mature as the study progresses, so iteration among the steps is imperative to take advantage of the lessons learned along the way, revisit assumptions, and refine analyses as additional information is attained. System architecting is as much art as it is science, requiring creativity and intellectual agility.
The problem definition phase involves defining the architecture trade study’s main parameters and developing a step-by-step plan for executing the study. At this early stage, frequent and effective communication with the customer is critical to ensure that the objectives of both the overall architecture and the study are established, and that the study scope, ground rules, assumptions, and constraints are identified. These factors must be documented in the study’s terms of reference or statement of work, and the major stakeholders who are commissioning the study must concur with them.
Another key factor to establish early in the architecture trade study is the set of key decision metrics, criteria, or measures of effectiveness that will be used to compare the alternatives. These include measures of system performance, cost, schedule, and risk, as well as operational resilience or compatibility with legacy systems. The deliberation leading to the selection of the decision metrics should include identifying stakeholder interests as well as the analysis, models, and methodology that will be used to assess the quantitative and qualitative factors determining the architecture alternatives. This process ensures a discussion about the level of depth for the study. The ideal level of depth of the study includes sufficiently detailed analysis to provide the architecting team with the insight to discriminate between the options under investigation.
The architecture trade study’s definition and planning also involve negotiating the study delivery schedule, funding level, roles and responsibilities, points of contact, rules of engagement, and delivery product. Concurrent with these negotiations, key team members should be recruited and consulted for engagement in the study planning activities, and their commitment for performing the actual study downstream should be secured.
Implementation and Operational Context
An important early step in the architecture design and evaluation process is to understand the context in which the system is intended to operate. This process should identify the end users, policy drivers, other stakeholders, technology availability, use cases, scenarios, and concept of operations. Understanding the functional capabilities and mission needs that the objective architecture is intended to meet is critical. For the Air Force, such formal system requirements are often documented in the DOD Joint Requirements Oversight Council (JROC) approved initial capability document (ICD), if the program has matured to that point in the
defense acquisition lifecycle. However, some architecture trade studies are performed prior to this, with the intent of iteratively developing requirements based on what materiel solutions can affordably provide versus nonmateriel solutions.
Since precisely meeting all stated requirements often results in an unaffordable system, understanding the extent to which these factors are flexible or negotiable is important. A good architecture trade study should provide the decision makers with an understanding of the trade-offs between cost and performance across a broad range of potential solutions. Since requirements documents typically do not state cost constraints, a key objective of the architecture study is to provide information about the potential costs—in terms of funding, schedule, and the risks of not meeting these budget and schedule targets—to implement the materiel solutions and bring the capabilities to realization. The architectures considered should be robust to the uncertain future in terms of budget availability, threats, operating environment, and technology.
To mitigate the risk that the architecture trade study will prematurely focus on a limited set of possible solutions representing only minor deviations from the status quo, exploring the widest possible trade space early in the study is important. Early brainstorming efforts are often useful for flushing out innovative solutions. A systematic approach of mapping out the entire solution space is also an effective approach to ensure consideration of all options.
The solution space can be envisioned as a multidimensional volume of possibilities, with each dimension of that space being a tradeable parameter. Each architecture option is a single point within that multidimensional trade volume. The tradeable parameters of interest often include at the system level payload type and technology (e.g., traveling wave tube amplifiers vs. solid-state power amplifiers; gimbal vs. pointing mirror assembly), bus technology (e.g., lithium ion vs. nickel hydride batteries; reaction wheels vs. control moment gyroscopes), and platform (e.g., free flyer vs. hosted vs. small satellite vs. commercial). At the architecture level, the tradeable parameters of interest include constellation, acquisition, and ground systems options. A table is created that captures the range of likely options for each trade space parameter.
Architecture candidates are formulated from combinations of the parameters and are often depicted in a trade tree, with the tree branches representing families of solutions or individual solutions. While the architecture candidates do not have to exhaustively cover every possible solution, they should address the full range of decision criteria or measures of effectiveness that are of interest to the customer, and should ascertain from the multidimensional parameter space the best and worst case solutions. For example, the study team members would want to see what a fully capable solution looks like (although it may be unaffordable) and what an affordable but high-risk or low-performance solution looks like.
Systematically exploring the key trade parameters and candidate architectures early on allows the study leads and customers to better understand the solution space and express their preferences for which areas to filter for more detailed study later. Engaging customers to capture their direction and preferences for the study and iterating the steps of the study in small cycles is better than waiting for the final results to get their feedback and then repeating the study.
A reasonable number of candidate architectures should be selected, covering as much as possible within the range of potential solutions. A good rule of thumb is to assess approximately half a dozen potential solutions, but the list usually grows as iteration of the architecture study continues, more information becomes known, and more stakeholders join the discussion.
Once the full range of architecture options has been pared down, each candidate architecture is defined as much as possible with roughly the same amount of detail, which allows direct comparisons between them. The candidate architectures’ characteristics are documented, identifying their constituent systems and interdependencies. Any known significant risks or issues like the inability to satisfy any requirements or the need for high-risk technology development should also be identified. A work breakdown structure, an operational view diagram (such as an operational concept graphic from the DOD Architecture Framework), and possibly other architectural description diagrams should be generated to communicate the candidate architecture’s content and distinguish these from the other alternatives.
At this stage, conceptual design activities, such as those performed by Aerospace’s Concept Design Center, are conducted to give more substance to the definitions of the architectures and the system solutions for those architectures. System solutions should be defined to a level of detail that facilitates estimating lifecycle costs, which usually means quantifying mass, power, and size for determining the costs of space and launch segments, and ground operations.
The system definition should extend beyond technical concepts at this stage by including the programmatic trade space and the identification of approaches to realizing the envisioned architecture through the acquisition process. For each candidate architecture, acquisition strategies and procurement options, identification of budget and schedule constraints, transition approaches from the current baseline to the new architecture, and program risks should be developed.
Analysis and Evaluation
Next, the candidate architectures should be evaluated in terms of the high-level decision criteria and the detailed performance capability measures that were defined earlier during the problem-definition phase of the study. These include measures of performance, cost, schedule, and risk.
Teams of experts representing each of these areas conduct the evaluations. The system performance team is the most varied since each mission area has a different subject matter expert. For example, depending on whether the current architecture study is related to missile warning, navigation, communication, or intelligence, surveillance, and reconnaissance, different analysis tools and experts are needed. A key function of the architecture study lead is to know who the experts are and what knowledge, experience, and credibility they can bring to the evaluation effort. Analyzing a candidate architecture’s military utility helps to determine the system’s value to warfighters. Other analyses help to determine technical measures of a system’s performance or capability.
Estimating the costs and cost risk of each of the candidate architectures is the next step. This should include nonrecurring development and recurring production costs. The roll-up of costs should be performed across the time period of interest and the near-term future years’ defense program horizon. If possible, operations costs should be included. Similarly, the schedule for implementing the architecture and transition from the legacy system needs to be evaluated.
Integration and Summary
The architecture study’s final phase is conducted to gather the assessments of the various candidates and provide a balanced view of the advantages and disadvantages of each. For example, a comparison might include reviewing candidate architectures for high performance at high cost vs. reduced performance at reduced cost. The main activity during this phase is capturing and summarizing the observations about the principal trade-offs between the study decision criteria and any other meaningful observed trends and insights. These observations should include the results of sensitivity analyses performed throughout the study process to reflect deep uncertainty present in any investigation of the future.
In the final phase, it is critical to communicate the architecture study results clearly and effectively to all stakeholders. While conveying the bottom-line results of the study to the decision makers is important, it is also crucial to provide a clear description of the methodology, rationales, and assumptions that were used, so they will have confidence in the chosen solution. Decision makers need to be assured that the study has been conducted as thoroughly as possible and with a level of analytical rigor and process discipline that is commensurate with the weight of the decision. Providing a means of interactively exploring the solution space via visual tools, thereby giving decision makers the ability to ask and answer what-if questions in real time, is ideal.
For many years, the architecture design and evaluation process has been implemented in several front-end studies at Aerospace. The architecture trade study process can be described in different ways, and its actual practice may vary from study to study, but the fundamental steps of an architecture study described here can be applied to many different types of space system evaluations. Aerospace’s training arm, The Aerospace Institute, offers courses covering much of the material in this article as part of the “Aerospace Systems Architecting and Engineering Certificate Program.”
Inki A. Min
Principal Engineering Specialist, Architecture and Design Subdivision, joined Aerospace in 1987. He leads and performs system- level analyses for various government projects. He has a B.S. in engineering from the University of California, Los Angeles; an M.S. in aeronautics and astronautics from Stanford University; and a Ph.D. in aeronautics from the California Institute of Technology.
Ryan A. Noguchi
Senior Project Leader, Architecture and Design Subdivision, joined Aerospace in 1997 and works on system-of-systems engineering, model-based systems engineering, and architecture studies. He was the Aerospace lead for the Delta IV Heavy upgrade program, as well as other development and acquisition projects. He has a B.S. in mechanical engineering from Princeton University and an M.S. in mechanical engineering from the University of California, Berkeley.
Back to the Spring 2013 Table of Contents