Flex21 — The Intelligent CompactPCI Platform

In the Telecom Industry, Time to Market is Critical

Today, new and emerging Telecom markets are once again driving a fast-paced build-out of converged and wireless networks. End-user demands are growing and diversifying quickly, and telecom equipment manufacturers (TEMs) of all sizes are being challenged to provide new, increasingly complex applications. Given limited resources, tight budgets, and short delivery timeframes, TEMs must find ways to balance the level of their R&D effort with stringent time-to-market and end-user cost requirements in order to satisfy customer expectations. In particular, today's Voice-over-IP and 3G Wireless markets are experiencing intense competitive pressures which dictate that strategies be found to enable quick and cost-effective application development and deployment solutions.

In the final analysis, telecom customers pay for services - software-based applications running on hardware platforms. If we make a distinction between the software providing the service (the application) and the hardware, software, and middleware that enables the service (the platform), we can address the two elements of a telecom solution separately as shown in Figure 1, below. Clearly, dependencies exist between these two elements. For example, software application development, debugging, and deployment requires the platform. The choice for a TEM is whether to allocate time and resources to develop and integrate the platform in-house, or whether to jump-start the application development process by leveraging an externally-sourced standardized platform. The latter approach allows the TEM to focus exclusively on application development.

Figure 1: Software application running on an Integrated Platform

The Case for Platform Outsourcing

Outsourcing the platform clearly has the advantage of reducing an application's time to market. However, less obvious is the often-overlooked complexity of developing the platform itself and maintaining it throughout the product lifecycle. Ironically, it is this complexity that has traditionally been the catalyst for proprietary platform development by TEMs, as previous generations of off-the-shelf hardware often lacked the sophistication, level of control, and manageability needed to create a robust telecom solution. As a result, TEMs were essentially forced to create proprietary hardware platforms to meet their requirements.

However, in the last few years tremendous advances have been made in the area of standards-based platforms through the efforts of industry organizations such as PCI Industrial Computer Manufacturers Group (PICMG) that work in conjunction with TEMs to define robust, carrier-grade standards. For example, the PICMG standard for Packet-Switched Backplane (PSB) specifies significantly improved reliability and performance, while Intelligent Platform Management Interface (IPMI) supports much more robust platform management and control.

While in theory the platform is simply a collection of modules that are integrated within a chassis enclosure, the reality of creating an integrated, scalable, and manageable platform is a very extensive undertaking. When designed correctly, a platform combines reliable architecture, system management, cooling, performance, flexibility, and high availability features in an elegantly simple offering. The end goal of the platform design process must be to produce a solution that delivers performance and simplicity to the TEM while supporting the revenue-generating application. To accomplish this, a platform provider must simultaneously fill the roles of designer, supplier, and integrator for a wide range of building block solutions that include chassis, blades, management software, high availability middleware, and protocol solutions.

CompactPCI Is Evolving

CompactPCI has led the evolution of reliable communications platforms for many years and has become the de facto telecom platform standard with countless successful deployments and a wide array of components available from a variety of vendors. One of the greatest strengths of CompactPCI has been its adaptability. As deficiencies in the architecture have been identified, equipment vendors working in conjunction with PICMG have been successful in addressing these issues.

A recent example of CompactPCI's adaptability involves the PCI bus itself. While the CompactPCI architecture was born from the PCI bus, as the reliability of CompactPCI systems improved the PCI bus slowly became the Achilles heel of the architecture since there was always the possibility that any malfunctioning card might take down the PCI bus and, in turn, the entire system. Recognizing this weakness, in early 2000 Continuous Computing released the first Hi-5 Network Bus Architecture CompactPCI platform. This platform replaced the PCI bus with redundant Ethernet connectivity to each card in the chassis as shown in Figure 2, below. Hailed as a "winner" in the August 2000 edition of Computer Telephony, this platform was designed for the purpose of improving reliability and performance while maintaining the benefits of the CompactPCI form factor.

Figure 2: Hi-5 Network Bus Architecture

Since the introduction of the first Hi-5 platform, the telecom industry has embraced redundant Ethernet connectivity within the chassis through efforts of the PICMG 2.16 working group. Further improving this architectural evolution, the working group took the next logical step by replacing external Ethernet cabling between cards with an embedded, dual-star Ethernet topology routed on the CompactPCI midplane itself. With standard pinouts defined for support of this more reliable and higher performance architecture, both platform and board providers rushed to introduce an assortment of PICMG 2.16-compliant technology solutions. A cursory look at the current CompactPCI market demonstrates the success of this innovation first pioneered by Continuous Computing.

Given the advantages of the Packet-Switched Backplane improvement and its widespread adoption in the industry, it is surprising that no attention has been focused on a similar situation that exists with the CompactPCI power bus. To appreciate this issue, let us first define two requirements for achieving maximum reliability in platform architecture:

1. No Single Point of Failure:
   Redundancy must be provided in every aspect of the architecture; no single point in the system should be able to cause failure.
2. Fault Isolation:
   Potential failures must be isolated to prevent them from spreading and resulting in larger system failures.

With these assumptions in mind, consider the CompactPCI power bus. While most platform providers offer 2N or N+1 redundant power supply configurations, the power from these supplies is distributed across a shared power bus to all cards in the system. This shared bus represents a single point of failure, and there is no possibility of isolating this fault because the power bus distributes power to all of the cards in the system. As a result, a failure of the power bus leads to a failure of the entire system.

Recognizing the inherent power deficiencies in the current generation of CompactPCI platforms, Continuous Computing once again introduced an innovation to the CompactPCI architecture with Flex21, a third-generation CompactPCI platform first introduced in 2003. As with the company's initial Hi-5 platform introduced in 2000, Flex21 improves reliability and performance while maintaining the benefits of the CompactPCI form factor, thus protecting telecom equipment manufacturers' investment in CompactPCI equipment.

Flex21: The Next Generation of CompactPCI

With up to 75 watts of non-shared power per slot, Flex21 supports the current generation of CompactPCI and PICMG 2.16 blades while opening the door to high performance blade designs previously unachievable in CompactPCI. Flex21 leapfrogs the competition with features that include the highest slot density of any CompactPCI platform available, ability to power and cool newer and higher performance blades, and redundant management and alarming functionality developed through years of experience working with Tier 1 TEMs including Alcatel, Cisco, Ericsson, Tellabs, and others. Available with Continuous Computing's award-winning upSuite high-availability middleware for protecting valuable data both in memory and on disk, the custom-configurable Flex21 is the fastest and most reliable path to successful application deployment for TEMs.

Figure 3: Flex21 front view (left) and rear view (right)

Higher Power Capacity

Flex21 solves the power deficiency problem in CompactPCI by rethinking the traditional approach of using large, expensive power supplies to distribute power to all cards in the system. Instead, each slot is powered by a small, efficient, and inexpensive power supply inserted above the standard 6U card cage. Plugged into the midplane, this hot-swappable card provides power to the standard CompactPCI pins for the single slot directly below it as illustrated in Figure 4, below.

Figure 4: Power cards enable independently power slots

The independently-powered slots architecture used by Flex21 has several advantages over traditional centralized power approaches, including:

1. Fault Isolation
   Since each slot is powered independently, any power failure is isolated to only one slot.
2. More Power
   Because each power supply card provides 75 watts per slot, the system is more flexible in supporting a wider range of power-hungry cards. A dual slot, high power version is also available for use with boards such as dual LV Xeon processor blades with large power consumption requirements.
3. Simplicity
   Power budgeting is no longer necessary since power is no longer shared between multiple slots.
4. Scalability
   Pricing is inherently linear. Unlike platforms with large, centralized power supplies that are disproportionately expensive for deployments with only a few cards, the independently-powered slots approach only requires power cards to be purchased for node slots that are being used.
5. Less Expensive Repair
   In the event of a power card failure, not only is the failure isolated to a single slot but the individual power card is also much less expensive to replace than a large, centralized power supply.

Each power card uses standard 3U CompactPCI mechanicals and provides consistent status LEDs on the front panel to indicate whether power is currently being provided to the slot beneath it and whether the power card needs to be replaced. In addition, IPMI/serial controllers on each power card support a radial IPMI architecture with separate IPMI communication paths between each power card in the system and the redundant management cards.

For the purpose of reliability, the power card and corresponding node card beneath it are considered a single entity. However, given the extremely high mean time between failure (MTBF) of the power card, the MTBF of this combined fault group is essentially the same as the MTBF of the node card itself. Also, when compared with older proprietary architectures that integrate the power supply onto the processor or input/output blade itself, the power card approach provides for lower replacement cost in the event that a failure occurs in either the power card or the node card beneath it.

Figure 5: -48VDC power distribution

As shown in Figure 5, above, Flex21 provides dual power input connections from the Central Office Power Distribution Unit (PDU) for each of the redundant -48VDC power feeds. This architecture ensures that amperage constraints on a single power connection, typically 20 Amps, do not limit the power capabilities of the platform. Each of the redundant power input trays support two connections to the redundant power input feeds, allowing either of the power input trays to support -48VDC distribution to the entire chassis. This configuration allows either power input tray to be replaced without affecting the power to the platform.

Flex21 Architecture

Figure 6: Flex21 platform

Flex21 is designed with redundant modules for all important platform functions, and all modules are easily replaceable. Moreover, to ensure easy serviceability, all replaceable modules have a "Replace" LED to avoid costly maintenance mistakes.

The Flex21 platform consists of the following modules:

1. FlexPower
   3U FlexPower modules are the heart of the Flex21 power architecture, enabling independently-powered node slots in a flexible chassis with linear price scalability and low entry cost. Located above each node slot, these hot-swappable modules also incorporate serial and IPMI controllers to enable unified chassis and system management capabilities.
2. FlexManager
   3U FlexManager modules provide redundant, modular, unified chassis- and system-level management capabilities with open, easily extensible interfaces for integration with applications. Located above each fabric slot (Ethernet switch slot), each hot-swappable FlexManager module also provides power to the corresponding Ethernet switch in the slot below.
3. FlexAlarm
   Located at the upper rear of the chassis, the redundant FlexAlarm modules combine management and alarming connectivity with power inputs and rear transition slot Replace LEDs. Each FlexAlarm module provides rear Ethernet, serial, and alarm connectivity for both FlexManager cards. Each FlexAlarm module also serves as a power input tray, providing dual -48VDC power input connectors, filters, breakers, and emergency power-on buttons. For status indications, each FlexAlarm module provides user-controllable Replace LEDs for each rear transition slot beneath the module in addition to a Replace LED for the module itself.
4. FlexConsole
   Located below the rear transition modules on the rear of the chassis, the FlexConsole module enables access to the device console port of every node card in the chassis through the Flex21 chassis management system. This provides convenient single-point access for low-level control and debugging of all cards.
5. FlexPCI
   A factory install option, FlexPCI modules provide the flexibility to configure the standard Flex21 PICMG 2.16 midplane with PCI segments exactly as needed by your application, without incurring the cost and lead-time that would be required to create a custom-designed midplane.

The Flex21 architecture is an evolution of CompactPCI that achieves the optimum combination of standards-based design and forward-looking innovation. The result is an open-standards platform that provides TEMs with the highest degree of flexibility for rapidly developing and deploying new applications. Because the Flex21 platform conforms to PICMG specifications, any standard blade compliant with CompactPCI or PICMG 2.16 may be used. In addition, the power and cooling benefits of Flex21 allow it to support high-performance board designs with power consumption requirements well beyond the limits imposed by other CompactPCI systems. This flexibility allows telecom equipment manufacturers to support today's wide range of CompactPCI and PICMG 2.16 blades while ensuring support for higher performance and more power-consuming blades in the future.

Figure 7: Flex21 PICMG 2.16 compliant connectivity architecture

Flex21 allows TEMs to leverage field-proven and widely available CompactPCI and PICMG 2.16 technology while recognizing substantial improvements in power and reliability. Flex21 conforms to specifications that include:

• PICMG 2.0 - CompactPCI Core
• PICMG 2.1 - CompactPCI Hot Swap
• PICMG 2.5 - CompactPCI H.110 Telephony Bus
• PICMG 2.9 - CompactPCI System Management (IPMI)
• PICMG 2.11 - CompactPCI Power Interface
• PICMG 2.16 - CompactPCI Packet-Switched Backplane (cPSB)

Cooling

A significant advantage of the independently-powered slots architecture is the capability to support 75W per slot, which translates to the ability to support higher-performance node cards with higher power consumption requirements. However, higher power consumption also results in greater cooling requirements. Cooling in Flex21 is accomplished using three redundant fan tray modules to provide high velocity vertical airflow across all slots. Designed specifically for the Flex21 platform, each hot-swappable fan tray module combines two high-performance 5" fans with integrated air intake and performance monitoring. A pressurized NEBS-compliant replaceable filter is provided below the node cards and the chassis is designed with plenum space between the filter and fan tray modules to ensure that airflow is evenly dispersed across all cards in the system. In the event of a fan failure, this air dispersal allows cooling to continue to be provided to boards above the failed fan.

Scalability

Flex21 is designed to provide scalability at both the chassis and system level. At the chassis level, the 21-slot platform provides 19 node slots, more than most other CompactPCI chassis products, which usually require that extra slots be allocated for special purposes such as power or cooling modules. The platform cost-effectively accommodates both lightly and heavily loaded configurations due to the independently-powered slots architecture. Unlike bulk power-based architectures, Flex21 only requires power cards for node slots that are being used. As additional node cards are required for performance or functionality, additional power cards are added to support these node cards. This ability to linearly scale from a small to a large system makes Flex21 ideal for any application from a lightly-loaded lab development system to a fully-loaded mission-critical application server.

At the system level, the Packet-Switched Backplane approach used by Flex21 makes Ethernet a ubiquitous interconnection mechanism both within the chassis and across a complete system deployment environment. Each blade operates as an independent server node with dual Ethernet connections to every other blade regardless of whether these blades are within the same chassis or hosted in a different chassis. In addition to simplifying traditional problem areas such as hot-swap, this approach allows systems to easily scale by simply adding additional chassis systems as required.

Management and Alarming

The fully redundant and hot-swappable FlexManager modules form the basis of management and alarming in the Flex21 platform. These modules are IPMI-based with radial IPMI connectivity to all other slots and modules in the Flex21 platform to ensure reliability.

It is important to understand that integrated platform management entails more than simply integrating management cards. To achieve sophisticated management and control, platform providers must incorporate manageability into every aspect of the platform design. Based on years of experience working with Tier 1 TEMs to meet the management and alarming requirements of the telecom operating environment, Flex21 offers a comprehensive set of capabilities with elegant solutions to many problems traditionally encountered within the central office. The Flex21 management subsystem is built around a registry database to ensure extensibility. Shown below, this architecture ensures a consistent management interface regardless of access method:

Figure 8: Registry-based Flex21 management software architecture

The FlexManager modules have been designed in conjunction with other Flex21 modules to provide seamless, single-point manageability that includes:

1. Telco Alarming
   Each FlexManager module provides critical, major, and minor alarm LED indications on the front panel as well as dry contact relays via FlexAlarm modules on the rear of the chassis. APIs are provided to support integration of alarming into the application. Alarm indications may be cleared remotely or locally at the FlexManager front panel.
2. Management Access
   Multiple management paths are supported both locally and remotely, including serial Command Line Interface (CLI), Telnet to CLI, Simple Network Management Protocol (SNMP), web-based interface, C/C++ APIs, and Python APIs.
3. LED Control
   To ensure easy serviceability, a user-controllable "Replace" LED is provided for each replaceable module in the platform including power cards, node cards, rear transition modules, fan trays modules, and alarm/power input modules. In addition to the Replace LED, a dual color "User" LED is provided for each node card to be used by the application as needed. Also, a "Power" LED is provided for each FlexPower module to indicate that power is being provided to the node card beneath it.
4. Inventory Management
   Each replaceable module in the platform has an associated electronic serial number, part number, and revision for inventory purposes. This inventory information is maintained on redundant EEPROMs located in the chassis and is accessible via IPMI.
5. Power Supply Control and Monitoring
   Power supply control allows each power card to be individually turned on and off. Power monitoring provides output voltage measurements and -48VDC feed detects.
6. Cooling System Monitoring
   Ambient temperature measurements are available at the inlet on each fan tray module and at the exhaust on every power and management card, thereby providing unsurpassed temperature monitoring on a per-slot basis. Performance monitoring is provided for each fan, allowing users to identify fan degradation so that fan tray modules may be swapped before failure occurs. Fan tray -48VDC feed detects are also supported.
7. CompactPCI Monitoring and Out-of-Band Management
   In addition to standard presence detection and hot-swap signals such as HEALTHY#, BDSEL#, and RESET#, single-point access to the serial console port of every node card in the platform is provided regardless of serial pinout used by that card. The FlexConsole panel enables pinout-agnostic access to serial console ports by routing serial data via the midplane to FlexPower cards for transport to the management cards over IPMI.

Flexibility

The standard Flex21 midplane provides PICMG 2.16 support for all slots and H.110 support for slots 3 through 19. PICMG 2.16 support provides dual Ethernet to every slot for high-performance card interconnection at up to Gigabit speeds. H.110 support allows for transport of TDM data between slots for payload applications such as media servers.

Flex21 offers a unique solution for TEM applications that require a PCI bus. Flex21 balances the architectural benefits of moving beyond a PCI bus with the real-world need for a PCI bus in some applications. The Flex21 solution is called the FlexPCI pallet board. A FlexPCI pallet board leverages feed-through pins on J1 and J2 of the Flex21 midplane. By adding appropriately-sized FlexPCI pallet boards to the standard Flex21 midplane as a factory install option, the TEM is able to specify the PCI segments exactly as needed for the application without the cost and lead-time typically associated with a custom-designed midplane.

The Flex21 platform is flexible enough to accommodate a wide variety of application needs. For example, as an open-architecture platform, Flex21 supports different processor architectures and operating system, allowing TEMs to select the most appropriate combination of SPARC, x86, and PowerPC processing resources for the application being deployed. In addition, since the power bus has been removed as a single point of failure, Flex21 platforms may be configured as duplex solutions with active and standby sides within the same chassis; such configurations traditionally required physically separate midplanes to protect against catastrophic power bus failures.

The logical extension of removing the power bus as a single point of failure in a CompactPCI system is that each of the 19 node slots are essentially a separate, isolated servers. As such, previous deployments using physically separate platforms for billing, OAM&P, signaling, media, softswitches, etc., can now be combined into a single, scalable platform with common control and management.

As processing, input/output, or new feature demands increase beyond a single chassis, an additional Flex21 chassis can simply be added as needed to accommodate growth, thereby providing virtually unlimited scalability. Whether using one Flex21 chassis or one hundred, this consolidated, scalable approach to application deployment provides a common platform, a common interconnection architecture, and a common management architecture that translates into a more reliable and serviceable deployment environment with lower total cost of ownership (TCO) over the life of the deployment.

The industry trend is clearly toward smaller, more distributed solutions. However, this is often misinterpreted as a trend toward small 1U or 2U servers. The goal is to reduce the footprint of deployed equipment overall, not necessarily to reduce the size of individual servers. Stacks of 1U and 2U servers are not only a much less space efficient, but the cabling introduced by using numerous small servers presents both serviceability and reliability challenges. Flex21 helps to reduce the overall deployment footprint by providing a high-density platform that efficiently scales in both performance and price. By using Flex21 as a standard platform solution, TEMs can realize cost benefits of dealing with a single, fully-integrated, fully-managed platform capable of accommodating a wide variety of projects and applications.

Flex21 Delivers High Availability

There are two key metrics to understand when discussing high availability (HA): Mean Time Between Failure (MTBF) and Mean Time To Repair (MTTR). MTBF simply refers to how often a piece of hardware fails. Typically calculated using standard Bellcore models, MTBF is a function of the combined reliability of the hardware components that comprise a specific hardware module. On the other hand, MTTR refers to the amount of time required to recover from a failure. Therefore, the goal of a high availability system is to combine redundant hardware components that maximize MTBF with software that works in conjunction with the hardware to assure that all potential and foreseeable fault condition recovery scenarios minimize MTTR.

From a hardware perspective, Flex21 clearly has been designed to maximize MTBF and minimize MTTR. With redundant, easily-replaceable modules for all subsystems including power, cooling, connectivity, and management, Flex21 ensures that any failure that requires physical replacement of a module can be quickly identified and replaced while redundant modules continue to provide service and protect against the potential of lost revenue. In addition, unlike other CompactPCI platforms that do not protect against the possibility of a power bus failure cascading to other slots and resulting in a catastrophic failure in the platform, the unique independently-powered slots architecture of Flex21 protects against this possibility by eliminating the power bus altogether. The worst-case scenario with Flex21 is the failure of a single node slot, in which case the application would simply failover to a redundant node slot without affecting service availability. In this situation the Replace LED of the failed module would illuminate so that service personnel could easily identify and replace the module in a timely manner.

From a software perspective, HA is an often-misunderstood term when applied to telecommunications applications. HA requires much more than simply having reliable hardware. HA refers to the amount of time an application is available to provide the service it has been designed to provide. As such, highly available systems require not only redundant hardware components but also the ability of the software application to manage fault detection and correction without human intervention. The importance of HA software components cannot be understated since these elements could literally spell the difference between keeping customers and losing customers - along with millions of dollars of revenue - as the result of service outages.

Given the importance of HA software components, it is logical to view them as essential building blocks of any highly available application deployment. In the past, TEMs spent a considerable amount of time and effort implementing HA software mechanisms such as application heartbeats, data replication, and failover initiation. Unfortunately, customers typically view availability as an assumed requirement, not a feature. Therefore, while implementation of HA software mechanisms lengthens the time to market and requires additional development resources, the outcome of this time and effort do not necessarily yield additional perceived benefits to the TEM. As a result, these HA mechanisms often fail to receive the necessary focus required to maximize application availability.

The solution is to eliminate the need for the TEM to develop proprietary HA software by integrating robust, field-proven HA components directly into the platform. This allows the TEM to focus all resources on developing the application while enabling the platform provider to focus on reliable and manageable equipment to provide a more complete, fully-integrated, high-availability platform.

Conclusion

Flex21 provides a highly available, fully manageable, and easily serviceable intelligent platform for CompactPCI-based solutions. It offers industry-leading features, including a scalable cost-efficient power architecture, built-in data replication with component failover mechanisms for 99.999% uptime, and an integrated overall system design featuring an award-winning high-speed network bus architecture with redundant Ethernet connectivity to each blade in the chassis. When combined with the robust Trillium suite of telecom protocol stacks (including distributed/fault tolerant Voice-over-IP, 3G Wireless, and SS7/SIGTRAN support, and much more), Flex21 forms the basis for Continuous Computing's complete family of CompactPCI system solutions.

By outsourcing platform development to Continuous Computing, TEMs can substantially decrease the time-to-market requirements of their product development while also realizing higher profit margins. Furthermore, TEMs also gain the ability to target the rapidly expanding and evolving market for converged and wireless network applications in a focused, efficient, and targeted manner.

About Continuous Computing

Continuous Computing provides integrated systems and services that enable telecom equipment manufacturers to rapidly deploy Next Generation Networks (NGN). Over 150 customers worldwide benefit from the company's unique blend of customized professional services, Trillium protocol software, AdvancedTCA and CompactPCI systems, and BladeCenter hardware. Continuous Computing helps customers reduce platform lifecycle costs, optimize data delivery, and accelerate deployments of NGN, 3G/4G Wireless, and IP Multimedia Subsystem (IMS) infrastructure. The company is ISO-9001 and CMMI certified and based in San Diego with development centers in China and India. For more information, visit www.ccpu.com.

Continuous Computing, the Continuous Computing logo, Create | Deploy | Converge, FlexTCA, Flex21, FlexChassis, FlexCompute, FlexCore, FlexDSP, FlexPacket, FlexStore, FlexSwitch, FlexTCA, Network Service-Ready Platform, Quick!Start, TAPA, Trillium, Trillium+plus, and the Trillium logo are trademarks or registered trademarks of Continuous Computing Corporation. Other names and brands may be claimed as the property of others.