Doc.

Code

Huawei FusionSphere 5.1

Technical White Paper on
OpenStack Cascading
Technology

Issue 01

Date 2015-04-20

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2015. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without prior
written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.
All other trademarks and trade names mentioned in this document are the property of their respective
holders.

Notice
The purchased products, services and features are stipulated by the contract made between Huawei and
the customer. All or part of the products, services and features described in this document may not be
within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,
information, and recommendations in this document are provided "AS IS" without warranties, guarantees or
representations of any kind, either express or implied.
The information in this document is subject to change without notice. Every effort has been made in the
preparation of this document to ensure accuracy of the contents, but all statements, information, and
recommendations in this document do not constitute a warranty of any kind, express or implied.

Huawei Technologies Co., Ltd.

Address: Huawei Industrial Base
Bantian, Longgang
Shenzhen 518129
People's Republic of China

Website: http://enterprise.huawei.com

Issue 01 (2015-04-20) Huawei Proprietary and Confidential i

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology About This Document

About This Document

Purpose
This document describes the FusionSphere OpenStack cascading functions.

Intended Audience
This document is intended for Huawei marketing and sales engineers, as well as FusionSphere
distributors in their market development projects.

Symbol Conventions
The symbols that may be found in this document are defined as follows.

Symbol Description
Indicates an imminently hazardous situation which, if not
avoided, will result in death or serious injury.

Indicates a potentially hazardous situation which, if not

avoided, could result in death or serious injury.

Indicates a potentially hazardous situation which, if not

avoided, may result in minor or moderate injury.

Indicates a potentially hazardous situation which, if not

avoided, could result in equipment damage, data loss,
performance deterioration, or unanticipated results.
NOTICE is used to address practices not related to personal
injury.
Calls attention to important information, best practices and
tips.
NOTE is used to address information not related to personal
injury, equipment damage, and environment deterioration.

Issue 01 (2015-04-20) Huawei Proprietary and Confidential ii

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology About This Document

Change History
Changes between document issues are cumulative. The latest document issue contains all the
changes made in earlier issues.
Issue 1.0 (2015-04-20)
This issue is the first official release.

Issue 01 (2015-04-20) Huawei Proprietary and Confidential iii

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology Contents

Contents

About This Document .................................................................................................................... ii

1 What Is OpenStack Cascading ................................................................................................... 1
1.1 Needs and Challenges for Multi-site Cloud Services ................................................................................................... 1
1.2 Limited Innovation Based on the Existing OpenStack Architecture ............................................................................. 2
1.3 Design Concepts on OpenStack Cascading .................................................................................................................. 2
1.4 OpenStack Cascading ................................................................................................................................................... 3
1.5 Requirement for Multi-site Cloud Computing Met Using Cascading OpenStack System ........................................... 4
1.5.1 Unified OpenStack APIs Exposed for Multi-site Cloud ............................................................................................ 4
1.5.2 AZ-Level Fault Isolation ............................................................................................................................................ 4
1.5.3 AZ-Level Routine O&M and Upgrade ...................................................................................................................... 4
1.5.4 Plug-and-play Integration to Prevent Vendor-lock-in Risks ...................................................................................... 4
1.5.5 Cross-site Horizontal Expansion Supported by the Scale-out Architecture ............................................................... 4
1.6 Typical Deployment Plan of OpenStack Cascading ..................................................................................................... 5
1.7 Typical Application Scenarios of OpenStack Cascading .............................................................................................. 5
1.7.1 Scenario 1: VDCs Created Across Physical Data Centers ......................................................................................... 5
1.7.2 Scenario 2: Cross-DC Automatic Scaling, Load Balancing, and DR ........................................................................ 5
1.7.3 Scenario 3: VM and Volume Migration Across Physical Data Centers ..................................................................... 5
1.8 Summary ....................................................................................................................................................................... 6

2 Overview of OpenStack Cascading Technology .................................................................... 7

2.1 OpenStack Cascading ................................................................................................................................................... 7
2.1.1 Implementation .......................................................................................................................................................... 7
2.1.2 Related Components .................................................................................................................................................. 8
2.2 Nova Cascading ............................................................................................................................................................ 8
2.3 Cinder Cascading .......................................................................................................................................................... 9
2.4 Neutron Cascading........................................................................................................................................................ 9
2.4.1 Point-to-point Large Layer 2 Network ....................................................................................................................... 9
2.4.2 DVR ......................................................................................................................................................................... 10

Issue 01 (2015-04-20) Huawei Proprietary and Confidential iv

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology 1 What Is OpenStack Cascading

1 What Is OpenStack Cascading

1.1 Needs and Challenges for Multi-site Cloud Services

With the growing popularity and wide application of OpenStack, OpenStack has become the
de facto standard for public and private cloud construction. However, when constructing
OpenStack for multiple sites, enterprises rise to the following challenges:
Challenge 1: Centralize cloud services and schedule resources across sites.
Some customers, such as large-sized enterprises and telecom carriers, require centralized
cloud services and resource scheduling capabilities. Because they have multiple scattered data
centers deployed to support their branches and service sites, thereby providing local access for
end users. Multi-site scheduling covers computing, network, and storage resources. Some
tenants also require centralized cloud services because they have virtual resources distributed
in multiple sites, but the resources need to be isolated on layer 2 and layer 3 networks. To
maintain a healthy service ecosystem, a centralized service system must provide standard
OpenStack application platform interfaces (APIs).
Challenge 2: Provide always-online, easy-to-manage cloud services.
The growth of distributed cloud service system scale presents a great challenge to availability
and manageability of cloud services. If any node fails, causing cloud service interruption, the
result is unacceptable. Therefore, faults must be locally isolated. Routine maintenance and
upgrade operations, such as configuration changing, troubleshooting, error rectification, patch
updating, and upgrading, are challenging for experienced engineers if the operations must
cover the whole cloud platform. These operations must be performed for minimum system
units one by one. The whole operation process may even last for several months.
Challenge 3: Provide scalability for multiple sites.
Managing large-scale instances, such as millions of virtual machines (VMs), is challenging
for a single OpenStack system. The management covers many aspects, including resource
collection and scheduling of computing nodes, Layer 2 information distribution, route update
of distributed virtual routers (DVRs), remote procedure call (RPC) message interaction
between large-scale nodes, and data addition, deletion, modification, and query of databases.
Even if an OpenStack instance can process millions of VMs, the OpenStack instance faces
many challenges when it covers multiple data centers. For example, how can RPC messages
communicate across data centers? If a message bus fails, API servers may be active. However,
all the computing nodes, L2 nodes, and L3 nodes have been hosted. Multi-vendor hardware
and software integration across data centers through the RPC is also a challenge.

Issue 01 (2015-04-20) Huawei Proprietary and Confidential 1

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology 1 What Is OpenStack Cascading

Challenge 4: Multi-vendor OpenStack distributions coexist, and multiple OpenStack instances

and versions are deployed in distributed multi-site cloud service systems.
Some customers, such as large-sized enterprises, telecom carriers, branches, and service sites,
have multiple branches and service sites that have long-term cooperation relationship with
different suppliers. Each site chooses one or more suppliers based on local resources and the
suppliers' delivery capabilities. Each site has its construction plan. However, maintenance and
upgrade operations require standard OpenStack APIs. Therefore, to manage multiple
OpenStack instances and distributions from multiple vendors across multiple sites, quick and
efficient integration must be implemented.

1.2 Limited Innovation Based on the Existing OpenStack

Architecture
The existing OpenStack architecture limits innovation, and the OpenStack community, as well
as other partners (from backend suppliers to cloud service carriers) in the ecosystem, does not
allow this architecture to be reshaped. Therefore, to build a new OpenStack architecture to
overcome these challenges is not practical. The only method is to implement the following
under the existing architecture:
 The existing OpenStack architecture remains.
 Standard OpenStack APIs are provided to allow multiple OpenStack systems to be
cascaded as a whole, which functions similar to a single OpenStack system and is
friendly to the ecosystem.
 Each site is allowed to work independently using RESTful APIs or command line
interfaces (CLIs). Each site must be always able to work and manage its services
independently. RESTful APIs provide higher availability than RPC messages for
cross-region communication and even communication over Internet.
 Based on multi-supplier service policies, each site can be at least established by different
suppliers. However, integration workload across vendors in the same time should be
reduced. Interoperability and integration of multi-vendor infrastructure takes much time.
When the granularities of configuration modifying, updating, restarting, and integrated
units are smaller, the interoperability and integration will take more time and be more
complex.
 Each distributed website can be integrated and hidden under a unified cloud service. To
quickly integrate the distributed systems, each site must provide stable RESTful APIs
and standard APIs.
Innovation potentials are limited for rising these challenges and providing solutions.

1.3 Design Concepts on OpenStack Cascading

OpenStack is designed for managing and scheduling computing clusters. Therefore, it may be
a good choice to regard an OpenStack system as a large-scale computing node.
OpenStack community supports remote management of hypervisor clusters, such as vCenter
and Nova Ironic. However, the management is only a working mechanism in Nova. Can this
mechanism function in Cinder, Neutron, Ceilometer, and even Glance?

Issue 01 (2015-04-20) Huawei Proprietary and Confidential 2

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology 1 What Is OpenStack Cascading

Only each OpenStack service architecture is understood, this problem can be easily answered.
OpenStack services typically consist of APIs, message buses, databases (DBs), and distributed
nodes supporting different backends.
OpenStack usually manages nova-compute, such as kernel-based virtual machines (KVMs)
and Xen, Cinder, such as Ceph and logical volume managers (LVMs), L2 and L3 agents, such
as open vSwitch (OVS), LinuxBridge, and Router, and Swift, such as Ceph and simple
storage service (S3).
Nova, Cinder, Neutron, Glance, and Ceilometer can be configured to the nova-compute
hypervisor, the cinder-volume storage backend, the network device used for the
interconnection of L2 and L3 agents, the Glance storage backend, and Ceilometer data
backend, respectively, using the OpenStack driver and agent mechanism.
Therefore, more OpenStack management and scheduling methods can be provided using the
OpenStack architecture and mechanism. This mode is called OpenStack cascading.

1.4 OpenStack Cascading

 The Cascading OpenStack system (parent OpenStack) exposes standard OpenStack
APIs.
 The Cascading OpenStack system (parent OpenStack) uses standard OpenStack APIs to
control multiple cascaded OpenStack systems.
 Each cascaded OpenStack system (child OpenStack) can be regarded as a component
that is similar to Amazon Web Services (AWS) AZ, and is hidden by the cascading
OpenStack system. Cascaded OpenStack systems can be distributed on multiple sites.

The cascading OpenStack system: provides APIs, scheduling, and orchestrating services, and
connects cascaded OpenStack systems.
The cascaded OpenStack system: is used to deploy VMs, volumes, and virtual network
resources.

Issue 01 (2015-04-20) Huawei Proprietary and Confidential 3

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology 1 What Is OpenStack Cascading

1.5 Requirement for Multi-site Cloud Computing Met

Using Cascading OpenStack System
1.5.1 Unified OpenStack APIs Exposed for Multi-site Cloud
The cascading OpenStack system has multiple cascaded OpenStack systems integrated based
on standard OpenStack APIs, and the cascading OpenStack system exposes only its own API
endpoints, rather than those of all the other cascaded OpenStack systems. All ecosystems
based on OpenStack APIs are compatible with OpenStack systems. If a third-party cloud
management system that does not provide standard OpenStack APIs is used to integrate
multi-site distributed cloud services, this system cannot leverage existing OpenStack API
ecosystem to meet service needs.

1.5.2 AZ-Level Fault Isolation

Each cascaded OpenStack system serves that is similar to Amazon Web Services (AWS) AZ
and provides the CLIs and RESTful APIs for management. The fault of any cascaded
OpenStack system does not adversely affect the services of other cascaded OpenStack
systems and the cascading OpenStack system. Even if the cascading OpenStack system fails,
resources of the cascaded OpenStack systems are still running and can be managed using
local OpenStack APIs. Therefore, the OpenStack cascading mechanism helps to set up
always-online, manageable, highly-available cloud services.

1.5.3 AZ-Level Routine O&M and Upgrade

The OpenStack community ensures compatibility of OpenStack APIs with their earlier
versions and supports coexistence of multiple versions. The cascading OpenStack system uses
built-in Python clients to interconnect cascaded OpenStack systems to make cascaded
OpenStack systems of multiple versions to run on a centralized cloud system and integrate
them through the cascading OpenStack system. Each cascaded OpenStack system can work
independently, and does not perceive whether other cascaded OpenStack systems are present
or run in cascading mode. Cascading allows routine O&M and upgrades to be performed AZ
by AZ, rather than on the whole cloud platform.

1.5.4 Plug-and-play Integration to Prevent Vendor-lock-in Risks

OpenStack APIs are compatible with earlier versions and coexisting with multiple versions.
The high compatibility benefits integration of distributed cloud platforms using OpenStack
APIs. OpenStack can be built in physical resources of each supplier, making integration (such
as parameter adjustment, configuration, and patch) to be completed before delivery, and
making delivery boundaries clear in the cloud computing project integration process. Any
suppliers can simplify the whole integration process using stable and standard OpenStack
APIs when providing cloud computing infrastructure. Therefore, the OpenStack cascading
technology can provide plug-and-play integration to prevent vendor-lock-in risks.

1.5.5 Cross-site Horizontal Expansion Supported by the Scale-out

Architecture
The cloud platform provides services using RESTful APIs of the cascading OpenStack system.
RESTful APIs allow cascaded OpenStack systems, especially those at the geographically
scattered sites, to be integrated across internal networks or even the Internet.

Issue 01 (2015-04-20) Huawei Proprietary and Confidential 4

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology 1 What Is OpenStack Cascading

1.6 Typical Deployment Plan of OpenStack Cascading

One data center contains one or multiple cascaded OpenStack systems, each of which is
considered as an AZ.
One cascading OpenStack system can manage one or multiple data centers.
The cascading OpenStack system only needs to manage dozens of proxy nodes; VMs,
volumes, and network resources are all running at the cascaded layer. If the cascading
OpenStack system manages 100 cascaded OpenStack systems, and each cascaded OpenStack
system manages 10,000 VMs, a distributed cloud data center that houses millions of VMs can
be easily established. Therefore, compared with managing millions of VMs in a single
OpenStack system, it is much easier to manage 1,000,000 computing nodes.

1.7 Typical Application Scenarios of OpenStack Cascading

Application programs using OpenStack APIs can be seamlessly applied to the cascading
OpenStack system. Because AZs are naturally distributed in different physical data centers, all
cross-DC features can be used.

1.7.1 Scenario 1: VDCs Created Across Physical Data Centers

A large-scale cloud carrier wants to create a virtual data centers (VDCs) over distributed
physical data centers, a VDC contains secure and isolated tenant resources, such as VMs, L2
and L3 networks across data centers, and advanced network services, including firewalls
(FWs), load balancers (LBs), and virtual private networks (VPNs). The cascading OpenStack
can provide L2 and L3 network services across data centers, and VMs and volumes can be
distributed in different data centers.

1.7.2 Scenario 2: Cross-DC Automatic Scaling, Load Balancing,

and DR
The cascading OpenStack can provide global L2 Virtual Extensible LANs (VXLANs) or
virtual local area networks (VLANs), and L3 DVRs across multiple cascaded OpenStack
systems. These networks allow the OpenStack system to provide cross-DC automatic scaling,
load balancing, and disaster recovery (DR) across DCs.

1.7.3 Scenario 3: VM and Volume Migration Across Physical Data

Centers
VMs and volumes can be migrated from a data center to another: Create a snapshot for a VM
or volume in an AZ in a data center, and use the snapshot to restore the VM or the volume in
another data center. If a VM is connected to a global L2 VXLAN, its IP and MAC addresses
remain unchanged. If the storage-based replication function across data centers is supported,
migration duration can be shortened to several minutes or even several seconds.

Issue 01 (2015-04-20) Huawei Proprietary and Confidential 5

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology 1 What Is OpenStack Cascading

1.8 Summary

OpenStack cascading allows one OpenStack system to manage other OpenStack systems,
provides unified OpenStack APIs, and centralizes management of multiple data centers.
 Open architecture: The cascading OpenStack system provides two levels of standard
OpenStack APIs.
 Plug-and-play integration: Any third-party infrastructure that provides standard
OpenStack APIs can be rapidly integrated into the OpenStack architecture.
 Reliable fault isolation system: A single cascaded OpenStack system can manage 1024
servers, restricting the fault impacts within a single cascaded OpenStack system. Even if
the cascading OpenStack system fails, cascaded OpenStack systems are still manageable.
The system is highly available and fault-tolerant.
 Isolation during an upgrade: A single cascaded OpenStack system is upgraded without
affecting other systems, and the system supports coexistence of multiple versions.
Upgrading a single component will not force the whole system to upgrade.
 Horizontal expansion: Cascaded OpenStack systems provide horizontal expansion
capabilities at the server and cascaded OpenStack system levels from several to millions
of servers.
 Multi-data center management and large-scale deployment: The OpenStack
architecture offloads a single OpenStack system by two-phase scheduling. In phase I, 1
million VMs can run on 100,000 servers, and in phase II, 10 million VMs can run on 1
million servers.

Issue 01 (2015-04-20) Huawei Proprietary and Confidential 6

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology 2 Overview of OpenStack Cascading Technology

2 Overview of OpenStack Cascading

Technology

2.1 OpenStack Cascading

2.1.1 Implementation

The cascading OpenStack system interconnects cascaded OpenStack systems in driver and agent mode
and provides the cascading service for Nova, Cinder, Neutron, and Ceilometer. As global components,
Keystone and Glance are deployed on the cascading OpenStack system.

Issue 01 (2015-04-20) Huawei Proprietary and Confidential 7

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology 2 Overview of OpenStack Cascading Technology

2.1.2 Related Components

The components in the cascading OpenStack system that interconnect with cascaded OpenStack systems
are called proxies.

2.2 Nova Cascading

The cascading OpenStack system typically has only AZ and host aggregate filtering algorithms
configured.

Issue 01 (2015-04-20) Huawei Proprietary and Confidential 8

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology 2 Overview of OpenStack Cascading Technology

2.3 Cinder Cascading

The cascading OpenStack system has the AZ filtering algorithm configured.

2.4 Neutron Cascading

2.4.1 Point-to-point Large Layer 2 Network

Issue 01 (2015-04-20) Huawei Proprietary and Confidential 9

Copyright © Huawei Technologies Co., Ltd.
Huawei FusionSphere 5.1
Technical White Paper on OpenStack Cascading
Technology 2 Overview of OpenStack Cascading Technology

2.4.2 DVR

Issue 01 (2015-04-20) Huawei Proprietary and Confidential 10

Copyright © Huawei Technologies Co., Ltd.