Hybrid Cloud and your applications lifecycle: 7 lessons learned

Hybrid Cloud is a must nowadays, I will not spend a word to convince you (you’d not be reading this post if you didn’t believe it). This is the story of a real project.

This post provides more context about the story I summarized at Just 1 step to deploy your applications in the cloud(s).

The structure of the post is:

• Motivation
• Use Cases
• Time
• Software Stack
• Benefit of the architecture we implemented
• Lessons Learned (the most important part)

Motivation for hybrid cloud, and most of the work in my customers' projects, include the following areas:

- Cost control (there is a strong debate: some swear it’s cheaper, others have discovered hidden costs: e.g. network traffic in production, after they made a business case just on the cost of VM provisioning).

- Governance model (IT must find a way to maintain control over resources usage, design patterns, compliance and security when application developers chose private cloud or public cloud).

- Mature technical solution: architecture and technology (there are many good products and system integrators in the market)

But, once you have made a decision, what will you run in the hybrid cloud?

Will your applications be spread across the boundary of your datacenter (one tier inside, other tiers outside)?

Or can we say that it is rather a multi-cloud deployment, where you have a number of resource pools that you can use as a target for deployments?

This project was made by a large corporation, to test how a hybrid cloud can be built and operated and to verify the impact on their current organization.

It is not a full production environment, it’s a pilot project that demonstrated on a small scale how easily you can build a software defined fully automated data center, including both resource pools from your local data center(s) and from public cloud providers.

The solution is expected to be cost effective, of course, but the greater benefits come from business agility and consistent governance.

Use Cases:

The evaluation was focused on 3 main use cases, all requiring that end users order the deployment of a complex software stack from a service catalog: the target for the deployment can be either the private cloud or the public cloud, or a combination of the two. These are the areas where the implementation demonstrated the value of the multi-cloud solution:

• Business Intelligence (self-deployment of R Studio and additional tools)
• ETL (self-deployment of a common software for ETL that data scientists would use in autonomy)
• High Performance Computing (HPC) on OpenStack, with the integration of a DevOps pipeline.

Subject matter experts were provided from different lines of business in the company to support the implementation activities and evaluate the result. 

The use cases represent some frequent activities that the company needs in their usual business, especially in R&D. Improving efficiency and quality in the associated processes will have an impact on the overall business outcome. Applications were selected for the self service catalog that are deployed frequently (every week) and whose installation process takes time (some man days, accounting for both infrastructure and software setup), delaying business objectives.

Time:

All the activities in the project were delivered in time (six weeks), including the setup of the hardware and software systems for the hybrid cloud, the implementation of the 3 main use cases and some additional use cases, the functional tests and the stress tests. This is a demonstration that a proper selection of the technology and a good organization of the project allow for immediate return.

Challenges like setup of the remote access to the lab for remote experts, constraints in the networking and security configuration in the lab, some missing information about the process to install the applications (essential to build the model for the automation) slowed down the implementation. See Lessons learned.

Software Stack:

This is a complete end to end solution: its adoption will happen with a phased approach, starting from the components that grant an easy and immediate impact on the most critical business requirements and adopting some non-functional components later to complete the architecture. The extension from private cloud (based on any combination of VMware, other hypervisors and OpenStack) to a hybrid cloud (integrating AWS, Azure and more) was very quick (it is just a matter of configuration and definition of the governance model). Checkmarks in the picture show what we realized in the short timeframe of the project. The rest is part of a phased plan. The blue boxes show the components provided by Cisco.

The fundamental component in this architecture is Cisco CloudCenter (CCC), that has 2 main roles: 
- providing an orchestration solution that offers users the possibility to self-deploy complex software stacks from blueprints offered in a catalog, 
- brokering cloud resources from both private and public clouds (in the project we integrated VMware, OpenStack and AWS, but more clouds are supported).

CloudCenter manages the lifecycle of software applications in the cloud (at a level of abstraction where the underlying physical infrastructure does not matter).

The OpenStack use cases for HPC are supported by a Cisco Validated Design named UCSO: it includes a reference architecture for running the Red Hat OSP8 distribution on a certified hardware platform made of Cisco UCS servers and Nexus 9000 switches. The setup process and the operations are defined by the official deployment guide and Cisco's technical support assumes responsibility on the entire stack, including the Red Hat software.

The management of the entire DC infrastructure from a single orchestration platform was made possible by Cisco UCSD (UCS Director): a single dashboard and workflow engine to manage servers, network and storage, both physical and virtual. The status, the performances and the remaining capacity of all the systems were monitored with Cisco UCSPM (UCS Performance Manager).

Benefits of the architecture we implemented

The implementation of the multi-cloud solution demonstrated the major benefits that a hybrid cloud delivers.

• A consistent architecture based on software (and eventually hardware) components that integrate easily and satisfy all the business and technical requirements.
• All components in the architecture are loosely coupled and their integration is based on standard protocols and documented open API. As a consequence, every component can be replaced by an alternative solution (from a different vendor, from the open source, from a custom build) with no fear of vendor lock in.
• The adoption of a hybrid cloud solution can happen gradually, starting with a core implementation with the most critical components (e.g. CCC, ACI and UCSO), adding more features as a second step (infrastructure automation and monitoring) with UCSD and UCSPM, eventually a unified service catalog and ITSM portal later.

Lessons Learned:

1. use cases
2. network topology
3. security and trust
4. reusable work (repositories and services)
5. engage SME and business owners
6. document
7. refine (iterations, devops)

Use Cases 
The selection of the use cases is important. You need a quick return to demonstrate the value of the hybrid cloud: the adoption of the hybrid model should address immediate business needs, that the end users can appreciate, rather than be driven just by an industry trend. 
IT projects should not start because a new technology is very smart, but because the outcome makes the business easier and more productive.
Always engage your end users in the planning phase and avoid academic use cases that have a limited appeal on the decision makers. In this project we were lucky because the preparation was done by the steering committee very well in advance.
Once the models for the automation were ready, we could test any combination of the deployment for the application tiers: everything in the private cloud, everything in the public cloud, or the front end deployed on one side and the back end on the other side. The benchmarking capabilities of the product (CCC) allowed to compare the price/performances ratio of the different options based on vSphere, Openstack and AWS - specifically for each application, with tailored reporting.
 

Network Topology

A hybrid environment connects - by definition - areas that were designed separately (your datacenter and the public cloud). They have security policies and configurations that are not meant to work together, and this makes it difficult. Before you start the setup, dedicate the right time to collect all the requirements and to design the connectivity properly. 
We had some issues with the network proxies and the firewalls because of the protocols and ports that we needed to open to allow a proper integration of the Cloud Management Platform (running on premise) with the orchestration engine (with one instance running in each cloud region used in the project, to leverage the local API exposed by the cloud provider and to manage the lifecycle of the applications in the cloud). 

Another important requirement is to have a unique repository for all the artifacts, the blueprints and the installation packages for the applications: it should be reachable from all the target clouds that you plan to use, regardless its location (it can be either in the private or in the public cloud, but all the servers you deploy will access it to stand up a new instance of the application). 
The same applies to any public repository that is used in the setup of the applications (both commercial software and open source components, e.g. packages installed using yum).
See also CCC Components Overview for more detail.

Security and Trust

It's important that a good level of trust is established between the architects building the hybrid cloud and the operations team, especially the security guys. Special rules and new policies need to be setup to allow the new platform to work, it's impossible to keep the same old governance model that addresses a single end user identity. 
Sometimes I feel like I'm living - again - the same conflict that I had with Database Administrators, when I tried to configure JDBC database connection pools in the first Java application servers in the 80's. The system should be trusted, and a delegation of the decisions (authentication, authorization and audit) accepted.

Reusable work (repositories and services)

When you model a software application to automate its deployment, you should identify any building block that can potentially be reused in a different model. If you create a reusable (parametric) deliverable and save it individually in a common repository, next time you'll have the work ready to be reused.
This applies to architectural building blocks like database servers, web servers, load balancers, firewalls, distributed caches, etc. 
If they have been created as separate services, instead of just being a part of a monolithic model, they will appear in your designer's palette everytime you model a similar application and you can drag and drop them in the topology. We did that in the project and we saved a lot of time in the implementation. 

Engage SME and business owners

It is important that subject matter experts (SME) collaborate at the definition of blueprints and the build of the automation model. Even though documentation exists for the deployment of the application, you should work together. 
The user knows all the requirements, he knows how to verify and troubleshoot, he has encountered all the setup issues already.
I've learned that the best way to document the setup process for an application, so that you can use it as a reference for the automation, is to ask the SME to install it in front of you in a clean environment where the application was never run, and record a video of the process. It's faster than writing documentation, more complete and reliable. We did that using the desktop sharing feature in Webex and we recorded the sessions.  

Document

While you do the work, keep track of all the steps. Take (maybe informal) notes, but mostly take a lot of screen shots to document what you did. You can keep them on a wiki or on a shared folder, they will help a lot when you have time to create the formal documentation of the project. If you need to troubleshoot, eventually involving other people, this information will be unvaluable.
Of course, versioning and taking snapshots of all deliverables also helps in case you need to go back for whatever reason. 

Refine (iterations, devops)

Create the implementation for a minimum viable product (MVP) as soon as you can. Get the product (i.e. the entire self service catalog, or just the implementation of a single application blueprint) to early customers as soon as possible, to get their feedback before you go too deep in the implementation.
Applied to a hybrid cloud scenario, this will help to evaluate:
- quality of the service you are building, including documentation
- how much the users need it and use it in the real world
- performances of the distributed environment and any bottleneck (network, computing, configuration)
- security implications 

You will have all the time to make it perfect, through iterations that improve the implementation, collect feedback, allow for tuning the design and the configuration. No need to work in a hurry and make mistakes, while you keep your users waiting for the final "perfect" product but they don't see any progress.

A simpler framework for hybrid cloud

Hybrid cloud is one of top mind projects for most IT managers, and there's little content that one can add to be original   ;-)

The hype and the attempt of many vendors (including... Cisco) to provide relevant solutions have populated the space of an incredible number of offers that make it hard to distinguish what works, what's manageable and cost effective, from what is only marketecture.

Recently Cisco decided to invest even more on cloud and, with the advent of a new CTO and some acquisitions, a revision of our approach to hybrid cloud made it easier and more effective. This post is not from official marketing and is not echoing company's direction: it's my attempt to rationalize my understanding of the new framework and to solicit your comments and feedback, so that I can leverage it when I discuss with my customers and partners.
The following picture represents the area where Cisco plays a role, offering hardware and software solutions.
When it comes to the software stack to manage the infrastructure and provide services to the users, we have a mix of Cisco products, open source solutions and integration with 3rd parties. The objective is to offer a set of pre-validated stacks that can match the different needs, granting a deterministic result.

I shared some thoughts with a group of colleagues because we're planning educational activities for our field people: instead of just providing a reference architecture (that would end being a list of products to be forced in every deal) we tried to represent the functions in the system as components of a framework, from which we'll pull the specific architecture for a given project. This, used cum grano salis, should help to be pragmatic and realize quick wins (for both the customers - think of Fast IT initiatives - and of course for Cisco).

As a result, next picture is separating the different functional layers so that they can be explained to sales guys and to customers.
It could also help to manage the possible overlap with alternative solutions that customers may choose – or already have – because every element is replaceable in the picture, based on the open API they expose/consume (as well as any well designed 3rd party product).

It is important to note that the top two layers in the picture are optional, since not all customers need those functions in their system. Based on the level of Governance that they want to have, the existing processes and the way they develop business applications (or use commercial software that only need a resource pool to be deployed), the entry point could be directly at the third layer (Multi-Cloud Management) and ITSM and PaaS would be removed.

So, while we explain all the possibilities as said above, we need to make them feel confident that it’s doable and not overly complex.
In that regard, my motto is that “cloud is not a product (or a set of), it’s a project and it’s complex in nature… regardless the products set you choose”. Generally the cost of hardware and software products is lower than development and consulting services, and customers know it.
If we can claim that a pre-built integration makes the project easier (and we can), I would stress the value of reducing the project risk and delivering outcomes faster rather than a cheaper implementation.

Selling licenses can be (almost) easy, but driving adoption with business outcomes for customers is different. Finally Cisco has built a practice that can deliver IT projects effectively and recruited partners that do the same: customers have different options to choose from.

Now, in the context of a end to end strategy defined with the customer, we can deliver projects based on agile methodologies (e.g. Scrum) and implement the architecture layers with a bottom up approach: from a strong capability to automate the Data Center (and the hybrid cloud) you can create services that are surfaced up to the consumption layers, including a self service catalog.

Software Defined What?

The bottom up approach stresses the value of the API exposed by UCS and ACI (with the further evolution from basic programmability to policy-based management, that I'm not mentioning yet - look out for next post). With the power and the granularity of those API, you can really realize a fully Software Defined Data Center (SDDC): servers and networks can be shaped via software interfaces.
By the way, I take the opportunity here to clarify that Software Defined Data Center does not mean Software Implemented Data Center: you don't necessarily need a software overlay that mimics the behavior of the hardware (living as a separate entity), you need software controllers that drive the shape and the behavior of both physical and virtual resources in the DC as a single system.
Better if they do that based on policies... like the Cisco architecture does  :-)
You will see a post dedicated to policies and application intent soon on this blog.

Competition?

We also recognize that many customers have already an ITSM solution in place, or any other form of governance. So we don't engage in competitive fights, like imposing Cisco Prime Service Catalog vs Service Now, but we rather integrate our solution with the existing components: this is a sort of compromise with a competitor that hurts my pride, but since it's for our customers' benefit... it's a good solution.

Cisco Cloud Center as a broker: the recent acquisition of Cliqr brings a great solution to Cisco to address the multi-cloud management use cases, the most important ones for the majority of customers. In the logical schema above you can see that the hybrid cloud scenario has been qualified better as Multi-Cloud management.
This reflects the fact that having a application deployed partly in your Data Center and partly in the public cloud is still a relevant use case, but many companies are more attracted by other scenarios... like moving from one project stage to next (e.g. Dev-Test-QA-Prod) using different resource pools (on premise or in cloud), or moving their assets from one cloud provider to a different one.

Cloud Brokering and Multi Cloud Management

In the first one (promotion to next stage) it could be useful to leverage resources that are allocated based on business convenience (e.g. cost or flexibility) or compliance (e.g. data sovereignty), so the application and all the needed infrastructure are moved back and forth to the public cloud.
In the second the driver could be a dual provider strategy, or maybe a change in the market conditions that makes one provider more appealing than the current one, or a strategic switch from private cloud to public (or vice versa).


In all these cases, we offer a solution to deploy a software stack (a complete custom application, a development platform, or a commercial software product) as a self service option, where the target can be selected dynamically from a list of available clouds.
You can deploy to your local private cloud, based on vmware or any other virtualization solution, or to a Openstack based cloud, or to any of the public cloud providers if you have an account there.
Any resource pool is a possible destination for the deployment (and the life cycle management, including autoscale or retirement of the application).
The model of the deployment of the application is completely de-coupled from the selection of the target, thanks to the capabilities of the orchestrator that can configure the needed resources in almost any cloud transparently.
It uses the API exposed by the element managers of a multi vendor infrastructure on premise (e.g. vcenter, UCS Manager, the ACI controller, etc.) and those exposed by public clouds like AWS, Azure, etc.

From a logical schema to a real deployment

So we can offer users a different entry point, based on their business needs (they might need a ticketing system, or a self service catalog, a PaaS solution or directly the web portal of the multi cloud manager to model deployments and deliver them).
The customer can have one or more resource pools, allocated wherever he likes (local or in cloud), and let the broker direct the selection of the target based on predefined policies.

The schema in next picture presents different products at every layer: a solution can be based on one of them, or a combination. We have the flexibility to match the specific needs with products from Cisco, from 3rd party vendors or open source.
As an example, MANTL is a new open source project that makes the development of microservices easier if you build cloud native applications.

I will expand the detail of the single products and the open source solutions shown in this picture in my next post.
Stay tuned...

References

http://www.cisco.com/c/en/us/solutions/executive-perspectives/fast_it.html
http://www.cisco.com/web/solutions/trends/futureofit/why-cisco.html
http://MANTL.io
http://Github.com/CiscoCloud/microservices-infrastucture 
http://lucarelandini.blogspot.it/2015/10/devops-docker-and-cisco-aci-part-1.html
http://lucarelandini.blogspot.it/2015/03/aci-for-dummies.html
http://lucarelandini.blogspot.it/2015/09/the-phoenix-project-how-devops-can.html