Become a cloud provider in 3 months

This is the story of a company that decided to become a Cloud Service Provider.
They were already a successful IT outsourcer in the financial industry, with many customers' environments running in their data center.
Outsourcing was a healthy business but they started having some challenges, due to slow and inefficient provisioning processes and operations.
Any new request from a customer started a new project, so their customers started exploring public cloud services to get more flexibility and speed.
For this reason, the company decided to adopt the cloud delivery model and to offer their customers a self service catalog.

Of course a cloud project cannot be done in one night, so they were cautious in their approach.
Both technology and operational processes needed to be proven before embarking in such a challenge, but the traditional waterfall methodology made the expected return appear uncertain and distant.
To make things worse, they had tried to implement a PaaS project with a different vendor and they had spent a lot of money without tangible return.

I was engaged to support the evaluation of a new IaaS catalog that could evolve to PaaS and to self service applications management.
I made sure that the Business and IT strategy were in sync and I proposed to start with small steps to validate the approach. I also invited them to qualify the quick wins that they would expect to justify the investment and show the stakeholders an immediate return, so that the project lived enough to reach the expected success.
As you know well, many projects last too much and die before showing any business return.

We analyzed the current situation and defined a future vision. This was the driver for a gap analysis and for the prioritization of user stories, that we decided to implement in short iterations (sprints of 2 weeks, according to the Agile Scrum methodology).
Their data center was mainly based on Cisco networks and servers, but this was not the main reason for selecting the Cisco software stack for the cloud project.
After the initial workshops, some product demo and talks about other projects they understood that our people - and our partner company that implemented the project with them - were experienced enough to plan the project seriously and to chase the quick wins that we all considered so important.

The Cloud Management Platform chosen for the project was the Cisco ONE Enterprise Cloud Suite (aka ECS).

One of the most important features considered in the decision was the possibility to create flexible templates, later exposed as self service options in the end user catalog, for the deployment of complex applications. A set of servers with different roles, and all the networks needed to make them work, can be provisioned as a dedicated and virtually separated environment (multi tenancy in a shared infrastructure that offers economy of scale).

As an example, the following picture shows a environment that could be ordered - fully configured - with a single click. It is based on a component of the ECS architecture that is named VACS (virtual application cloud segmentation):

 

It was easy to engage the SME (subject matter experts) for the servers, the network, the storage and the virtualization in the customer organization and to ask them to define the basic policies that we would use as building blocks for all the services to be offered.
This model-based implementation is quicker to build and easier to maintain, and it can be exposed to the end users in a way that they understand and trust soon.

The automation that we built was considered useful by the SME (after winning their initial suspicion, because every good craftsman loves manual work) because it set them free from the manual operations that previously made their work tedious and error prone.
Delegating the configuration to an automated service gave their customers a faster service and a higher quality (no rework needed because of manual errors or missing information).

One more component in the architecture is the Stack Designer.
It is a tool provided by the Cisco ECS to create templates for application provisioning. It takes IaaS templates - made in the infrastructure management layer, that in our case is UCSD, to deploy a topology of servers and networks - and layers the software stack on top of them.

You can decide what software products (or custom applications) must be installed - and configured based on the input parameters provided by the end user - including monitoring agents and backup agents, and save this new template in the repository.
The integration with Puppet, an open source solution used to provision software applications, is leveraged to install and configure the entire software stack from the images in the repository.

The new template can now be offered as a self service option in the catalog, so that the end users don't need to install and configure the software stack themselves. A end-to-end solution is provided, up and running and ready to be used.
All the components of the ECS solution are pre-integrated and this makes the project faster than you would expect. But, since they communicate through standard protocols and open API, every component of the architecture could be replaced by an alternative product (from a different vendor or from the open source community). You should not be afraid of vendor lock in  :-)

Agile Delivery

In terms of project delivery, the following table shows the different iterations that allowed to complete the delivery in only 3 months.
But the amazing result is that at every sprint (i.e. every 2 weeks) new use cases were available in a usable environment.
The first demo to a real customer (a customer of my customer) was done after 2 months from the start of the project, and the first customer was onboarded after the 5th sprint (i.e. 2.5 months).

Conclusion

This quick win demonstrates that even complex projects like building a public cloud platform can be done in a reasonable amount of time.
The era of endless projects, based on complex technology and measured in function points, has passed forever.
There are simple solutions (like ECS) that make your work easier, but a good organization and the right methodology allow for incremental building and refinement of the solution. Every iteration of the project delivers a usable result in the production environment, and you don't need to wait the completion of the entire project to start using the solution.
If you are a service provider, you can start selling your services soon and produce a ROI.
More services will be added incrementally and the catalog will be richer at every iteration.

References

Cisco Enterprise Cloud Suite
and its individual components:
- Cisco PSC - Prime Service Catalog 
- Cisco UCSD - UCS Director
- Cisco VACS - Virtual Application Cloud Segmentation

Fast IT
Cisco Prime Service Catalog in action: Cisco eStore

Scrum (agile development) 

Governance in the hybrid cloud

This post shows how a company can solve one of the main issue that CIOs have today: the so called Shadow IT.

This term defines the usage of cloud services (either IaaS, PaaS, SaaS) in a project without any control, decided by the application developers or designers because they think it's beneficial for the agility of the project.

Sometimes leveraging available services is really good for a project: it's useless to rebuild something that is easily available as a standardized service. Even when the IT organization of your company (or your customer, if you're a consulting company) provides the building blocks that you need for your architecture, it could be difficult to get approvals or a fast enough provisioning.
So there are different valid reasons to incorporate public cloud services, we can't blame those that try to fully exploit a Service Oriented Architecture.

Unfortunately this way of assembling applications using any available resource you consider useful creates troubles for the IT organization.
Besides additional costs, that arrive as a surprise (developers bill to a personal credit card or to a corporate one, but sooner or later those costs will be factored into the cost of the project), some corporate rules could be violated without even being aware.
Just a few examples: storing reserved data in a database outside the company's datacenter, or invoking services without encrypting the input/output parameters, not granting the end to end High Availability or Disaster Recovery of the entire system.

The subject of costs can be easily underestimated: at development time you need very limited cloud resources, for a limited time. It costs near to zero, before the application goes to a full production environment. But after that, it will need more computing power and more storage, and of course more bandwidth, to serve all the users. Cloud services tend to increase surprisingly in these conditions.

So the CIO has a dilemma: to try to block, or limit, the usage of cloud services - limiting cost and risk but appearing like the one that slows the innovation down and prevents the lines of business from achieving their business result - or to allow maximum freedom, with the additional risk of becoming not relevant because they can bypass the IT organization?

There is a solution in the middle: IT could offer a facilitated access to cloud services, adding them to a Service Catalog where users can self serve, granting compliance by design.
Public cloud services will be selected based on agreed architectural and security policies, they will be documented, audited and reported, eventually subject to approval from a financial standpoint.

One possible implementation of such a catalog can be based on the Cisco ONE Enterprise Cloud Suite, as I did in a recent project at one of my customers.

The Cisco ECS is a reference architecture comprising one flexible Service Catalog, a automation engine and a platform for hybrid cloud that allows the extension of your datacenter into a kind of "bubble" in the public cloud. In case you need additional power, you can burst your workloads into the virtual private datacenter keeping all the security and networking policies you defined in your private cloud: even the IP address of the virtual machines does not change, as long as the secure segmentation of the application layers and any other policy.

I'm not going to describe the Cisco ECS, because you can find the official documentation here.
I'm showing how we extended the services offered in this catalog with CliQr Cloud Center for managing the provisioning and the lifecycle management for applications in the cloud. So the great capabilities of Cisco ECS in term of IaaS are complemented with the offer of the deployments of simple or complex applications and software stacks, that you can target at any cloud just selecting from a drop down list.

I mean that the template for the deployment is not cloud dependent,  and the user can - within the limits of his authorization level and the corporate policies - choose to provision it in the private cloud (e.g. on vmware in the corporate data center) or in the public cloud (e.g. AWS or Azure).
The lifecycle operations (start, stop, resume, delete, etc.) will be also offered as well as the migration to a different cloud: from private to public after the QA test is done and you're ready for production, from a public provider to a more convenient one, etc.

THIS POST HAS BEEN REDACTED

After the publication of this post Cisco announced the intent to acquire Cliqr (not because of the post :-) ), and our policies require that we don't speak of deals while they are in progress. I can't show the way we integrated Cliqr in this project because the official statement on the reference architecture will be communicated by Cisco after the acquisition is eventually completed.

References:
http://blogs.cisco.com/datacenter/introducing-cisco-one-enterprise-cloud-suite
http://www.cliqr.com/

DevOps, Docker and Cisco ACI - part 2

This post is a follow up to the initial discussion of the DevOps approach based on Linux containers (specifically with Docker).
Here I elaborate on the advantage provided by Cisco ACI (and some more projects in the open source space) when you work with containers.

Policies and Containers  

Cisco ACI offers a common policy model for managing IT operations.
It is agnostic: bare metal, virtual machines, and containers are treated the same, offering a unified policy language: one clear security model, regardless of how an application is deployed.

ACI models how components of an application interact, including their requirements for connectivity, security, quality of service (e.g. reserved bandwidth for a specific service), and network services.   ACI offers a clear path to migrate existing workloads towards container-based environments without any changes to the network policy, thanks to two main technologies:

• ACI Policy Model and OpFlex   
• Open vSwitch (OVS)   

OpFlex is a distributed policy protocol that allows application-centric policies to be enforced within a virtual switch such as OVS.
Each container can attach to an OVS bridge, just as a virtual machine would, and the OpFlex agent helps ensure that the appropriate policy is established within OVS (because it's able to communicate with the Controller, bidirectionally). 

The result of this integration is the ability to build and manage a complete infrastructure that spans across physical, virtual, and container-based environments.
Cisco plans to release support for ACI with OpFlex, OVS, and containers before the end of 2015.

Value added from Cisco ACI to containers

I will explain how ACI supports the main two networking types in Docker: veth and macvlan.
This can be done already, because it's not based only on Opflex. 

Containers Networking option 1 - veth

vEth is the networking mode that leads to virtual bridging with br0 (a linux bridge, the default option with Docker) or OVS (Open Virtual Switch, usually adopted with KVM and Openstack).
As a premise, I remind you that ACI manages the connectivity and the policies for bare metal servers, VMs on any hypervisor and network services (LB, FW, etc.) consistently and easily:

On top of that, you can add containers running on bare metal Linux servers or inside virtual machines (different use cases make one of the options preferred, but from a ACI standpoint it's the same):

 

That means that applications (and the policies enabling them) can span across any platform: servers, VM, containers at the same time. Every service or component that makes up the application can be deployed on the platform that is more convenient for it in terms of scalability, reliability and management:

 

And the extension of the ACI fabric to virtual networks (with the Opflex-enabled OVS switch) allows applying the policies to any virtual End Point that uses virtual ethernet, like Docker containers configured with the veth mode.

Advantages from ACI with Docker veth:

With this architecture we can get two main results:
- Consistency of connectivity and services policy between physical, virtual and/or container (LXC and Docker);
- Abstraction of the end-to-end network policy for location independence altogether with Docker portability (via shared repositories) 

Containers Networking option 2 - macvlan

MACVLAN does not bring a network bridge for the ethernet side of a Docker container to connect.
You can think  MACVLAN as a hypotetical cable where one side is the eth0 at the Docker and the other side is the interface on the physical switch / ACI leaf.
The hook between them is the VLAN (or the trunked VLAN) in between.
In short, when specifying a VLAN with the MACVLAN, you tell a container binding on eth0 on Linux to use the VLAN XX (defined as access or trunked).
The connectivity will be “done” when the match happens with the other side of the cable at the VLAN XX on the switch (access or trunk).

At this point you can match vlans with EPG (End Point Groups) in ACI, to build policies that group containers as End Points needing the same treatment, i.e. applying Contracts to the groups of Containers:

Advantages from ACI with Docker macvlan:

This configuration provides two advantages (the first one is common to veth):
- Extend the Docker containers based portability for applications through the independence of ACI policies from the server's location.
- Performance increase on network throughput from 5% to 15% (mileage varies, further tuning and tests will provide more detail) because there’s no virtual switching consuming CPU cycles on the host.

Intent based approach

A new intent based approach is making its way in networking. An intent based interface enables a controller to manage and direct network services and network resources based on describing the “Intent” for network behaviors. Intents are described to the controller through a generalized and abstracted policy semantics, instead of using Openflow-like flow rules. The Intent based interface  allows for a descriptive way to get what is desired from the infrastructure, unlike the current SDN interfaces which are based on describing how to provide different services. This interface will accommodate orchestration services and network and business oriented SDN applications, including OpenStack Neutron, Service Function Chaining, and Group Based Policy.

Docker plugins for networks and volumes

Cisco is working at a open source project that aims at enabling intent-based configuration for both networking and volumes. This will exploit the full ACI potential in terms of defining the system behavior via policies, but will work also with non-ACI solutions. 
Contiv netplugin is a generic network plugin for Docker, designed to handle networking use cases in clustered multi-host systems.
It's still work in progress, detail can't be shared at this time but... stay tuned to see how Cisco is leading also in the open source world.

Mantl: a Stack to manage Microservices  

And, just for you to know, another project that Cisco is delivering is targeted at the lifecycle and the orchestration of microservices.
Mantl has been developed in house, as a framework to manage the cloud services offered by Cisco. It can be used by everyone for free under the Apache License.
You can download Mantl from github and see the documentation here.

Mantl allows teams to run their services on any cloud provider. This includes bare metal services, OpenStack, AWS, Vagrant and GCE. Mantl uses tools that are industry-standard in the DevOp community, including Marathon, Mesos, Kubernetes, Docker, Vault and more.
Each layer of Mantl’s stack allows for a unified, cohesive pipeline between support, managing Mesos or Kubernetes clusters during a peak workload, or starting new VMs with Terraform. Whether you are scaling up by adding new VMs in preparation for launch, or deploying multiple nodes on a cluster, Mantl allows for you to work with every piece of your DevOps stack in a central location, without backtracking to debug or recompile code to ensure that the microservices you need will function when you need them to.

When working in a container-based DevOps environment, having the right microservices can be the difference between success and failure on a daily basis. Through the Marathon UI, one can stop and restart clusters, kill sick nodes, and manage scaling during peak hours. With Mantl, adding more VMs for QA testing or live use is simple with Terraform — without one needing to piece together code to ensure that both pieces work well together without errors. Addressing microservice conflicts can severely impact productivity. Mantl cuts down time spent working through conflicts with microservices so DevOps can spend more time working on an application.

Key takeaways

ACI offers a seamless policy framework for application connectivity for VM, physical hosts and containers.

ACI integrates Docker without requiring gateways (otherwise required if you build the overlay from within the host) so Virtual and Physical can be merged in the deployment of a single application.

Intent based configuration makes networking easier. Plugins for enabling Docker to intent based configuration and integration with SDN solutions are coming fast.

Microservices are a key component of cloud native applications. Their lifecycle can be complicated, but tools are emerging to orchestrate it end to end. Cisco Mantl is a complete solution for this need and is available for free on github.

References

Much of the information has been taken from the following sources.
You can refer to them for a deeper investigation of the subject:

https://docs.docker.com/userguide/
https://docs.docker.com/articles/security/
https://docs.docker.com/articles/networking/   
http://www.dedoimedo.com/computers/docker-networking.html
https://mesosphere.github.io/presentations/mug-ericsson-2014/  
Exploring Opportunities: Containers and OpenStack
ACI for Simple Minds
http://www.networkworld.com/article/2981630/data-center/containers-key-as-cisco-looks-to-open-data-center-os.html
http://blogs.cisco.com/datacenter/docker-and-the-rise-of-microservices    
ACI and Containers white paper 
Cisco and Red Hat white paper   
Opendaylight and intent
Intent As The Common Interface to Network Resources
Mantl Introduces Microservices as a Stack
Project mantl

DevOps, Docker and Cisco ACI - part 1

In this post I try to describe the connection between the need for a fast IT, the usage of Linux containers to quicky deploy cloud native applications and the advantage provided by Cisco ACI in containers' networking.
The discussion is split in two posts to make it more... agile.
A big thank you to Carlos Pereira (@capereir), Frank Brockners (@brockners) and Juan Lage (@JuanLage) that provided content and advice on this subject. 

DevOps – it’s not tooling, it’s a process optimization

I will not define DevOps again, you can find it in this post and in this book.

I just want to remind that it’s not a product or a technology, but it’s a way of doing things.

Its goal is to bring fences down between the software development teams and the operations team, streamlining the flow of a IT project from development to production.

Steps are:

1. alleviate bottlenecks (systems or people) and automate as much as possible, 
2. feed information back so problems are solved by desing in next iteration, 
3. iterate as often as possible (continuous delivery).

Business owners push the IT to deliver faster, and application development via DevOps is changing the behavior of IT.

Gartner defined the Bimodal IT as the parallel management of cloud native applications (DevOps) and more mature systems that require consolidated best practices (like ITIL) and tools supporting their lifecycle.

One important aspect of DevOps is that the infrastructure must be flexible and provisioned on demand (and disposed when no longer needed). So, if it is programmable it fits much better in this vision. 

 

Infrastructure as code

Infrastructure as code is one of the mantra of DevOps: you can save the definition of the infrastructure (and the policies that define its behavior) in source code repository, as well as you do with the code for your applications.

In this way you can automate the build and the management very easily.

There are a number of tools supporting this operational model. Some examples:

One more example of tool for DevOps is the ACI toolkit, a set of python libraries that expose the ACI network fabric to DevOps as a code library.

 

You can download it from:

https://github.com/datacenter/acitoolkit

http://datacenter.github.io/acitoolkit/

The ACI Toolkit exposes the ACI object model to programming languages so that you can create, modify and manage the fabric as needed.

Remember that one of the most important advantage of Cisco’s vision of SDN is that you can manage the entire system as a whole.

No need to configure or manage single devices one by one, like other approaches to SDN (e.g. Openflow).

So you can create, modify and delete all of the following objects and their relationships:

Linux Containers and Docker

Docker is an open platform for Sys Admins and developers to build, ship and run distributed applications.  Applications are easy and quickly assembled from reusable and portable components, eliminating the silo-ed approach between development, QA, and production environments.

Individual components can be microservices coordinated by a program that contains the business process logic (an evolution of SOA, or Service Oriented Architecture). They can be deployed independently and scaled horizontally as needed, so the project benefits from flexibility and efficient operations. This is of great help in DevOps.

At a high-level, Docker is built of:
- Docker Engine: a portable and lightweight, runtime and packaging tool
- Docker Hub: a cloud service for sharing applications and automating workflows
There are more components (Machine, Swarm) but that's beyond the basic overview I'm giving here.

Docker’s main purpose is the lightweight packaging and deployment of applications.   

Containers are lightweight, portable, isolated, self-sufficient "slices of a server" that contain any application (often they contain microservices).
They deliver on full DevOps goal:
- Build once… run anywhere (Dev, QA, Prod, DR).
- Configure once… run anything (any container).  

Processes in a container are isolated from processes running on the host OS or in other Docker containers.
All processes share the same Linux kernel.
Docker leverages Linux containers to provide separate namespaces for containers, a technology that has been present in Linux kernels for 5+ years. The default container format is called libcontainer. Docker also supports traditional Linux containers using LXC.
It also uses Control Groups (cgroups), which have been in the Linux kernel even longer, to implement resources (such as CPU, memory, I/O) auditing and limiting, and Union file systems that support layering of the container's file system.

 

Kernel namespaces isolate containers, avoiding visibility between containers and containing faults.   Namespaces isolate:
◦     pid (processes)
◦     net (network interfaces, routing)
◦     ipc (System V interprocess communication [IPC])
◦     mnt (mount points, file systems)
◦     uts (host name)
◦     user (user IDs [UIDs])    

Containers or Virtual Machines

Containers are isolated, portable environments where you can run applications along with all the libraries and dependencies they need.
Containers aren’t virtual machines. In some ways they are similar, but there are even more ways that they are different. Like virtual machines, containers share system resources for access to compute, networking, and storage. They are different because all containers on the same host share the same OS kernel, and keep applications, runtimes, and various other services separated from each other using kernel features known as namespaces and cgroups.
Not having a separate instance of a Guest OS for each VM saves space on disk and memory at runtime, improving also the performances.
Docker added the concept of a container image, which allows containers to be used on any host with a modern Linux kernel. Soon Windows applications will enjoy the same portability among Windows hosts as well.
The container image allows for much more rapid deployment of applications than if they were packaged in a virtual machine image.

Containers networking

When Docker starts, it creates a virtual interface named docker0 on the host machine.
docker0 is a virtual Ethernet bridge that automatically forwards packets between any other network interfaces that are attached to it.
For every new container, Docker creates a pair of “peer” interfaces: one “local” eth0 interface and one unique name (e.g.: vethAQI2QT), out in the namespace of the host machine.
Traffic going outside is NATted

You can create different types of networks in Docker:

veth: a peer network device is created with one side assigned to the container and the other side is attached to a bridge specified by the lxc.network.link.   

vlan: a vlan interface is linked with the interface specified by the lxc.network.link and assigned to the container.

phys:  an already existing interface specified by the lxc.network.link is assigned to the container.

empty: will create only the loopback interface (at kernel space).

macvlan:  a  macvlan interface is linked with the interface specified by the lxc.network.link and assigned to the container.  It also specifies the mode the macvlan will use to communicate between  different macvlan on the same upper device.  The accepted modes are: private, Virtual Ethernet Port Aggregator (VEPA) and bridge

Docker Evolution - release 1.7, June 2015  

Important innovation has been introduced in the latest release of Docker, that is still experimental.

Plugins  

A big new feature is a plugin system for Engine, the first two available are for networking and volumes. This gives you the flexibility to back them with any third-party system.
For networks, this means you can seamlessly connect containers to networking systems such as Weave, Microsoft, VMware, Cisco, Nuage Networks, Midokura and Project Calico.  For volumes, this means that volumes can be stored on networked storage systems such as Flocker.

Networking  

The  release includes a huge update to how networking is done.

Libnetwork provides a native Go implementation for connecting containers.  The goal of libnetwork is to deliver a robust Container Network Model that provides a consistent programming interface and the required network abstractions for applications.
NOTE: libnetwork project is under heavy development and is not ready for general use.
There are many networking solutions available to suit a broad range of use-cases. libnetwork uses a driver / plugin model to support all of these solutions while abstracting the complexity of the driver implementations by exposing a simple and consistent Network Model to users.

Containers can now communicate across different hosts (Overlay Driver).  You can now create a network and attach containers to it.

Example:

docker network create -d overlay net1    

docker run -itd --publish-service=myapp.net1 debian:latest  

Orchestration and Clustering for containers  

Real world deployments are automated, single CLI commands are less used. Most important orchestrators are Mesos/Marathon, Google Kubernetes, Docker Swarm
Most use JSON or YAML formats to describe an application: a declarative language that says what an application looks like.
That is similar to ACI declarative language with high level abstraction to say what an application needs from the network, and have a network implement it.
This validates Cisco’s vision with ACI, very different from the NSX's of the world.

Next post explains the advantage provided by Cisco ACI (and some other projects in the open source space) when you use containers.

References

Much of the information has been taken from the following sources.
You can refer to them for a deeper investigation of the subject:

https://docs.docker.com/userguide/
https://docs.docker.com/articles/security/
https://docs.docker.com/articles/networking/   
http://www.dedoimedo.com/computers/docker-networking.html
https://mesosphere.github.io/presentations/mug-ericsson-2014/ 
http://blog.oddbit.com/2014/08/11/four-ways-to-connect-a-docker/
Exploring Opportunities: Containers and OpenStack
ACI for Simple Minds
http://www.networkworld.com/article/2981630/data-center/containers-key-as-cisco-looks-to-open-data-center-os.html
http://blogs.cisco.com/datacenter/docker-and-the-rise-of-microservices    
ACI and Containers white paper 
Cisco and Red Hat white paper    

Some content from the Docker documentation reused based on the Apache 2 License.

The Phoenix Project - how DevOps can change your life

It’s been a long time since I did my last post: as promised, I only post information from my experience in the real world and I avoid echoing messages from marketing   :-)

I’ve not been at rest, though, but I’ve worked at customer projects that can’t be mentioned publicly (yet).

But I’ve also been in vacation and I could finally read a great book, “The Phoenix Project”. 

It is a novel and a very educational reading at the same time.

I wholehearted recommend you to read it (though I’m not earning anything from the book) because I enjoied it a lot and I learned important lessons that deserve to be spread - for our common benefit as IT community.

You are not required to be a IT professional but, if you are, you will benefit the most and it will recall many familiar stories.

Since I’ve led some mission critical projects, and my skin is still impressed with both tragedy and triumph, this story reminded me those great moments. 

If you are new to DevOps, you can read my introductory posts in this blog.

Essentially, The Phoenix Project describes the evolution of IT in a company that, on the verge of a complete failure, pioneers DevOps and revolutionizes the way they work.

The impact on the core business is huge and their strategy creates a gap with the competition thanks to agility and flexibility.

Also personal lives are affected because the new organization ends the tribal war among Development, Operations, Security and the business stakeholders: they establish respect, trust and satisfaction for all the involved parties.

Of course the DevOps methodology is not a magic wand that makes the miracle for them: it is the outcome of a new way of thinking and working together.

This is a story of people, rather that technology.

If every IT department put themselves in the shoes of the others, instead of finger pointing, they can help each other to reach a common goal.

If the whole IT is not a counterpart of the LOBs but is a partner (understanding why they are asked something instead of focusing on how to do it), they can offer a huge value to the company… and be highly rewarded (see the coup de théâtre at the end of the story).

This would stop the “dysfunctional marriage” between two parties that don’t understand each other and suffer from a forced relationship.

In my experience, most of the business people see the IT as the provider of a services that is never satisfactory.

On the other side, IT sees that business people don’t understand the complexity and the effort required and ask for impossible things.

In most cases, they are bound to a traditional way of working and don’t even raise their head to see that they already own what’s needed to win.

They are overwhelmed by current tasks, troubleshooting and budget cuts, so they can’t think strategically.

The great idea, here, is importing the concepts and the experience from Lean Manufacturing into IT.

They start considering the IT organization similar to a production plant and optimizing its organization.

Finding bottlenecks and avoiding rework are the first steps, then automation follows to free the smart guys from the routine work and so the quality skyrockets.

At the end of the story the release of new features required by the business no longer takes months (and high risk at the roll out) but they can deploy 10 project builds per day!

That is not impressing if you think that these days some companies achieve 1000s of deployments per day thanks to Continuos Integration and Continuous Deployment.

But it is light years ahead of what most of my customers are doing, though some are exploring DevOps now.

Of course, one organization cannot change overnight.

You shouldn’t see the adoption of DevOps as a single step, and be scared by the effort.

In the book, they learn gradually and improve accordingly: you could do the same.

They go through a process that is made of Three Ways, until they master all.
A brief description of the three ways follows, thanks to Richard Campbell:

The First Way – Systems Thinking

• Understand the entire flow of work

• Seek to increase the flow of work

• Stop problems early and often – Don’t let them flow downstream

• Keep everyone thinking globally

• Deeply understand your systems

First Way Goals

• One source of truth – Code, environment and configuration in one place

• Consistent release process – Automation is essential (one click)

• Decrease cycle times, Faster release cadence

The Second Way – Feedback Loops

• Understand and respond to the needs of all customers (internal and external)

• Shorten and amplify all feedback loops

• With feedback comes quality

Second Way Goals

• Defects and performance issues fixed faster

• Ops and InfoSec user stories appear as part of the application

• Everyone is communicating better

• More work getting done

The Third Way – Synergy

• Consistent process and effective feedback result in agility

• Now use that agility to experiment

• You only learn from failure – So fail often, but recover quickly

Third Way Goals

• Ability to anticipate, even define new business needs through visibility in the systems

• Ability to test and optimize new business opportunities in the system while managing risk

• Joy

You should not think that The Phoenix Project is a technical book: though I’ve learned new things or reinforced concepts I knew already, the value I found in it is motivational.

It really moves you to action, and you want to measure the immediate improvement you can get.

More, you want to partner with other stakeholders to achieve common goals.

The Essence of DevOps

• Better Software, Faster

• Pride in the Software You Build and Operate

• Ability to Identify, Respond and Improve Business Needs

My final take from this story is that everybody in the IT (like in other fields) should:

- take risk and innovate - if you fail, probably the result would not be worse than staying still

- invest time - at cost of delaying important targets - to think strategically: the return will overpay the effort

- study what others have done already: learning by examples is much easier

- always try to understand your counterpart before fighting by principle, there could be a common advantage if you shift your perspective

Some useful references:

Amazon - http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592
R. Campbell's presentation -  http://techdays.org/wp-content/uploads/2013/05/Essence-of-DevOps.pptx

Other DevOps books:
- Visible Ops Handbook (Gene Kim)
- Web Operations (Allspaw/Robbins)
- Continuous Delivery (Humble/Farley)
- Lean Startup (Eric Reis)

A powerful DevOps tool: Ansible

Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications— automate in a language that approaches plain English, using SSH, with no agents to install on remote systems.

http://ansible.com/

At the Openstack Summit in Vancouver I attended a great session presented by two Cisco colleagues:

Juergen Brendel (@brendelconsult), David Lapsey (@devlaps) both from Cisco Metacloud.

These are my notes, that you could find useful as a easy introduction.

But I suggest you to watch the recording of their session at the end of this post, because it is very educational.

Configuration Management tools

They are better than scripts, that in turn are better than written manual instructions, that are better than a seasoned administrator's memory.

CM tools describe the desired state of a resource (i.e. a server) via assertions (ensure that… exists/installed/...): a declarative way to provision resources.
Comparison of existing tools:

puppet dates 2005, chef dates 2009 - they are powerful and rich

salt dates 2011, ansible dates 2012 - they are easy and quick

Ansible

It's written in Python, uses YAML to create Playbooks (description of the desired state)

It's simple: no central server to maintain, no keys management, NO AGENT on the managed servers - but requires ssh and python on the target server (powershell support is coming).

Ansible executes commands in explicit order (so there are no race conditions due to dependencies).

Modules

Modules are pieces of code that do a single thing.

There are hundreds of modules available to reuse.

They’re copied to the target server at runtime, executed there (they return results) and then deleted.

Inventory file

It defines hosts and groups them so that you can apply same commands to all at once.

Adhoc commands apply to groups - example: ansible -i hosts europe -a “uname -a", where europe is a group.

Playbooks

they are written in YAML and tell Ansible what to do (a sequence of tasks)

Projects layout

A Ansible project is made of:
config files

inventory files

group variables

yaml file

Roles

contains tasks, handlers, templates, files, vars

apply to servers (that have the same role)

can be included in playbooks

Usage of API

to manage infrastructure and services

there are modules available for public cloud and private cloud management systems

Vagrant

Vagrant is a tool that matches Ansible very well:
it is used to create VM in cloud

it can use Ansible as a provisioner

written in Ruby

commands:

vagrant up - creates the vm

vagrant provision - calls Ansible

Takeaways

A single Ansible playbook can be used to deploy apps locally and in the cloud

Download Ansible for free from Github.

Openstack and Cisco

Cisco is investing a lot in Openstack, as other vendors do these days.
Initiatives include being a Gold member of the Openstack Foundation, being in the board of directors, contribute to different projects in Openstack (mainly Neutron, that manages networking, but also Nova and Ironic) with blueprints and code development.

Cisco also uses Openstack in his own data centers, to provide cloud services to the internal IT (our private cloud) and to customers and partners (the Cisco Cloud Services in the Intercloud ecosystem). We also have a managed private cloud offer based on Openstack (formerly named Metacloud).


Based on this experience, a CVD (Cisco Validated Design) has been published to allow customers to deploy the Openstack platform on the Cisco servers and network. The prescriptive documentation guides you to install and configure the hardware and the software in such a way that you get the expected results in terms of scale and security. It's been fully tested and validated in partnership with Red Hat.

Another important point is the offer of the Cisco ACI data model to the open source community. The adoption of such a model in Openstack (the GBP, i.e. the Group Based Policy) is a great satisfaction for us.

Openstack will also be managed by the Stack Designer in Cisco Prime Service Catalog (PSC 11.0), to create PaaS services based on Heat (similarly to what we do now with Stack Designer + UCS Director). Templates to deploy a given Data Center topology will be added as services in the catalog and, based on them, other services could be offered with the deployment of a software stack on top of the Openstack IaaS. The user will be able to order, in a single request, the end to end deployment of a new application.

 

In this post I will tell you about the main topics in the Cisco-Openstack relationship:

1 - Available Plugins for Cisco products (Nexus switches, UCS servers, ACI, CSR, ASR)
2 - GBP: Group Based Policy (the ACI model adopted by the Openstack community)

Available Plugins for Cisco products

Plugins exist for the following projects in Openstack: Neutron, Nova, Ironic.

You can leverage the features of the Cisco products while you maintain the usual operations with Openstack: the integration of the underlying infrastructure is transparent for the user.

 

Networking - Project Neutron

Plugins for all the Nexus switching family      
 - Tenant network creation is based on VLAN or VXLAN
Plugins for ACI      
 - Neutron Networks and Routers are created as usual and the plugin has the role to integrate the API exposed by the Cisco APIC controller

A number of Neutron plugins are available already: Nexus 1000v, 3000, 5000, 6000, 7000 and 9000 Series Switches are supported (see http://www.cisco.com/c/en/us/products/collateral/switches/nexus-3000-series-switches/data_sheet_c78-727737.html).

You can also scale the OpenStack L3 services using the Cisco ASR1K platform (see http://blogs.cisco.com/datacenter/scaling-openstack-l3-using-cisco-asr1k-platform#more-163906) and use the Cloud Services Router (CSR) for Openstack VPN as a Service (see Neutron blueprints web site for Kilo and http://specs.openstack.org/openstack/neutron-specs/specs/kilo/cisco-vpnaas-and-router-integration.html).

Network Service Plug-in Architecture (ML2)

This pluggable architecture has been designed to allow for common API, rapid innovation and vendor differentiation:

Based on the delegation of the real networking service to the underlying infrastructure, the Openstack user does not care what networking devices are used: he only knows what service he needs, and he gets exactly that.

Use the existing Neutron API with APIC and Cisco ACI   

When the Openstack user creates the usual constructs (Networks, Subnets, Routers) via Horizon or the Neutron API, the APIC ML2 plugin intercepts the request and send commands to the APIC API.
Network profiles, made of End Point Groups and Contracts, are created and pushed to the fabric. Virtual networks created in the OVS virtual switch in KVM are matched to the networks in the physical fabric, so that traffic can flow to and from the external world.

Another plugin is the one for the Cisco UCS servers, leveraging the UCS Manager API.
This integration allows you to leverage the single point of management of a UCS domain (up to 160 servers) instead of configuring networking on the single blades or - as in competing server architectures - on the individual switches in the chassis.

An additional advantage offered by UCS servers is the VM-FEX (VM fabric extender) feature: virtual NICs can be offered to the VM directly from the hw, bypassing the virtual switch in the hypervisor thanks to SR-IOV and gaining performances and centralization of the management. 

Next picture shows the automated VLAN and VM-FEX Support offered by the Cisco UCS Manager plugin for OpenStack Neutron:

Bare metal deployment - Project Ironic  

Plugin for UCS Manager to deploy Service Profiles for bare metal workloads on the UCS blades

Ironic is the OpenStack service which provides the capability to provision bare metal servers. The initial version of Ironic pxe_cisco driver adds support to manage power operations of Cisco UCS B/C series servers that are UCSM managed and provides vendor_passthru APIs.
User can control the power operations using pxe_cisco driver. This doesn’t require IPMI protocol to be enabled on the servers as the operations are controlled via Service Profiles.

The vendor_passthru APIs allows the user to enroll the nodes automatically to Ironic DB. Also provides APIs to get the Node specific information like, Inventory, Faults, Location, Firmware Version etc.
Code is available in GitHub @ https://github.com/CiscoUcs/Ironic-UCS

GBP: Group Based Policy

The most exciting news is the adoption of the GBP (Group Based Policy) model and API in Neutron, that derives from the way the Cisco APIC controller manages end point groups and contracts in the ACI architecture. A powerful demonstration of the Cisco thought leadership in networking.

The Group Based Policy (GBP) extension introduces a declarative policy driven framework for networking in OpenStack. The GBP abstractions allow application administrators to express their networking requirements using group and policy abstractions, with the specifics of policy enforcement and implementation left to the underlying policy driver. This facilitates clear separation of concerns between the application and the infrastructure administrator.

Two Options for the OpenStack Neutron API

The Neutron user can now select the preferred option between two choices: the usual building blocks in Neutron (Network, Subnet, Router) and the new - optional - building blocks offered by GBP.

 

In addition to support for the OpenStack Neutron Modular Layer 2 (ML2) interface, Cisco APIC supports integration with OpenStack using Group-Based Policy (GBP). GBP was created by OpenStack developers to offer declarative abstractions for achieving scalable, intent-based infrastructure automation within OpenStack. It supports a plug-in architecture connecting its policy API to a broad range of open source and vendor solutions, including APIC.
This means that other vendors could provide plugins for their infrastructure, to use with the GBP API.
While GBP is a northbound API for Openstack, the plugins are a southbound implementation.

In this case the Neutron plugin for the APIC controller has a easier task: instead of translating from the legacy constructs (Networks, Subnets, Routers) to the corresponding ACI constructs (EPG, Contracts), it will just resend (proxy) identical commands to APIC.

Read more about group-based policy at https://wiki.openstack.org/wiki/GroupBasedPolicy and the Cisco Application Policy Infrastructure Controller Driver for OpenStack Group-Based Policy Data Sheet

In few days, at the Openstack Summit in Vancouver, we'll see all the latest news about the Cisco contribution to Openstack. Don't miss it!

[Added on June 14, 2016]
You can read how easy is to start with Openstack in Why don't you try Openstack (without getting your hands dirty)?

Useful Links:

http://www.cisco.com/c/en/us/solutions/data-center-virtualization/openstack-at-cisco/index.html 
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-733126.pdf
http://specs.openstack.org/openstack/neutron-specs/specs/kilo/cisco-vpnaas-and-router-integration.html

GBP
https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/group-based-policy-extension-for-networking
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/openstack-at-cisco/datasheet-c78-734181.html
https://www.rdoproject.org/Neutron_GBP