DevOps, Docker and Cisco ACI - part 2

This post is a follow up to the initial discussion of the DevOps approach based on Linux containers (specifically with Docker).
Here I elaborate on the advantage provided by Cisco ACI (and some more projects in the open source space) when you work with containers.

Policies and Containers  

Cisco ACI offers a common policy model for managing IT operations.
It is agnostic: bare metal, virtual machines, and containers are treated the same, offering a unified policy language: one clear security model, regardless of how an application is deployed.

ACI models how components of an application interact, including their requirements for connectivity, security, quality of service (e.g. reserved bandwidth for a specific service), and network services.   ACI offers a clear path to migrate existing workloads towards container-based environments without any changes to the network policy, thanks to two main technologies:

• ACI Policy Model and OpFlex   
• Open vSwitch (OVS)   

OpFlex is a distributed policy protocol that allows application-centric policies to be enforced within a virtual switch such as OVS.
Each container can attach to an OVS bridge, just as a virtual machine would, and the OpFlex agent helps ensure that the appropriate policy is established within OVS (because it's able to communicate with the Controller, bidirectionally). 

The result of this integration is the ability to build and manage a complete infrastructure that spans across physical, virtual, and container-based environments.
Cisco plans to release support for ACI with OpFlex, OVS, and containers before the end of 2015.

Value added from Cisco ACI to containers

I will explain how ACI supports the main two networking types in Docker: veth and macvlan.
This can be done already, because it's not based only on Opflex. 

Containers Networking option 1 - veth

vEth is the networking mode that leads to virtual bridging with br0 (a linux bridge, the default option with Docker) or OVS (Open Virtual Switch, usually adopted with KVM and Openstack).
As a premise, I remind you that ACI manages the connectivity and the policies for bare metal servers, VMs on any hypervisor and network services (LB, FW, etc.) consistently and easily:

On top of that, you can add containers running on bare metal Linux servers or inside virtual machines (different use cases make one of the options preferred, but from a ACI standpoint it's the same):

 

That means that applications (and the policies enabling them) can span across any platform: servers, VM, containers at the same time. Every service or component that makes up the application can be deployed on the platform that is more convenient for it in terms of scalability, reliability and management:

 

And the extension of the ACI fabric to virtual networks (with the Opflex-enabled OVS switch) allows applying the policies to any virtual End Point that uses virtual ethernet, like Docker containers configured with the veth mode.

Advantages from ACI with Docker veth:

With this architecture we can get two main results:
- Consistency of connectivity and services policy between physical, virtual and/or container (LXC and Docker);
- Abstraction of the end-to-end network policy for location independence altogether with Docker portability (via shared repositories) 

Containers Networking option 2 - macvlan

MACVLAN does not bring a network bridge for the ethernet side of a Docker container to connect.
You can think  MACVLAN as a hypotetical cable where one side is the eth0 at the Docker and the other side is the interface on the physical switch / ACI leaf.
The hook between them is the VLAN (or the trunked VLAN) in between.
In short, when specifying a VLAN with the MACVLAN, you tell a container binding on eth0 on Linux to use the VLAN XX (defined as access or trunked).
The connectivity will be “done” when the match happens with the other side of the cable at the VLAN XX on the switch (access or trunk).

At this point you can match vlans with EPG (End Point Groups) in ACI, to build policies that group containers as End Points needing the same treatment, i.e. applying Contracts to the groups of Containers:

Advantages from ACI with Docker macvlan:

This configuration provides two advantages (the first one is common to veth):
- Extend the Docker containers based portability for applications through the independence of ACI policies from the server's location.
- Performance increase on network throughput from 5% to 15% (mileage varies, further tuning and tests will provide more detail) because there’s no virtual switching consuming CPU cycles on the host.

Intent based approach

A new intent based approach is making its way in networking. An intent based interface enables a controller to manage and direct network services and network resources based on describing the “Intent” for network behaviors. Intents are described to the controller through a generalized and abstracted policy semantics, instead of using Openflow-like flow rules. The Intent based interface  allows for a descriptive way to get what is desired from the infrastructure, unlike the current SDN interfaces which are based on describing how to provide different services. This interface will accommodate orchestration services and network and business oriented SDN applications, including OpenStack Neutron, Service Function Chaining, and Group Based Policy.

Docker plugins for networks and volumes

Cisco is working at a open source project that aims at enabling intent-based configuration for both networking and volumes. This will exploit the full ACI potential in terms of defining the system behavior via policies, but will work also with non-ACI solutions. 
Contiv netplugin is a generic network plugin for Docker, designed to handle networking use cases in clustered multi-host systems.
It's still work in progress, detail can't be shared at this time but... stay tuned to see how Cisco is leading also in the open source world.

Mantl: a Stack to manage Microservices  

And, just for you to know, another project that Cisco is delivering is targeted at the lifecycle and the orchestration of microservices.
Mantl has been developed in house, as a framework to manage the cloud services offered by Cisco. It can be used by everyone for free under the Apache License.
You can download Mantl from github and see the documentation here.

Mantl allows teams to run their services on any cloud provider. This includes bare metal services, OpenStack, AWS, Vagrant and GCE. Mantl uses tools that are industry-standard in the DevOp community, including Marathon, Mesos, Kubernetes, Docker, Vault and more.
Each layer of Mantl’s stack allows for a unified, cohesive pipeline between support, managing Mesos or Kubernetes clusters during a peak workload, or starting new VMs with Terraform. Whether you are scaling up by adding new VMs in preparation for launch, or deploying multiple nodes on a cluster, Mantl allows for you to work with every piece of your DevOps stack in a central location, without backtracking to debug or recompile code to ensure that the microservices you need will function when you need them to.

When working in a container-based DevOps environment, having the right microservices can be the difference between success and failure on a daily basis. Through the Marathon UI, one can stop and restart clusters, kill sick nodes, and manage scaling during peak hours. With Mantl, adding more VMs for QA testing or live use is simple with Terraform — without one needing to piece together code to ensure that both pieces work well together without errors. Addressing microservice conflicts can severely impact productivity. Mantl cuts down time spent working through conflicts with microservices so DevOps can spend more time working on an application.

Key takeaways

ACI offers a seamless policy framework for application connectivity for VM, physical hosts and containers.

ACI integrates Docker without requiring gateways (otherwise required if you build the overlay from within the host) so Virtual and Physical can be merged in the deployment of a single application.

Intent based configuration makes networking easier. Plugins for enabling Docker to intent based configuration and integration with SDN solutions are coming fast.

Microservices are a key component of cloud native applications. Their lifecycle can be complicated, but tools are emerging to orchestrate it end to end. Cisco Mantl is a complete solution for this need and is available for free on github.

References

Much of the information has been taken from the following sources.
You can refer to them for a deeper investigation of the subject:

https://docs.docker.com/userguide/
https://docs.docker.com/articles/security/
https://docs.docker.com/articles/networking/   
http://www.dedoimedo.com/computers/docker-networking.html
https://mesosphere.github.io/presentations/mug-ericsson-2014/  
Exploring Opportunities: Containers and OpenStack
ACI for Simple Minds
http://www.networkworld.com/article/2981630/data-center/containers-key-as-cisco-looks-to-open-data-center-os.html
http://blogs.cisco.com/datacenter/docker-and-the-rise-of-microservices    
ACI and Containers white paper 
Cisco and Red Hat white paper   
Opendaylight and intent
Intent As The Common Interface to Network Resources
Mantl Introduces Microservices as a Stack
Project mantl

DevOps, Docker and Cisco ACI - part 1

In this post I try to describe the connection between the need for a fast IT, the usage of Linux containers to quicky deploy cloud native applications and the advantage provided by Cisco ACI in containers' networking.
The discussion is split in two posts to make it more... agile.
A big thank you to Carlos Pereira (@capereir), Frank Brockners (@brockners) and Juan Lage (@JuanLage) that provided content and advice on this subject. 

DevOps – it’s not tooling, it’s a process optimization

I will not define DevOps again, you can find it in this post and in this book.

I just want to remind that it’s not a product or a technology, but it’s a way of doing things.

Its goal is to bring fences down between the software development teams and the operations team, streamlining the flow of a IT project from development to production.

Steps are:

1. alleviate bottlenecks (systems or people) and automate as much as possible, 
2. feed information back so problems are solved by desing in next iteration, 
3. iterate as often as possible (continuous delivery).

Business owners push the IT to deliver faster, and application development via DevOps is changing the behavior of IT.

Gartner defined the Bimodal IT as the parallel management of cloud native applications (DevOps) and more mature systems that require consolidated best practices (like ITIL) and tools supporting their lifecycle.

One important aspect of DevOps is that the infrastructure must be flexible and provisioned on demand (and disposed when no longer needed). So, if it is programmable it fits much better in this vision. 

 

Infrastructure as code

Infrastructure as code is one of the mantra of DevOps: you can save the definition of the infrastructure (and the policies that define its behavior) in source code repository, as well as you do with the code for your applications.

In this way you can automate the build and the management very easily.

There are a number of tools supporting this operational model. Some examples:

One more example of tool for DevOps is the ACI toolkit, a set of python libraries that expose the ACI network fabric to DevOps as a code library.

 

You can download it from:

https://github.com/datacenter/acitoolkit

http://datacenter.github.io/acitoolkit/

The ACI Toolkit exposes the ACI object model to programming languages so that you can create, modify and manage the fabric as needed.

Remember that one of the most important advantage of Cisco’s vision of SDN is that you can manage the entire system as a whole.

No need to configure or manage single devices one by one, like other approaches to SDN (e.g. Openflow).

So you can create, modify and delete all of the following objects and their relationships:

Linux Containers and Docker

Docker is an open platform for Sys Admins and developers to build, ship and run distributed applications.  Applications are easy and quickly assembled from reusable and portable components, eliminating the silo-ed approach between development, QA, and production environments.

Individual components can be microservices coordinated by a program that contains the business process logic (an evolution of SOA, or Service Oriented Architecture). They can be deployed independently and scaled horizontally as needed, so the project benefits from flexibility and efficient operations. This is of great help in DevOps.

At a high-level, Docker is built of:
- Docker Engine: a portable and lightweight, runtime and packaging tool
- Docker Hub: a cloud service for sharing applications and automating workflows
There are more components (Machine, Swarm) but that's beyond the basic overview I'm giving here.

Docker’s main purpose is the lightweight packaging and deployment of applications.   

Containers are lightweight, portable, isolated, self-sufficient "slices of a server" that contain any application (often they contain microservices).
They deliver on full DevOps goal:
- Build once… run anywhere (Dev, QA, Prod, DR).
- Configure once… run anything (any container).  

Processes in a container are isolated from processes running on the host OS or in other Docker containers.
All processes share the same Linux kernel.
Docker leverages Linux containers to provide separate namespaces for containers, a technology that has been present in Linux kernels for 5+ years. The default container format is called libcontainer. Docker also supports traditional Linux containers using LXC.
It also uses Control Groups (cgroups), which have been in the Linux kernel even longer, to implement resources (such as CPU, memory, I/O) auditing and limiting, and Union file systems that support layering of the container's file system.

 

Kernel namespaces isolate containers, avoiding visibility between containers and containing faults.   Namespaces isolate:
◦     pid (processes)
◦     net (network interfaces, routing)
◦     ipc (System V interprocess communication [IPC])
◦     mnt (mount points, file systems)
◦     uts (host name)
◦     user (user IDs [UIDs])    

Containers or Virtual Machines

Containers are isolated, portable environments where you can run applications along with all the libraries and dependencies they need.
Containers aren’t virtual machines. In some ways they are similar, but there are even more ways that they are different. Like virtual machines, containers share system resources for access to compute, networking, and storage. They are different because all containers on the same host share the same OS kernel, and keep applications, runtimes, and various other services separated from each other using kernel features known as namespaces and cgroups.
Not having a separate instance of a Guest OS for each VM saves space on disk and memory at runtime, improving also the performances.
Docker added the concept of a container image, which allows containers to be used on any host with a modern Linux kernel. Soon Windows applications will enjoy the same portability among Windows hosts as well.
The container image allows for much more rapid deployment of applications than if they were packaged in a virtual machine image.

Containers networking

When Docker starts, it creates a virtual interface named docker0 on the host machine.
docker0 is a virtual Ethernet bridge that automatically forwards packets between any other network interfaces that are attached to it.
For every new container, Docker creates a pair of “peer” interfaces: one “local” eth0 interface and one unique name (e.g.: vethAQI2QT), out in the namespace of the host machine.
Traffic going outside is NATted

You can create different types of networks in Docker:

veth: a peer network device is created with one side assigned to the container and the other side is attached to a bridge specified by the lxc.network.link.   

vlan: a vlan interface is linked with the interface specified by the lxc.network.link and assigned to the container.

phys:  an already existing interface specified by the lxc.network.link is assigned to the container.

empty: will create only the loopback interface (at kernel space).

macvlan:  a  macvlan interface is linked with the interface specified by the lxc.network.link and assigned to the container.  It also specifies the mode the macvlan will use to communicate between  different macvlan on the same upper device.  The accepted modes are: private, Virtual Ethernet Port Aggregator (VEPA) and bridge

Docker Evolution - release 1.7, June 2015  

Important innovation has been introduced in the latest release of Docker, that is still experimental.

Plugins  

A big new feature is a plugin system for Engine, the first two available are for networking and volumes. This gives you the flexibility to back them with any third-party system.
For networks, this means you can seamlessly connect containers to networking systems such as Weave, Microsoft, VMware, Cisco, Nuage Networks, Midokura and Project Calico.  For volumes, this means that volumes can be stored on networked storage systems such as Flocker.

Networking  

The  release includes a huge update to how networking is done.

Libnetwork provides a native Go implementation for connecting containers.  The goal of libnetwork is to deliver a robust Container Network Model that provides a consistent programming interface and the required network abstractions for applications.
NOTE: libnetwork project is under heavy development and is not ready for general use.
There are many networking solutions available to suit a broad range of use-cases. libnetwork uses a driver / plugin model to support all of these solutions while abstracting the complexity of the driver implementations by exposing a simple and consistent Network Model to users.

Containers can now communicate across different hosts (Overlay Driver).  You can now create a network and attach containers to it.

Example:

docker network create -d overlay net1    

docker run -itd --publish-service=myapp.net1 debian:latest  

Orchestration and Clustering for containers  

Real world deployments are automated, single CLI commands are less used. Most important orchestrators are Mesos/Marathon, Google Kubernetes, Docker Swarm
Most use JSON or YAML formats to describe an application: a declarative language that says what an application looks like.
That is similar to ACI declarative language with high level abstraction to say what an application needs from the network, and have a network implement it.
This validates Cisco’s vision with ACI, very different from the NSX's of the world.

Next post explains the advantage provided by Cisco ACI (and some other projects in the open source space) when you use containers.

References

Much of the information has been taken from the following sources.
You can refer to them for a deeper investigation of the subject:

https://docs.docker.com/userguide/
https://docs.docker.com/articles/security/
https://docs.docker.com/articles/networking/   
http://www.dedoimedo.com/computers/docker-networking.html
https://mesosphere.github.io/presentations/mug-ericsson-2014/ 
http://blog.oddbit.com/2014/08/11/four-ways-to-connect-a-docker/
Exploring Opportunities: Containers and OpenStack
ACI for Simple Minds
http://www.networkworld.com/article/2981630/data-center/containers-key-as-cisco-looks-to-open-data-center-os.html
http://blogs.cisco.com/datacenter/docker-and-the-rise-of-microservices    
ACI and Containers white paper 
Cisco and Red Hat white paper    

Some content from the Docker documentation reused based on the Apache 2 License.

Openstack and Cisco

Cisco is investing a lot in Openstack, as other vendors do these days.
Initiatives include being a Gold member of the Openstack Foundation, being in the board of directors, contribute to different projects in Openstack (mainly Neutron, that manages networking, but also Nova and Ironic) with blueprints and code development.

Cisco also uses Openstack in his own data centers, to provide cloud services to the internal IT (our private cloud) and to customers and partners (the Cisco Cloud Services in the Intercloud ecosystem). We also have a managed private cloud offer based on Openstack (formerly named Metacloud).


Based on this experience, a CVD (Cisco Validated Design) has been published to allow customers to deploy the Openstack platform on the Cisco servers and network. The prescriptive documentation guides you to install and configure the hardware and the software in such a way that you get the expected results in terms of scale and security. It's been fully tested and validated in partnership with Red Hat.

Another important point is the offer of the Cisco ACI data model to the open source community. The adoption of such a model in Openstack (the GBP, i.e. the Group Based Policy) is a great satisfaction for us.

Openstack will also be managed by the Stack Designer in Cisco Prime Service Catalog (PSC 11.0), to create PaaS services based on Heat (similarly to what we do now with Stack Designer + UCS Director). Templates to deploy a given Data Center topology will be added as services in the catalog and, based on them, other services could be offered with the deployment of a software stack on top of the Openstack IaaS. The user will be able to order, in a single request, the end to end deployment of a new application.

 

In this post I will tell you about the main topics in the Cisco-Openstack relationship:

1 - Available Plugins for Cisco products (Nexus switches, UCS servers, ACI, CSR, ASR)
2 - GBP: Group Based Policy (the ACI model adopted by the Openstack community)

Available Plugins for Cisco products

Plugins exist for the following projects in Openstack: Neutron, Nova, Ironic.

You can leverage the features of the Cisco products while you maintain the usual operations with Openstack: the integration of the underlying infrastructure is transparent for the user.

 

Networking - Project Neutron

Plugins for all the Nexus switching family      
 - Tenant network creation is based on VLAN or VXLAN
Plugins for ACI      
 - Neutron Networks and Routers are created as usual and the plugin has the role to integrate the API exposed by the Cisco APIC controller

A number of Neutron plugins are available already: Nexus 1000v, 3000, 5000, 6000, 7000 and 9000 Series Switches are supported (see http://www.cisco.com/c/en/us/products/collateral/switches/nexus-3000-series-switches/data_sheet_c78-727737.html).

You can also scale the OpenStack L3 services using the Cisco ASR1K platform (see http://blogs.cisco.com/datacenter/scaling-openstack-l3-using-cisco-asr1k-platform#more-163906) and use the Cloud Services Router (CSR) for Openstack VPN as a Service (see Neutron blueprints web site for Kilo and http://specs.openstack.org/openstack/neutron-specs/specs/kilo/cisco-vpnaas-and-router-integration.html).

Network Service Plug-in Architecture (ML2)

This pluggable architecture has been designed to allow for common API, rapid innovation and vendor differentiation:

Based on the delegation of the real networking service to the underlying infrastructure, the Openstack user does not care what networking devices are used: he only knows what service he needs, and he gets exactly that.

Use the existing Neutron API with APIC and Cisco ACI   

When the Openstack user creates the usual constructs (Networks, Subnets, Routers) via Horizon or the Neutron API, the APIC ML2 plugin intercepts the request and send commands to the APIC API.
Network profiles, made of End Point Groups and Contracts, are created and pushed to the fabric. Virtual networks created in the OVS virtual switch in KVM are matched to the networks in the physical fabric, so that traffic can flow to and from the external world.

Another plugin is the one for the Cisco UCS servers, leveraging the UCS Manager API.
This integration allows you to leverage the single point of management of a UCS domain (up to 160 servers) instead of configuring networking on the single blades or - as in competing server architectures - on the individual switches in the chassis.

An additional advantage offered by UCS servers is the VM-FEX (VM fabric extender) feature: virtual NICs can be offered to the VM directly from the hw, bypassing the virtual switch in the hypervisor thanks to SR-IOV and gaining performances and centralization of the management. 

Next picture shows the automated VLAN and VM-FEX Support offered by the Cisco UCS Manager plugin for OpenStack Neutron:

Bare metal deployment - Project Ironic  

Plugin for UCS Manager to deploy Service Profiles for bare metal workloads on the UCS blades

Ironic is the OpenStack service which provides the capability to provision bare metal servers. The initial version of Ironic pxe_cisco driver adds support to manage power operations of Cisco UCS B/C series servers that are UCSM managed and provides vendor_passthru APIs.
User can control the power operations using pxe_cisco driver. This doesn’t require IPMI protocol to be enabled on the servers as the operations are controlled via Service Profiles.

The vendor_passthru APIs allows the user to enroll the nodes automatically to Ironic DB. Also provides APIs to get the Node specific information like, Inventory, Faults, Location, Firmware Version etc.
Code is available in GitHub @ https://github.com/CiscoUcs/Ironic-UCS

GBP: Group Based Policy

The most exciting news is the adoption of the GBP (Group Based Policy) model and API in Neutron, that derives from the way the Cisco APIC controller manages end point groups and contracts in the ACI architecture. A powerful demonstration of the Cisco thought leadership in networking.

The Group Based Policy (GBP) extension introduces a declarative policy driven framework for networking in OpenStack. The GBP abstractions allow application administrators to express their networking requirements using group and policy abstractions, with the specifics of policy enforcement and implementation left to the underlying policy driver. This facilitates clear separation of concerns between the application and the infrastructure administrator.

Two Options for the OpenStack Neutron API

The Neutron user can now select the preferred option between two choices: the usual building blocks in Neutron (Network, Subnet, Router) and the new - optional - building blocks offered by GBP.

 

In addition to support for the OpenStack Neutron Modular Layer 2 (ML2) interface, Cisco APIC supports integration with OpenStack using Group-Based Policy (GBP). GBP was created by OpenStack developers to offer declarative abstractions for achieving scalable, intent-based infrastructure automation within OpenStack. It supports a plug-in architecture connecting its policy API to a broad range of open source and vendor solutions, including APIC.
This means that other vendors could provide plugins for their infrastructure, to use with the GBP API.
While GBP is a northbound API for Openstack, the plugins are a southbound implementation.

In this case the Neutron plugin for the APIC controller has a easier task: instead of translating from the legacy constructs (Networks, Subnets, Routers) to the corresponding ACI constructs (EPG, Contracts), it will just resend (proxy) identical commands to APIC.

Read more about group-based policy at https://wiki.openstack.org/wiki/GroupBasedPolicy and the Cisco Application Policy Infrastructure Controller Driver for OpenStack Group-Based Policy Data Sheet

In few days, at the Openstack Summit in Vancouver, we'll see all the latest news about the Cisco contribution to Openstack. Don't miss it!

[Added on June 14, 2016]
You can read how easy is to start with Openstack in Why don't you try Openstack (without getting your hands dirty)?

Useful Links:

http://www.cisco.com/c/en/us/solutions/data-center-virtualization/openstack-at-cisco/index.html 
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-733126.pdf
http://specs.openstack.org/openstack/neutron-specs/specs/kilo/cisco-vpnaas-and-router-integration.html

GBP
https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/group-based-policy-extension-for-networking
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/openstack-at-cisco/datasheet-c78-734181.html
https://www.rdoproject.org/Neutron_GBP

ACI for (Smarter) Simple Minds

In a previous post I tried to describe the new Cisco ACI architecture in simple terms, from a software designer standpoint.

My knowledge on networking is limited, compared to my colleagues at Cisco that hold CCIE certifications… I am a software guy the just understands the API   ;-)

Though, now I would like to share some more technical information with the same “not for specialists” language.

You can still go to the official documentation for the detail, or look at one of the brilliant demo recorded on YouTube.

These are the main points that I want to describe:

- You don’t program the single switches, but the entire fabric (via the sw controller)

- The fabric has all active links (no spanning tree)

- Policies and performances benefit from a ASIC design that perfectly fits the SDN model

- You can manage the infrastructure as code (hence, really do DevOps)

- The APIC controller manages also L4-7 network services from 3rd parties

- Any orchestrator can drive the API of the controller

- The virtual leaf of the fabric extends into the hypervisor (AVS)
- You get immediate visibility of the Health Score for the Fabric, Tenants, Applications

Next picture shows how the fabric is build, using two types of switches: the Spines are used to scale and connect all the leaves in a non blocking fabric that ensures performances and reliability.
The Leaf switches hold the physical ports where servers are attached: both bare metal servers (i.e. running a Operating System) and virtualized servers (i.e. running ESXi, Hyper-V and KVM hypervisors).
The software controller for the fabric, named APIC, runs on a cluster of (at least) 3 dedicated physical servers and is not in the data path: so it does not affect performances and reliability of the fabric, as it could happen with other solutions on the market.

 

The ACI fabric supports more than 64,000 dedicated tenant networks. A single fabric can support more than one million IPv4/IPv6 endpoints, more than 64,000 tenants, and more than 200,000 10G ports. The ACI fabric enables any service (physical or virtual) anywhere with no need for additional software or hardware gateways to connect between the physical and virtual services and normalizes encapsulations for Virtual Extensible Local Area Network (VXLAN) / VLAN / Network Virtualization using Generic Routing Encapsulation (NVGRE).

The ACI fabric decouples the endpoint identity and associated policy from the underlying forwarding graph. It provides a distributed Layer 3 gateway that ensures optimal Layer 3 and Layer 2 forwarding. The fabric supports standard bridging and routing semantics without standard location constraints (any IP address anywhere), and removes flooding requirements for the IP control plane Address Resolution Protocol (ARP) / Generic Attribute Registration Protocol (GARP). All traffic within the fabric is encapsulated within VXLAN.

The ACI fabric decouples the tenant endpoint address, its identifier, from the location of the endpoint that is defined by its locator or VXLAN tunnel endpoint (VTEP) address. The following figure shows decoupled identity and location.

 

Forwarding within the fabric is between VTEPs. The mapping of the internal tenant MAC or IP address to a location is performed by the VTEP using a distributed mapping database. After a lookup is done, the VTEP sends the original data packet encapsulated in VXLAN with the Destination Address (DA) of the VTEP on the destination leaf. The packet is then de-encapsulated on the destination leaf and sent down to the receiving host. With this model, we can have a full mesh, loop-free topology without the need to use the spanning-tree protocol to prevent loops.

You can attach virtual servers or physical servers that use any network virtualization protocol to the Leaf ports, then design the policies that define the traffic flow among them regardless the local (to the server or to its hypervisor) encapsulation.
So the fabric acts as a normalizer for the encapsulation and allows you to match different environments in a single policy.

Forwarding is not limited to nor constrained by the encapsulation type or encapsulation-specific ‘overlay’ network:





As explained in ACI for Dummies, policies are based on the concept of EPG (End Points Group).
Special EPG represent the outside network (outside the fabric, that means other networks in your datacenter or eventually the Internet or a MPLS connection):

The integration with the hypervisors is made through a bidirectional connection between the APIC controller and the element manager of the virtualization platform (vCenter, System Center VMM, Red Hat EVM...). Their API are used to create local virtual networks that are connected and integrated with the ACI fabric, so that policies are propagated to them.
The ultimate result is the creation of Port Groups, or the like of, where VM can be connected.
A Port Groups represents a EPG.
Events generated by the VM lifecycle (power on/off, vmotion...) will be sent back to APIC so that the traffic is managed accordingly.

How Policies are enforced in the fabric

The policy contains a source EPG, a destination EPG and rules known as Contracts, made of Subjects (security, QoS...). They are created in the Controller and pushed to all the leaf switches where they are enforced.
When a packet arrives to a leaf, if the destination EPG is known it is processed locally.
Otherwise it is forwarded to a Spine, to reach the destination EPG through a Leaf that knows it.

There are 3 cases, and the local and global tables in the leaf are used based on the fact that the destination EP is known or not:
1 - If the target EP is known and it's local (local table) to the same leaf, it's processed locally (no traffic through the Spine).
2 - If the target EP is known and it's remote (global table) it's forwarded to the Spine to be sent to the destination VTEP, that is known.
3 - If the target EP is unknown the traffic is sent to the Spine for a proxy forwarding (that means that the Spine discovers what is the destination VTEP).

You can manage the infrastructure as code.

The fabric is stateless: this means that all the configuration/behavior can be pushed to the network through the controller's API. The definition of Contracts and EPG, of POD and Tenants, every Application Profile is a (set of) XML document that can be saved as text.
Hence you can save it in the same repository as the source code of your software applications.

You can extend the DevOps pipeline that builds the application, deploys it and tests it automatically by adding a build of the required infrastructure on demand.
This means that you can use a slice of a shared infrastructure to create a environment just when it's needed and destroy it soon after, returning the resources to the pool.

You can also use this approach for Disaster Recovery, simply building a clone of the main DC if it's lost.

Any orchestrator can drive the API of the controller.

The XML (or JSON) content that you send to build the environment and the policies is based on a standard language. The API are well documented and lot of samples are available.
You can practice with the API, learn how to use them with any REST client and then copy the same calls into your preferred orchestrator.
Though some products have out of the box native integration with APIC (Cisco UCSD, Microsoft), any other can be used easily with the approach I described above.
See an example in The Elastic Cloud Project.

The APIC controller manages also L4-7 network services from 3rd parties. 

The concept of Service Graph allows a automated and scalable L4-L7 service insertion.  The fabric forwards the traffic into a Service Graph, that can be one or more service nodes pre-defined in a series, based on a routing rule.  Using the service graph simplifies and scales service operation: the following pictures show the difference from a traditional management of the network services.

The same result can be achieved with the insertion of a Service Graph in the contract between two EPG:

The virtual leaf of the fabric extends into the hypervisor (AVS).

Compared to other hypervisor-based virtual switches, AVS provides cross-consistency in features, management, and control through Application Policy Infrastructure Controller (APIC), rather than through hypervisor-specific management stations. As a key component of the overall ACI framework, AVS allows for intelligent policy enforcement and optimal traffic steering for virtual applications.

The AVS offers:

• Single point of management and control for both physical and virtual workloads and infrastructure
• Optimal traffic steering to application services
• Seamless workload mobility
• Support for all leading hypervisors with a consistent operational model across implementations for simplified operations in heterogeneous data centers

Cisco AVS is compatible with any upstream physical access layer switch that complies with the Ethernet standard, including Cisco Nexus Family switches. Cisco AVS is compatible with any server hardware listed in the VMware Hardware Compatibility List (HCL). Cisco AVS is a distributed virtual switch solution that is fully integrated into the VMware virtual infrastructure, including VMware vCenter for the virtualization administrator. This solution allows the network administrator to configure virtual switches and port groups to establish a consistent data center network policy.

Next picture shows a topology that includes Cisco AVS with Cisco APIC and VMware vCenter with the Cisco Virtual Switch Update Manager (VSUM).

 

Health Score

The APIC uses a policy model to combine data into a health score. Health scores can be aggregated for a variety of areas such as for infrastructure, applications, or services.

The APIC supports the following health score types:

●      System—Summarizes the health of the entire network.

●      Leaf—Summarizes the health of leaf switches in the network. Leaf health includes hardware health of the switch including fan tray, power supply, and CPU.

●      Tenant—Summarizes the health of a tenant and the tenant’s applications.

Health scores allow you to isolate performance issues by drilling down through the network hierarchy to isolate faults to specific managed objects (MOs). You can view network health by viewing the health of an application (by tenant) or by the health of a leaf switch (by pod).

You can subscribe to a health score to receive notifications if the health score crosses a threshold value. You can receive health score events via SNMP, email, syslog, and Cisco Call Home.  This can be particularly useful for integration with 3rd party monitoring tools. 

Health Score Use case: 
An application administrator could subscribe to the health score of their application - and receive automatic notifications from ACI if the health of the specific application is degraded from an infrastructure point of view - truly an application-aware infrastructure.

Conclusion

I hope that these few lines were enough to show the advantage that modern network architectures can bring to your Data Center.
Cisco ACI joins all the benefit of the SDN and the overlay networks with a powerful integration with the hardware fabric, so you get flexibility without losing control, visibility and performances.

One of the most important aspects is the normalization of the encapsulation, so that you can merge different network technologies (from heterogeneous virtual environments and bare metal) into a single well managed policy model.

Policies (specifically, the Application Network Policies created in APIC based on EPG and Contracts) allow a easier communication between software application designers and infrastructure managers, because they are simple to represent, create/maintain and enforce.

Now all you need is just a look at ACI Fundamentals on the Cisco web site.

Software Defined Networking For Dummies

A very simple, yet complete description of what SDN is, now available as a free ebook that you can download from http://www.cisco.com/go/sdnfordummies

Software defined networking (SDN) is a new way of looking at how networking and cloud solutions should be automated, efficient, and scalable in a new world where application services may be provided locally, by the data center, or even the cloud. This is impossible with a rigid system that’s difficult to manage, maintain, and upgrade. Going forward, you need flexibility, simplicity, and the ability to quickly grow to meet changing IT and business needs.

Software Defined Networking For Dummies, Cisco Special Edition, shows you what SDN is, how it works, and how you can choose the right SDN solution. This book also helps you understand the terminology, jargon, and acronyms that are such a part of defining SDN.
Along the way, you’ll see some examples of the current state of the art in SDN technology and see how SDN can help your organization. 

You can find additional information about Cisco’s take on SDN by visiting:
✓ http://cisco.com/go/aci
✓ http://cisco.com/go/sdn
✓ http://blogs.cisco.com/tag/sdn

The Elastic Cloud project - Porting to UCSD

Porting to a new platform

This post shows how we did the porting of the Elastic Cloud project to a different platform.
The initial implementation was done on Cisco IAC (Intelligent Automation for Cloud) orchestrating Openstack, Cisco ACI (Application Centric Infrastructure) and 3 hypervisors.

Later we decided to implement the same use case (deploy a 3 tier application to 3 different hypervisors, using Openstack and ACI) with Cisco UCS Director, aka UCSD.

The objective was to offer another demonstration of flexibility and openness, targeting IT administrators rather than end users like we did in the first project.
You will find a brief description of UCS Director in the following paragraphs: essentially it is not used to abstract complexity, but to allow IT professionals to do their job faster and error-proof.
UCSD is also a key element in a new Cisco end-to-end architecture for cloud computing, named Cisco ONE Enterprise Cloud suite.

The implementation was supported by the Cisco dCloud team, the organization that provides excellent remote demo capabilities on a number of Cisco technologies. They offered me the lab environment to build the new demo and, in turn, the complete demo will be offered publicly as a self service environment on the dCloud platform.

The dCloud demo environment

Cisco dCloud provides Customers, Partners and Cisco Employees with a way to experience Cisco Solutions. From scripted, repeatable demos to fully customizable labs with complete administrative access, Cisco dCloud can work for you. Just login to dcloud.cisco.com with your Cisco account and you'll find all the available demo:

Cisco UCS Director

UCSD is a great tool for Data Center automation: it manages servers, network, storage and hypervisors, providing you a consistent view on physical and virtual resources in your DC.

Despite the name (that could associate it to Cisco UCS servers only) it integrates with a multi-vendor heterogeneous infrastructure, offering a single dashboard plus the automation engine (with a library containing 1300+ tasks) and the SDK to create your own adapters if needed.

UCSD offers open API so that you can run its workflows from the UCSD catalog or from a 3rd party tool (a portal, a orchestrator, a custom script).

There is a basic workflow editor, that we used to create the custom process integrating Openstack, ACI and all the hypervisors to implement our use case. We don't consider UCSD a full business level orchestrator because it's not meant to integrate also the BSS (Business Support Systems) in your company, but it does the automation of the DC infrastructure including Cisco and 3rd party technologies pretty well.

Implementing the service in UCS Director

Description of the process

The service consists in the deployment of the famous 3 tier application with a single click.
The first 2 tiers of the application (web and application servers and their networks) are deployed on Openstack. The first version of the demo uses KVM as the target hypervisor for both tiers, next version will replace one of the Openstack compute nodes with Hyper-V.
The 3rd tier (the database and its network) is deployed on ESXi.
On every hypervisor, virtual networks are created first. Then virtual machines are created and attached to the proper network.

To connect the virtual networks in their different virtualized environments we used Cisco ACI, creating policies through the API of the controller.
One End Point Group is created for each of the application tiers, Contracts are created to allow the traffic to flow from one tier to next one (and only there).
If you are not familiar with the ACI policy model, you can see my ACI for Dummies post.

All these operations are executed by a single workflow created in the UCSD automation engine.
We just dropped the tasks from the library to the workflow editor, provided input values for each task (from the output of previous tasks) and connected them in the right sequence drawing arrows.
The resulting workflow executes the same sequence of atomic actions that the administrator would do manually in the GUI, one by one.

The implementation was quite easy because we were porting an identical process created in Cisco IAC: the tool to implement the workflow is different, but the sequence and the content of the tasks is the same.

Integration out-of-the-box

Most of the tasks in our process are provided by the UCSD automation library: all the operations on ACI (through its APIC controller) and on ESXi VM and networks (through vCenter).

When you use these tasks, you can immediately see the effect in the target system.
As an example, this is the outcome of creating a Router in Openstack using UCSD: the two networks are connected in the hypervisor and the APIC plugin in Neutron talks immediately to Cisco ACI, creating the corresponding Contract between the two End Point Groups (please check the Router ID in Openstack and the Contract name in APIC).

 

Custom tasks

The integration with Openstack required us to build custom tasks, adding them to the library.
We created 15 new tasks, to call the API exposed by the Openstack subsystems: Neutron (to create the networks) and Nova (to create the VM instances).
The new tasks were written in Javascript, tested with the embedded interpreter, then added to the library.

After that, they were available in the automation library among the tasks provided by the product itself.
This is a very powerful demonstration of the flexibility and ease of use of UCSD.

I should add that the custom integration with Openstack was built for fun, and as a demonstration.
To implement the deployment of the tiers of the application to 3 different hypervisors we could use the native integration that UCSD has with KVM, Hyper-V and ESXi (through their managers).
There's no need to use Openstack as a mediation layer, as we did here.

The workflow editor

Here you can drag 'n drop the task, validate the workflow, run the process to test it and see the executed steps (with their log and all their input and output values).

Amount of effort

The main activities in building this demo are two:
- creating the custom tasks to integrate Openstack
- creating the process to automate the sequence of atomic tasks.

The first activity (skills required: Javascript programming and understanding of the Openstack API) took 1 hour per task: a total of 2 days.
Jose, who created the custom tasks, has also published a generic custom task to execute REST API calls from UCSD: https://github.com/erjosito/stuff/blob/master/UCSD_REST_custom_tasks.wfdx
In addition, he suggests a simple method to understand what REST call corresponds to a Openstack CLI command.
If you use the  --debug option in the Openstack CLI you will see that immediately.

As an example, to boot a new instance:
nova --debug boot --image cirros-0.3.1-x86_64-uec --flavor m1.tiny --nic net-id=f85eb42a-251b-4a75-ba90-723f99dbd00f vm002


The second activity (create the process, test it step by step, expose it in the catalog and run it end to end) took 3 sessions of 2 hours each.
This was made easier by the experience we matured during the implementation of the Elastic Cloud Project. We knew already the atomic actions we needed to perform, their sequence and the input/output parameter for each action.
If we had to build everything from scratch, I would add 2-3 days to understand the use case.

Demo available on dCloud

The demo will be published on the Cisco dCloud site soon for your consumption.
There are also a number of demonstrations available already, focused on UCS Director.
You can learn how UCSD manages the Data Center infrastructure, how it drives the APIC controller in the ACI architecture, and how it is leveraged by Cisco IAC when it uses the REST API exposed by UCSD.

Acknowledgement

A lot of thanks to Simon Richards and Manuel Garcia Sanes from Cisco dCloud, to Russ Whitear from my same team and to Jose Moreno from the Cisco INSBU (Insieme Business Unit).
Great people that focus on Data Center orchestration and many other technologies at Cisco!

You can also find a powerful, yet easy demonstration of how UCSD workflows can be called from a client (a front end portal, another orchestrator...) at Invoking UCS Director Workflows via the Northbound REST API