The Phoenix Project - how DevOps can change your life

It’s been a long time since I did my last post: as promised, I only post information from my experience in the real world and I avoid echoing messages from marketing   :-)

I’ve not been at rest, though, but I’ve worked at customer projects that can’t be mentioned publicly (yet).

But I’ve also been in vacation and I could finally read a great book, “The Phoenix Project”. 

It is a novel and a very educational reading at the same time.

I wholehearted recommend you to read it (though I’m not earning anything from the book) because I enjoied it a lot and I learned important lessons that deserve to be spread - for our common benefit as IT community.

You are not required to be a IT professional but, if you are, you will benefit the most and it will recall many familiar stories.

Since I’ve led some mission critical projects, and my skin is still impressed with both tragedy and triumph, this story reminded me those great moments. 

If you are new to DevOps, you can read my introductory posts in this blog.

Essentially, The Phoenix Project describes the evolution of IT in a company that, on the verge of a complete failure, pioneers DevOps and revolutionizes the way they work.

The impact on the core business is huge and their strategy creates a gap with the competition thanks to agility and flexibility.

Also personal lives are affected because the new organization ends the tribal war among Development, Operations, Security and the business stakeholders: they establish respect, trust and satisfaction for all the involved parties.

Of course the DevOps methodology is not a magic wand that makes the miracle for them: it is the outcome of a new way of thinking and working together.

This is a story of people, rather that technology.

If every IT department put themselves in the shoes of the others, instead of finger pointing, they can help each other to reach a common goal.

If the whole IT is not a counterpart of the LOBs but is a partner (understanding why they are asked something instead of focusing on how to do it), they can offer a huge value to the company… and be highly rewarded (see the coup de théâtre at the end of the story).

This would stop the “dysfunctional marriage” between two parties that don’t understand each other and suffer from a forced relationship.

In my experience, most of the business people see the IT as the provider of a services that is never satisfactory.

On the other side, IT sees that business people don’t understand the complexity and the effort required and ask for impossible things.

In most cases, they are bound to a traditional way of working and don’t even raise their head to see that they already own what’s needed to win.

They are overwhelmed by current tasks, troubleshooting and budget cuts, so they can’t think strategically.

The great idea, here, is importing the concepts and the experience from Lean Manufacturing into IT.

They start considering the IT organization similar to a production plant and optimizing its organization.

Finding bottlenecks and avoiding rework are the first steps, then automation follows to free the smart guys from the routine work and so the quality skyrockets.

At the end of the story the release of new features required by the business no longer takes months (and high risk at the roll out) but they can deploy 10 project builds per day!

That is not impressing if you think that these days some companies achieve 1000s of deployments per day thanks to Continuos Integration and Continuous Deployment.

But it is light years ahead of what most of my customers are doing, though some are exploring DevOps now.

Of course, one organization cannot change overnight.

You shouldn’t see the adoption of DevOps as a single step, and be scared by the effort.

In the book, they learn gradually and improve accordingly: you could do the same.

They go through a process that is made of Three Ways, until they master all.
A brief description of the three ways follows, thanks to Richard Campbell:

The First Way – Systems Thinking

• Understand the entire flow of work

• Seek to increase the flow of work

• Stop problems early and often – Don’t let them flow downstream

• Keep everyone thinking globally

• Deeply understand your systems

First Way Goals

• One source of truth – Code, environment and configuration in one place

• Consistent release process – Automation is essential (one click)

• Decrease cycle times, Faster release cadence

The Second Way – Feedback Loops

• Understand and respond to the needs of all customers (internal and external)

• Shorten and amplify all feedback loops

• With feedback comes quality

Second Way Goals

• Defects and performance issues fixed faster

• Ops and InfoSec user stories appear as part of the application

• Everyone is communicating better

• More work getting done

The Third Way – Synergy

• Consistent process and effective feedback result in agility

• Now use that agility to experiment

• You only learn from failure – So fail often, but recover quickly

Third Way Goals

• Ability to anticipate, even define new business needs through visibility in the systems

• Ability to test and optimize new business opportunities in the system while managing risk

• Joy

You should not think that The Phoenix Project is a technical book: though I’ve learned new things or reinforced concepts I knew already, the value I found in it is motivational.

It really moves you to action, and you want to measure the immediate improvement you can get.

More, you want to partner with other stakeholders to achieve common goals.

The Essence of DevOps

• Better Software, Faster

• Pride in the Software You Build and Operate

• Ability to Identify, Respond and Improve Business Needs

My final take from this story is that everybody in the IT (like in other fields) should:

- take risk and innovate - if you fail, probably the result would not be worse than staying still

- invest time - at cost of delaying important targets - to think strategically: the return will overpay the effort

- study what others have done already: learning by examples is much easier

- always try to understand your counterpart before fighting by principle, there could be a common advantage if you shift your perspective

Some useful references:

Amazon - http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592
R. Campbell's presentation -  http://techdays.org/wp-content/uploads/2013/05/Essence-of-DevOps.pptx

Other DevOps books:
- Visible Ops Handbook (Gene Kim)
- Web Operations (Allspaw/Robbins)
- Continuous Delivery (Humble/Farley)
- Lean Startup (Eric Reis)

Invoking UCS Director Workflows via the Northbound REST API

This is a guest post, offered by a colleague of mine: Russ Whitear.

Russ is the UCS Director guru  in our team and, when I saw an internal email where he explained how to use the UCS Director API from an external client, I asked his permission to publish it.

I believe it will be useful for many customers and partners to integrate UCSD in a broader ecosystem.

This short post explains how to invoke UCS Director workflows via the northbound REST API. Authentication and role is controlled by the use of a secure token.  Each user account within UCS Director has a unique API token, which can accessed via the GUI like so:

Firstly, from within the UCS Director GUI, click the current username at the top right of the screen. Like so:

User Information will then be presented. Select the ‘Advanced’ tab in order to reveal the API Access token for that user account.

Once retrieved, this token needs to be added as an HTTP header for all REST requests to UCS Director.  The HTTP header name must be X-Cloupia-Request-Key.

X-Cloupia-Request-Key : E0BEA013C6D4418C9E8B03805156E8BB

Once this step is complete, the next requirement is to construct an appropriate URI for the HTTP request in order to invoke the required UCS Director workflow also supplying the required User Inputs (Inputs that would ordinarily be entered by the end user when executing the workflow manually).

UCS Director has two versions of northbound API. Version 1 uses HTTP GET requests with a JSON (Java Standard Object Notation) formatted URI. Version 2 uses HTTP POST with XML (eXtensible Markup Language) bodytext.

Workflow invokation for UCS Director uses Version 1 of the API (JSON). A typical request URL would look similar to this:

http://<UCSD_IP>/app/api/rest?formatType=json

                 &opName=userAPISubmitWorkflowServiceRequest

                 &opData={SOME_JSON_DATA_HERE}

A very quick JSON refresher

JSON formatted data consists of either dictionaries or lists. Dictionaries consist of name/value pairs that are separated by a colon. Name/value pairs are separated by a comman and dictionaries are bounded by curly braces. For example:

{“animal”:”dog”, “mineral”:”rock”, “vegetable”:”carrot”}

Lists are used in instances where a single value is insufficient. Lists are comma separated and bounded by square braces. For example:

{“animals”:[“dog”,”cat”,”horse”]}

To ease readability, it is often worth temporarily expanding the structure to see what is going on. 

{

    “animals”:[

        “dog”,

        ”cat”,

        ”horse”

    ]

}

Now things get interesting. It is possible (And common) for dictionaries to contain lists, and for those lists to contain dictionaries rather than just elements (dog, cat, horse etc…). 

{ “all_things”:{

        “animals”:[

            “dog”,

            ”cat”,

            ”horse”

        ],

        “minerals”:[

            “Quartz”,

            “Calcite”

        ],

        “vegetable”:”carrot”

    }

}

With an understanding of how JSON objects are structured, we can now look at the required formatting of the URI for UCS Director. When invoking a workflow via the REST API, UCS Director must be called with three parameters, param0, param1 and param2. ‘param0’ contains the name of the workflow to be invoked. The syntax of the workflow name must match EXACTLY the name of the actual workflow. ‘param1’ contains a dictionary, which itself contains a list of dictionaries detailing each user input and value that should be inserted for that user input (As though an end user had invoked the workflow via the GUI and had entered values manually.

The structure of the UCS Director JSON URI looks like so:

{

    param0:"<WORKFLOW_NAME>",

    param1:{

                "list":[

                       {“name":"<INPUT_1>","value":"<INPUT_VALUE>"},

                       {"name":"<INPUT_2>","value":"<INPUT_VALUE"}

                ]

            },

    param2:-1

}

So, let’s see this in action. Take the following workflow, which happens to be named ‘Infoblox Register New Host’ and has the user inputs ‘Infoblox IP:’,’Infoblox Username:’,’Infoblox Password:’,’Hostname:’,’Domain:’ and ‘Network Range:’.

The correct JSON object (Shown here in pretty form) would look like so:

Note once more, that the syntax of the input names must match EXACTLY that of the actual workflow inputs.

After removing all of the readability formatting, the full URL required in order to invoke this workflow with the ‘user’ inputs as shown above would look like this:

Now that we have our URL and authentication token HTTP header, we can simply enter this information into a web based REST client (e.g. RESTclient for Firefox or Postman for Chrome) and execute the request. Like so:

 

If the request is successful, then UCS Director will respond with a “serviceError” of null (No error) and the serviceResult will contain the service request ID for the newly invoked workflow:

Progress of the workflow can either be monitored by other API requests or via the UCS Director GUI:

Service request logging can also be monitored via either further API calls or via the UCS Director GUI:

This concludes the example, that you could easily test on your own instance of UCS Director or, if you don't have one at hand, in a demo lab on dcloud.cisco.com.

It should be enough to demonstrate how simple is the integration of the automation engine provided by UCSD, if you want to execute its workflows from an external system: a front end portal, another orchestrator, your custom scripts.

See also The Elastic Cloud project - Porting to UCSD for the deployment of a 3 tier application to 3 different hypervisors, using Openstack and ACI with Cisco UCS Director.

The Elastic Cloud project - Porting to UCSD

Porting to a new platform

This post shows how we did the porting of the Elastic Cloud project to a different platform.
The initial implementation was done on Cisco IAC (Intelligent Automation for Cloud) orchestrating Openstack, Cisco ACI (Application Centric Infrastructure) and 3 hypervisors.

Later we decided to implement the same use case (deploy a 3 tier application to 3 different hypervisors, using Openstack and ACI) with Cisco UCS Director, aka UCSD.

The objective was to offer another demonstration of flexibility and openness, targeting IT administrators rather than end users like we did in the first project.
You will find a brief description of UCS Director in the following paragraphs: essentially it is not used to abstract complexity, but to allow IT professionals to do their job faster and error-proof.
UCSD is also a key element in a new Cisco end-to-end architecture for cloud computing, named Cisco ONE Enterprise Cloud suite.

The implementation was supported by the Cisco dCloud team, the organization that provides excellent remote demo capabilities on a number of Cisco technologies. They offered me the lab environment to build the new demo and, in turn, the complete demo will be offered publicly as a self service environment on the dCloud platform.

The dCloud demo environment

Cisco dCloud provides Customers, Partners and Cisco Employees with a way to experience Cisco Solutions. From scripted, repeatable demos to fully customizable labs with complete administrative access, Cisco dCloud can work for you. Just login to dcloud.cisco.com with your Cisco account and you'll find all the available demo:

Cisco UCS Director

UCSD is a great tool for Data Center automation: it manages servers, network, storage and hypervisors, providing you a consistent view on physical and virtual resources in your DC.

Despite the name (that could associate it to Cisco UCS servers only) it integrates with a multi-vendor heterogeneous infrastructure, offering a single dashboard plus the automation engine (with a library containing 1300+ tasks) and the SDK to create your own adapters if needed.

UCSD offers open API so that you can run its workflows from the UCSD catalog or from a 3rd party tool (a portal, a orchestrator, a custom script).

There is a basic workflow editor, that we used to create the custom process integrating Openstack, ACI and all the hypervisors to implement our use case. We don't consider UCSD a full business level orchestrator because it's not meant to integrate also the BSS (Business Support Systems) in your company, but it does the automation of the DC infrastructure including Cisco and 3rd party technologies pretty well.

Implementing the service in UCS Director

Description of the process

The service consists in the deployment of the famous 3 tier application with a single click.
The first 2 tiers of the application (web and application servers and their networks) are deployed on Openstack. The first version of the demo uses KVM as the target hypervisor for both tiers, next version will replace one of the Openstack compute nodes with Hyper-V.
The 3rd tier (the database and its network) is deployed on ESXi.
On every hypervisor, virtual networks are created first. Then virtual machines are created and attached to the proper network.

To connect the virtual networks in their different virtualized environments we used Cisco ACI, creating policies through the API of the controller.
One End Point Group is created for each of the application tiers, Contracts are created to allow the traffic to flow from one tier to next one (and only there).
If you are not familiar with the ACI policy model, you can see my ACI for Dummies post.

All these operations are executed by a single workflow created in the UCSD automation engine.
We just dropped the tasks from the library to the workflow editor, provided input values for each task (from the output of previous tasks) and connected them in the right sequence drawing arrows.
The resulting workflow executes the same sequence of atomic actions that the administrator would do manually in the GUI, one by one.

The implementation was quite easy because we were porting an identical process created in Cisco IAC: the tool to implement the workflow is different, but the sequence and the content of the tasks is the same.

Integration out-of-the-box

Most of the tasks in our process are provided by the UCSD automation library: all the operations on ACI (through its APIC controller) and on ESXi VM and networks (through vCenter).

When you use these tasks, you can immediately see the effect in the target system.
As an example, this is the outcome of creating a Router in Openstack using UCSD: the two networks are connected in the hypervisor and the APIC plugin in Neutron talks immediately to Cisco ACI, creating the corresponding Contract between the two End Point Groups (please check the Router ID in Openstack and the Contract name in APIC).

 

Custom tasks

The integration with Openstack required us to build custom tasks, adding them to the library.
We created 15 new tasks, to call the API exposed by the Openstack subsystems: Neutron (to create the networks) and Nova (to create the VM instances).
The new tasks were written in Javascript, tested with the embedded interpreter, then added to the library.

After that, they were available in the automation library among the tasks provided by the product itself.
This is a very powerful demonstration of the flexibility and ease of use of UCSD.

I should add that the custom integration with Openstack was built for fun, and as a demonstration.
To implement the deployment of the tiers of the application to 3 different hypervisors we could use the native integration that UCSD has with KVM, Hyper-V and ESXi (through their managers).
There's no need to use Openstack as a mediation layer, as we did here.

The workflow editor

Here you can drag 'n drop the task, validate the workflow, run the process to test it and see the executed steps (with their log and all their input and output values).

Amount of effort

The main activities in building this demo are two:
- creating the custom tasks to integrate Openstack
- creating the process to automate the sequence of atomic tasks.

The first activity (skills required: Javascript programming and understanding of the Openstack API) took 1 hour per task: a total of 2 days.
Jose, who created the custom tasks, has also published a generic custom task to execute REST API calls from UCSD: https://github.com/erjosito/stuff/blob/master/UCSD_REST_custom_tasks.wfdx
In addition, he suggests a simple method to understand what REST call corresponds to a Openstack CLI command.
If you use the  --debug option in the Openstack CLI you will see that immediately.

As an example, to boot a new instance:
nova --debug boot --image cirros-0.3.1-x86_64-uec --flavor m1.tiny --nic net-id=f85eb42a-251b-4a75-ba90-723f99dbd00f vm002


The second activity (create the process, test it step by step, expose it in the catalog and run it end to end) took 3 sessions of 2 hours each.
This was made easier by the experience we matured during the implementation of the Elastic Cloud Project. We knew already the atomic actions we needed to perform, their sequence and the input/output parameter for each action.
If we had to build everything from scratch, I would add 2-3 days to understand the use case.

Demo available on dCloud

The demo will be published on the Cisco dCloud site soon for your consumption.
There are also a number of demonstrations available already, focused on UCS Director.
You can learn how UCSD manages the Data Center infrastructure, how it drives the APIC controller in the ACI architecture, and how it is leveraged by Cisco IAC when it uses the REST API exposed by UCSD.

Acknowledgement

A lot of thanks to Simon Richards and Manuel Garcia Sanes from Cisco dCloud, to Russ Whitear from my same team and to Jose Moreno from the Cisco INSBU (Insieme Business Unit).
Great people that focus on Data Center orchestration and many other technologies at Cisco!

You can also find a powerful, yet easy demonstration of how UCSD workflows can be called from a client (a front end portal, another orchestrator...) at Invoking UCS Director Workflows via the Northbound REST API