How to create a service end to end in Cisco ONE ECS

Training and real world usage of the products

Sometimes training is more focused on the procedural detail of the individual components than on the real world usage of a system.
You might miss the understanding of the end-to-end architecture and the use cases that you could address with that solution so you go home, at the end of the training, without a complete awareness.

In the case of the Cisco ONE Enterprise Cloud Suite, that is composed of a number of components, in a course for beginners you will learn how to use Prime Service Catalog, UCS Director, Intercloud Fabric Director and VACS.
But, after you know how to configure them and what's the value provided by every tool, you might still wonder "what I'm going to do with this architecture?" or "how complex would it be to implement a complete project?".

I put this sample use case together to show what is the process to create a brand new service in the self service catalog, complete with all the implementation of the delivery of the service. My colleague Maxim Khavankin helped me to document all the steps.
If you download PSC and UCSD and run them with the evaluation license, you could run through this exercise very easily and make friends with the platform.

Hello World with Cisco ECS

I implemented a very simple service, just to have a context to show the implementation.
No business logic is involved, or integration with back end systems, to keep you focused on the mechanics: you can easily extend this framework to your real use case.

The idea is to order a service in PSC, providing a input, and let UCSD deliver the outcome.
In our case the expected result is writing a "Hello <your name>" message in the log file.

Generally workflows in UCSD make use of tasks from the library (you have more than 2000 tasks to automate servers, network, storage and virtualization). But a specific use case might require a task that is not available already, so you build it and add it to the library.

I created a custom task in UCSD just to write to the log: of course, you could replace this extremely exciting logic with a call to the REST API - or any other API - of the system you want to target: infrastructure managers in your data center, enterprise software systems, your partner's API for a B2B service, etc. 

 

Then I created a custom workflow in UCSD, that takes your name as a input and makes use of the task I mentioned to deliver the "Hello World" service. The workflow can be executed in UCSD (e.g. for unit testing) or invoked via the UCSD API.

Prime Service Catalog has a wizard that explores the API exposed by UCSD, discovers and imports all the entities it finds (including workflows) so that you can immediately offer them as services in the catalog for end users. Of course a customization can be added with the tools provided by PSC.

So the end to end procedure to create a business services is described by the following steps:

1. Create a custom task (if required)
2. Define a workflow that uses the custom task -> define input variables
3. Create a catalog item in UCSD -> offer the workflow from step 2
4. Synchronize PSC and UCSD
5. Use the wizard to import the service in PSC
6. Customize the service in the PSC catalog with Service Designer (optional)
7. Order the customized service
8. Check the order status on PSC side
9. Check the order status and outcome in UCSD

I illustrate every step with some pictures:

Create a custom task (if required)    

Custom tasks can be added to the existing library where 2000+ tasks are offered to manage servers, network, storage and virtualization.

You can group tasks in Categories so that they can be found easily in the workflow editor later. 

 

Custom tasks can have (optional) input and output parameters, that you define based on variable types: in this case I used a generic text variable to contain the name to send greetings to:

The format, contraints and presentation style can be defined:

You can skip the steps "Custom Task Outputs" and "Controller" in the wizard to create the task: we don't need them in this use case.

Finally we create the logic for our use case: a small piece of Javascript code that executes the custom action we want to add to the automation library.

The UCSD logger object has a method to write an Information/Warning/Error message to the UCSD log file. As I wrote earlier, you could use http calls here to invoke REST API if this was a real world use case.

After you've created your custom task it's available in the automation library.
Now you have to define a workflow that uses the custom task: to pass the input that the task requires, you will define a input variable in the workflow.

The workflow is an entity that contains a number of tasks. The workflow itself has its own input and output parameters, that can be used by the individual tasks.

Input and output parameters of the workflow are defined in the same way as tasks' input and output.

They can be useful if you launch the workflow via the REST API exposed by UCSD.

Now that you've created the workflow, it's time to add some tasks to it picking from the library (exposed in the left panel of the workflow editor).
We'll only add one task (the custom task that we created): select it from the library, eventually searching for the word "hello".
Drag and drop the task in the editor canvas, then configure it.

You will see a screen similar to this one:

 Configure the new task giving it a name:

Map the input variable of the task to the input parameter of the workflow that you created:

If you had not a variable holding the value for this task's input, you could still hard code the input value here (but it's not our case: this form would appear different if you hadn't mapped the variable in the previous screen).

 

The task does not produce any output value, so there's no option to map it to the output parameters of the workflow.

Finally we see the complete workflow (one single task, in our example) and we can validate it: it's a formal check that all the tasks are connected and all the variables assigned.

 

Then we can execute it from the same window, to check that it behaves correctly. You will have access to the log file from the same window that pops up when you execute the workflow, so you can see that the greetings appear in the log.

Next action is to expose this workflow to users in UCSD (in the GUI and via the API).

Create catalog item in UCSD -> offer workflow from Step 2   

UCSD catalog items are offered to non-admin users if you so choose. They are grouped in folders in the user interface, and you can make them visible to specific users or groups.

You can give them a name and a description and associate a service, that could be the provisioning of a resource or a custom workflow - like in our case.

The workflow is selected and associated here: 

After defining the new catalog item, you'll see it here - and in the end-user web GUI.

If the service is offered to technical users (e.g. the IT operations team), your work could be considered complete.
They can access UCSD and launch the workflow. The essential user interface of the tool is good enough for technical users that only need efficiency.

But if you're building a private cloud you might want to offer your end users a more sophisticated user interface and a complete self service catalog populated with any kind of services, where you apply the governance rules for your business.

So we'll go on and expose the "Hello World" service in Cisco Prime Service Catalog.

Synchronize PSC and UCSD   

Login to PSC as admin, go to Administration -> Manage Connections.
Click on the connection to UCSD (previously defined by giving it the target ip address and credentials) and click "Connect & Import".

PSC will discover all the assets offered by UCSD.
Now you can use the wizard to import the "Hello World" service in PSC. With few clicks it will be exposed in the service catalog.  

The wizard allows you to associate an image and a description with the service. Here you can describe it at the level of detail and abstraction that are more appropriate for your users (or customers).
You have a full graphic editor that does not require any skills as a web designer.

 

Additional metadata (attributes of the service) can be added, so that users can find it when searching the catalog: there is a search engine that PSC provides out of the box.

And finally you decide who can see and order the service in the catalog: you can map it to single users, groups, roles, organizations or just offer it to everyone.

 

At this time the service is fully working in the self service catalog and his lifecycle is managed. But, if you like, you can still apply customization and leverage the power of PSC. 

Customize the service in the PSC catalog with Service Designer (optional)   

There a subsystem in PSC, accessible only to specific user roles, that is called "Service Designer". It can be used to build services from scratch or to edit existing services, like the one that the wizard generated for us. Just go there and select the "Hello World" service.

The user interface of the service is built on reusable elements, that are called Active Forms (one active form could be reused in many services). The wizard generated a Active Form for our service, with a name corresponding to it.

 

If you select the active form and go to the panel "Display Properties" you can change the appearance and the behavior of the order form.

 

As an example the only input field, named "person", can be transformed into a drop down list with pre-populated items. Items could even be obtained from a database query or from a call to a web service, so that the list is dynamically populated.

 

The power of the Service Designer offers many more customization options, but here we want to stay on the easiest side so we'll stop here   :-)

Order the customized service   

Go to the home page of the Service Catalog. Browse the categories (did you create a custom category or just put the Hello World service in one of the existing categories?). You can also search for it using the search function, accessed via the magnifier glass icon.

In this picture you also see a review made by one of the users of the catalog that has already used the service. You can add your own after you've ordered it at least once.

You will be asked to provide the required input:

 

When you submit a request, your order is tracked in My Stuff -> Open Orders.

This is also used for audit activities.

Check order status on PSC side  

You will see the progress of the delivery process for your order: in general it has different phases including, if needed, the approval by specific users.

Check order status and outcome in UCSD     

If you go back to the admin view in UCSD (Organizations -> Service Requests) you will see that a new service request has been generated: double click on it to see the status.

if you click on the Log tab you can check the result of the execution of the service: the hello message has been delivered!

Now that you appreciated how easy is to build new services with PSC+UCSD you're ready to use all the features provided by the products and the pre-built integration that makes it very quick.

All your data center infrastructure is managed by UCSD, so that you can automate provisioning and configuration of servers, network and storage (of course, from any vendor and both physical and virtual). Once you've the automation done, offering services in the self service catalog takes just few minutes.

References

Cisco Enterprise Cloud Suite
and its individual components:
- Cisco PSC - Prime Service Catalog 
- Cisco UCSD - UCS Director

Governance in the hybrid cloud

This post shows how a company can solve one of the main issue that CIOs have today: the so called Shadow IT.

This term defines the usage of cloud services (either IaaS, PaaS, SaaS) in a project without any control, decided by the application developers or designers because they think it's beneficial for the agility of the project.

Sometimes leveraging available services is really good for a project: it's useless to rebuild something that is easily available as a standardized service. Even when the IT organization of your company (or your customer, if you're a consulting company) provides the building blocks that you need for your architecture, it could be difficult to get approvals or a fast enough provisioning.
So there are different valid reasons to incorporate public cloud services, we can't blame those that try to fully exploit a Service Oriented Architecture.

Unfortunately this way of assembling applications using any available resource you consider useful creates troubles for the IT organization.
Besides additional costs, that arrive as a surprise (developers bill to a personal credit card or to a corporate one, but sooner or later those costs will be factored into the cost of the project), some corporate rules could be violated without even being aware.
Just a few examples: storing reserved data in a database outside the company's datacenter, or invoking services without encrypting the input/output parameters, not granting the end to end High Availability or Disaster Recovery of the entire system.

The subject of costs can be easily underestimated: at development time you need very limited cloud resources, for a limited time. It costs near to zero, before the application goes to a full production environment. But after that, it will need more computing power and more storage, and of course more bandwidth, to serve all the users. Cloud services tend to increase surprisingly in these conditions.

So the CIO has a dilemma: to try to block, or limit, the usage of cloud services - limiting cost and risk but appearing like the one that slows the innovation down and prevents the lines of business from achieving their business result - or to allow maximum freedom, with the additional risk of becoming not relevant because they can bypass the IT organization?

There is a solution in the middle: IT could offer a facilitated access to cloud services, adding them to a Service Catalog where users can self serve, granting compliance by design.
Public cloud services will be selected based on agreed architectural and security policies, they will be documented, audited and reported, eventually subject to approval from a financial standpoint.

One possible implementation of such a catalog can be based on the Cisco ONE Enterprise Cloud Suite, as I did in a recent project at one of my customers.

The Cisco ECS is a reference architecture comprising one flexible Service Catalog, a automation engine and a platform for hybrid cloud that allows the extension of your datacenter into a kind of "bubble" in the public cloud. In case you need additional power, you can burst your workloads into the virtual private datacenter keeping all the security and networking policies you defined in your private cloud: even the IP address of the virtual machines does not change, as long as the secure segmentation of the application layers and any other policy.

I'm not going to describe the Cisco ECS, because you can find the official documentation here.
I'm showing how we extended the services offered in this catalog with CliQr Cloud Center for managing the provisioning and the lifecycle management for applications in the cloud. So the great capabilities of Cisco ECS in term of IaaS are complemented with the offer of the deployments of simple or complex applications and software stacks, that you can target at any cloud just selecting from a drop down list.

I mean that the template for the deployment is not cloud dependent,  and the user can - within the limits of his authorization level and the corporate policies - choose to provision it in the private cloud (e.g. on vmware in the corporate data center) or in the public cloud (e.g. AWS or Azure).
The lifecycle operations (start, stop, resume, delete, etc.) will be also offered as well as the migration to a different cloud: from private to public after the QA test is done and you're ready for production, from a public provider to a more convenient one, etc.

THIS POST HAS BEEN REDACTED

After the publication of this post Cisco announced the intent to acquire Cliqr (not because of the post :-) ), and our policies require that we don't speak of deals while they are in progress. I can't show the way we integrated Cliqr in this project because the official statement on the reference architecture will be communicated by Cisco after the acquisition is eventually completed.

References:
http://blogs.cisco.com/datacenter/introducing-cisco-one-enterprise-cloud-suite
http://www.cliqr.com/

The Elastic Cloud project - Porting to UCSD

Porting to a new platform

This post shows how we did the porting of the Elastic Cloud project to a different platform.
The initial implementation was done on Cisco IAC (Intelligent Automation for Cloud) orchestrating Openstack, Cisco ACI (Application Centric Infrastructure) and 3 hypervisors.

Later we decided to implement the same use case (deploy a 3 tier application to 3 different hypervisors, using Openstack and ACI) with Cisco UCS Director, aka UCSD.

The objective was to offer another demonstration of flexibility and openness, targeting IT administrators rather than end users like we did in the first project.
You will find a brief description of UCS Director in the following paragraphs: essentially it is not used to abstract complexity, but to allow IT professionals to do their job faster and error-proof.
UCSD is also a key element in a new Cisco end-to-end architecture for cloud computing, named Cisco ONE Enterprise Cloud suite.

The implementation was supported by the Cisco dCloud team, the organization that provides excellent remote demo capabilities on a number of Cisco technologies. They offered me the lab environment to build the new demo and, in turn, the complete demo will be offered publicly as a self service environment on the dCloud platform.

The dCloud demo environment

Cisco dCloud provides Customers, Partners and Cisco Employees with a way to experience Cisco Solutions. From scripted, repeatable demos to fully customizable labs with complete administrative access, Cisco dCloud can work for you. Just login to dcloud.cisco.com with your Cisco account and you'll find all the available demo:

Cisco UCS Director

UCSD is a great tool for Data Center automation: it manages servers, network, storage and hypervisors, providing you a consistent view on physical and virtual resources in your DC.

Despite the name (that could associate it to Cisco UCS servers only) it integrates with a multi-vendor heterogeneous infrastructure, offering a single dashboard plus the automation engine (with a library containing 1300+ tasks) and the SDK to create your own adapters if needed.

UCSD offers open API so that you can run its workflows from the UCSD catalog or from a 3rd party tool (a portal, a orchestrator, a custom script).

There is a basic workflow editor, that we used to create the custom process integrating Openstack, ACI and all the hypervisors to implement our use case. We don't consider UCSD a full business level orchestrator because it's not meant to integrate also the BSS (Business Support Systems) in your company, but it does the automation of the DC infrastructure including Cisco and 3rd party technologies pretty well.

Implementing the service in UCS Director

Description of the process

The service consists in the deployment of the famous 3 tier application with a single click.
The first 2 tiers of the application (web and application servers and their networks) are deployed on Openstack. The first version of the demo uses KVM as the target hypervisor for both tiers, next version will replace one of the Openstack compute nodes with Hyper-V.
The 3rd tier (the database and its network) is deployed on ESXi.
On every hypervisor, virtual networks are created first. Then virtual machines are created and attached to the proper network.

To connect the virtual networks in their different virtualized environments we used Cisco ACI, creating policies through the API of the controller.
One End Point Group is created for each of the application tiers, Contracts are created to allow the traffic to flow from one tier to next one (and only there).
If you are not familiar with the ACI policy model, you can see my ACI for Dummies post.

All these operations are executed by a single workflow created in the UCSD automation engine.
We just dropped the tasks from the library to the workflow editor, provided input values for each task (from the output of previous tasks) and connected them in the right sequence drawing arrows.
The resulting workflow executes the same sequence of atomic actions that the administrator would do manually in the GUI, one by one.

The implementation was quite easy because we were porting an identical process created in Cisco IAC: the tool to implement the workflow is different, but the sequence and the content of the tasks is the same.

Integration out-of-the-box

Most of the tasks in our process are provided by the UCSD automation library: all the operations on ACI (through its APIC controller) and on ESXi VM and networks (through vCenter).

When you use these tasks, you can immediately see the effect in the target system.
As an example, this is the outcome of creating a Router in Openstack using UCSD: the two networks are connected in the hypervisor and the APIC plugin in Neutron talks immediately to Cisco ACI, creating the corresponding Contract between the two End Point Groups (please check the Router ID in Openstack and the Contract name in APIC).

 

Custom tasks

The integration with Openstack required us to build custom tasks, adding them to the library.
We created 15 new tasks, to call the API exposed by the Openstack subsystems: Neutron (to create the networks) and Nova (to create the VM instances).
The new tasks were written in Javascript, tested with the embedded interpreter, then added to the library.

After that, they were available in the automation library among the tasks provided by the product itself.
This is a very powerful demonstration of the flexibility and ease of use of UCSD.

I should add that the custom integration with Openstack was built for fun, and as a demonstration.
To implement the deployment of the tiers of the application to 3 different hypervisors we could use the native integration that UCSD has with KVM, Hyper-V and ESXi (through their managers).
There's no need to use Openstack as a mediation layer, as we did here.

The workflow editor

Here you can drag 'n drop the task, validate the workflow, run the process to test it and see the executed steps (with their log and all their input and output values).

Amount of effort

The main activities in building this demo are two:
- creating the custom tasks to integrate Openstack
- creating the process to automate the sequence of atomic tasks.

The first activity (skills required: Javascript programming and understanding of the Openstack API) took 1 hour per task: a total of 2 days.
Jose, who created the custom tasks, has also published a generic custom task to execute REST API calls from UCSD: https://github.com/erjosito/stuff/blob/master/UCSD_REST_custom_tasks.wfdx
In addition, he suggests a simple method to understand what REST call corresponds to a Openstack CLI command.
If you use the  --debug option in the Openstack CLI you will see that immediately.

As an example, to boot a new instance:
nova --debug boot --image cirros-0.3.1-x86_64-uec --flavor m1.tiny --nic net-id=f85eb42a-251b-4a75-ba90-723f99dbd00f vm002


The second activity (create the process, test it step by step, expose it in the catalog and run it end to end) took 3 sessions of 2 hours each.
This was made easier by the experience we matured during the implementation of the Elastic Cloud Project. We knew already the atomic actions we needed to perform, their sequence and the input/output parameter for each action.
If we had to build everything from scratch, I would add 2-3 days to understand the use case.

Demo available on dCloud

The demo will be published on the Cisco dCloud site soon for your consumption.
There are also a number of demonstrations available already, focused on UCS Director.
You can learn how UCSD manages the Data Center infrastructure, how it drives the APIC controller in the ACI architecture, and how it is leveraged by Cisco IAC when it uses the REST API exposed by UCSD.

Acknowledgement

A lot of thanks to Simon Richards and Manuel Garcia Sanes from Cisco dCloud, to Russ Whitear from my same team and to Jose Moreno from the Cisco INSBU (Insieme Business Unit).
Great people that focus on Data Center orchestration and many other technologies at Cisco!

You can also find a powerful, yet easy demonstration of how UCSD workflows can be called from a client (a front end portal, another orchestrator...) at Invoking UCS Director Workflows via the Northbound REST API