Protecting your border or offering a service to others?

The value of automation in the DataCenter

Everyone is aware of the value of the automation.
Many companies and individual engineers implemented various ways to save time, from shell scripts to complex programs and to fully automated IaaS solutions.

It helps reducing the so called "Shadow IT", a phenomenon that happens when developers can't get a fast enough response from the IT of the company and rush to the public cloud to get what they need. Doing that they complete and release their project soon, but sometimes troubles start with the production phase of the deployment (unexpected additional budget for the IT, new technologies that they are not ready to manage, etc.).

shadow IT happens when corporate IT is not fast enough

For sure, some departments are organized in silos (a team responsible for servers, one for storage, one for networking, one for virtual machines, of course one for security...) and the provisioning of even simple requests takes too long.

process inefficiency due to silos and wait time

Pressure on the infrastructure managers

So there is inefficiency in the company, that affects the business outcome of every project.
Longer time to market for strategic initiatives, higher costs for infrastructure and people.
Finger pointing starts, to identify who is responsible for the bottleneck.

The efficiency of teams and individuals is questioned, and responsibility is cascaded through the organization from project managers to developers, to the server team, to the storage team and generally the network is at the end of the chain... so that they have no one else to blame.

Those on the top (they consider themselves on top of the value chain) believe - or try to demonstrate - that their work is slowed down by the inefficiency of the teams they depend on. They try to suggest solutions like: "you said that your infrastructure is programmable, now give me your API and I will create everything I need on demand".

Of course this approach could bring some value (not much, as we'll see in the rest of the post) but it mines the relevance of the specialists teams that are supposed to manage the infrastructure according to best practices, to apply architectural blueprints that have been optimized for the company's specific business, to know the technology in deeper detail.
So they can't accept to be bypassed by a bunch of developers that want to corrupt the system playing with precious assets with their dirty hands.

The definitive question is: who owns the automation?
Should it be left to people that know what they need (e.g. Developers)?
Should it be owned by people that know how technology works, and at the end of the day are responsible for the SLA including performances, security and reliability that could be affected by a configuration made by others (i.e. IT Administrators)?


In my opinion, and based on the experience shared with many customers, the second answer is the correct one.
By definition the developer is not an expert on security: if he can easily program a switch via its REST API to get a network segment, it’s not the same when traffic needs to be secured and inspected.

The IT Admin patrolling the infrastructure

Offering a self service catalog (or API)

A first, immediate solution could be the introduction of an easy automation tool like Cisco UCS Director, that manages almost every element in a multi vendor Data Center infrastructure: from servers to networks to storage to virtualization in a single dashboard. But what is more interesting is that every atomic action you do in the GUI is also reflected in a task in the automation library, that allows you to create custom workflows lining all the tasks for a process that you want to automate.
A common example of automation workflow is the creation of a 4-hypervisors server farm.
A single workflow starts from the SAN storage creating a volume and 4 LUN, where the hypervisor will be installed to enable remote boot for the servers. Then a network is created (or the existing management network will be used) and 4 Service Profiles (the definition of a server in Cisco UCS) are created from a template, with individual ip address, mac address and wwn for each network interface. Then, zoning and masking are executed to map every new server to a specific LUN and the service profiles are associated to 4 available servers (either blades or rack mount servers). The hypervisors are installed using the PXE boot, writing the bytes in the remote storage, configured and customized, and finally added to a (new) cluster in the hypervisor manager (e.g. vCenter).

All this process takes less then one hour: you could launch it and go to lunch, when you're back you'll find the cluster up and running. Compare it to a manual provisioning of the same server farm, eventually performed by a number of different teams (see the picture above): it would take days, sometimes weeks. 
Other use cases are simpler: maybe just creating a 3 tier application with VM and dedicated networks.

Once the automation workflow has been built and validated, it can be used by the IT admin or by the Operations everyday, to save time and ensure consistent outcome (no manual errors). But it can also be offered as a service to all the departments that depend on the IT for their projects. 

You can build a service catalog with enterprise features: multitenancy, role based access control, reporting, chargeback, approvals, etc. But you can also offer (secured) access to the API to launch the workflow, offering a degree of autonomy to your consumers. Eventually, using a resource quota: you don’t want everyone to be able to create dozens of VMs every hour if the capacity of the system can't sustain it. 

They will appreciate the efficiency improvement, for sure.

What's in it for me?

If you allow your internal clients to self serve, you will: 

• get less requests for trivial tasks, that consume time and give no satisfaction (let them play with it),
• be the hero of the productivity increase (no requests pending in your queue)
• dedicate your time and skill to designing the architectural blueprint that will be offered as a service to your clients (so that everybody plays according to your rules)
• use policy based provisioning, so that you define the rules just once and map them to tenants and environments: every deployment will inherit them
• maintain control on resource consumption and system capacity, hence on costs and budget
• increase your relevance: they will come to you to discuss their needs, propose new services, collaborate in governance

Example: network provisioning

The discussion above is valid for the entire infrastructure in the Data Center.
Now I tell you the story of a customer that implemented it specifically for the networking.

They were influenced by the trend about SDN and initially they were caught in the marketing trap "SDN means software implemented networking, hence overlay". Then they realized the advantage provided by ACI and selected it as the SDN platform ("software defined networking", thanks to the software controller and the ACI policy model).

Developers and the Architecture department asked to access the API exposed to self provision what they needed for new projects, but this was seen as an invasion of the property (see the picture with the dirty hands).

It would have worked, but it implied a transfer of knowledge and delegation of responsibility on a critical asset. At the end of the day, if developers and software designers had knowledge in networking, specialists would not exist.

So the network admins built a number of workflows in UCS Director, using the hundreds of tasks offered by the automation library, to implement some use cases ranging from basic tasks (allow this VM to be reached from the DMZ) to more complex scenarios (create a new environment for a multi tier application including load balancer and firewall configuration, plus access from the monitoring tools, with a single request).

Blueprint designed in collaboration with Security and Software Architects

Graphical Editor for the workflow

These workflows are offered in a web portal (a service catalog is offered by UCSD out of the box) and through the REST API exposed by UCSD. Sample calls were provided to consumers as python clients, powershell clients and Postman collections, so that the higher level orchestration tool maintained by the Architecture dept was able to invoke the workflows immediately, inserting them in the business process automation that was already in place.

Example of python client running a UCSD workflow

All the executions of the workflows - launched through the self service catalog or through the REST API - are tracked in the system and the administrator can inspect the requests and their outcome:

The Service Requests are audited and can be inspected and rolled back

 Any run of the workflow can be inspected in full detail, look at the tabs in the window:

The Admin has full control (see the tabs in the window)

References

Cisco UCS Director
Cisco ACI 
ACI for Simple Minds
ACI for (Smarter) Simple Minds
Invoking UCS Director Workflows via the Northbound API 

How to create a service end to end in Cisco ONE ECS

Training and real world usage of the products

Sometimes training is more focused on the procedural detail of the individual components than on the real world usage of a system.
You might miss the understanding of the end-to-end architecture and the use cases that you could address with that solution so you go home, at the end of the training, without a complete awareness.

In the case of the Cisco ONE Enterprise Cloud Suite, that is composed of a number of components, in a course for beginners you will learn how to use Prime Service Catalog, UCS Director, Intercloud Fabric Director and VACS.
But, after you know how to configure them and what's the value provided by every tool, you might still wonder "what I'm going to do with this architecture?" or "how complex would it be to implement a complete project?".

I put this sample use case together to show what is the process to create a brand new service in the self service catalog, complete with all the implementation of the delivery of the service. My colleague Maxim Khavankin helped me to document all the steps.
If you download PSC and UCSD and run them with the evaluation license, you could run through this exercise very easily and make friends with the platform.

Hello World with Cisco ECS

I implemented a very simple service, just to have a context to show the implementation.
No business logic is involved, or integration with back end systems, to keep you focused on the mechanics: you can easily extend this framework to your real use case.

The idea is to order a service in PSC, providing a input, and let UCSD deliver the outcome.
In our case the expected result is writing a "Hello <your name>" message in the log file.

Generally workflows in UCSD make use of tasks from the library (you have more than 2000 tasks to automate servers, network, storage and virtualization). But a specific use case might require a task that is not available already, so you build it and add it to the library.

I created a custom task in UCSD just to write to the log: of course, you could replace this extremely exciting logic with a call to the REST API - or any other API - of the system you want to target: infrastructure managers in your data center, enterprise software systems, your partner's API for a B2B service, etc. 

 

Then I created a custom workflow in UCSD, that takes your name as a input and makes use of the task I mentioned to deliver the "Hello World" service. The workflow can be executed in UCSD (e.g. for unit testing) or invoked via the UCSD API.

Prime Service Catalog has a wizard that explores the API exposed by UCSD, discovers and imports all the entities it finds (including workflows) so that you can immediately offer them as services in the catalog for end users. Of course a customization can be added with the tools provided by PSC.

So the end to end procedure to create a business services is described by the following steps:

1. Create a custom task (if required)
2. Define a workflow that uses the custom task -> define input variables
3. Create a catalog item in UCSD -> offer the workflow from step 2
4. Synchronize PSC and UCSD
5. Use the wizard to import the service in PSC
6. Customize the service in the PSC catalog with Service Designer (optional)
7. Order the customized service
8. Check the order status on PSC side
9. Check the order status and outcome in UCSD

I illustrate every step with some pictures:

Create a custom task (if required)    

Custom tasks can be added to the existing library where 2000+ tasks are offered to manage servers, network, storage and virtualization.

You can group tasks in Categories so that they can be found easily in the workflow editor later. 

 

Custom tasks can have (optional) input and output parameters, that you define based on variable types: in this case I used a generic text variable to contain the name to send greetings to:

The format, contraints and presentation style can be defined:

You can skip the steps "Custom Task Outputs" and "Controller" in the wizard to create the task: we don't need them in this use case.

Finally we create the logic for our use case: a small piece of Javascript code that executes the custom action we want to add to the automation library.

The UCSD logger object has a method to write an Information/Warning/Error message to the UCSD log file. As I wrote earlier, you could use http calls here to invoke REST API if this was a real world use case.

After you've created your custom task it's available in the automation library.
Now you have to define a workflow that uses the custom task: to pass the input that the task requires, you will define a input variable in the workflow.

The workflow is an entity that contains a number of tasks. The workflow itself has its own input and output parameters, that can be used by the individual tasks.

Input and output parameters of the workflow are defined in the same way as tasks' input and output.

They can be useful if you launch the workflow via the REST API exposed by UCSD.

Now that you've created the workflow, it's time to add some tasks to it picking from the library (exposed in the left panel of the workflow editor).
We'll only add one task (the custom task that we created): select it from the library, eventually searching for the word "hello".
Drag and drop the task in the editor canvas, then configure it.

You will see a screen similar to this one:

 Configure the new task giving it a name:

Map the input variable of the task to the input parameter of the workflow that you created:

If you had not a variable holding the value for this task's input, you could still hard code the input value here (but it's not our case: this form would appear different if you hadn't mapped the variable in the previous screen).

 

The task does not produce any output value, so there's no option to map it to the output parameters of the workflow.

Finally we see the complete workflow (one single task, in our example) and we can validate it: it's a formal check that all the tasks are connected and all the variables assigned.

 

Then we can execute it from the same window, to check that it behaves correctly. You will have access to the log file from the same window that pops up when you execute the workflow, so you can see that the greetings appear in the log.

Next action is to expose this workflow to users in UCSD (in the GUI and via the API).

Create catalog item in UCSD -> offer workflow from Step 2   

UCSD catalog items are offered to non-admin users if you so choose. They are grouped in folders in the user interface, and you can make them visible to specific users or groups.

You can give them a name and a description and associate a service, that could be the provisioning of a resource or a custom workflow - like in our case.

The workflow is selected and associated here: 

After defining the new catalog item, you'll see it here - and in the end-user web GUI.

If the service is offered to technical users (e.g. the IT operations team), your work could be considered complete.
They can access UCSD and launch the workflow. The essential user interface of the tool is good enough for technical users that only need efficiency.

But if you're building a private cloud you might want to offer your end users a more sophisticated user interface and a complete self service catalog populated with any kind of services, where you apply the governance rules for your business.

So we'll go on and expose the "Hello World" service in Cisco Prime Service Catalog.

Synchronize PSC and UCSD   

Login to PSC as admin, go to Administration -> Manage Connections.
Click on the connection to UCSD (previously defined by giving it the target ip address and credentials) and click "Connect & Import".

PSC will discover all the assets offered by UCSD.
Now you can use the wizard to import the "Hello World" service in PSC. With few clicks it will be exposed in the service catalog.  

The wizard allows you to associate an image and a description with the service. Here you can describe it at the level of detail and abstraction that are more appropriate for your users (or customers).
You have a full graphic editor that does not require any skills as a web designer.

 

Additional metadata (attributes of the service) can be added, so that users can find it when searching the catalog: there is a search engine that PSC provides out of the box.

And finally you decide who can see and order the service in the catalog: you can map it to single users, groups, roles, organizations or just offer it to everyone.

 

At this time the service is fully working in the self service catalog and his lifecycle is managed. But, if you like, you can still apply customization and leverage the power of PSC. 

Customize the service in the PSC catalog with Service Designer (optional)   

There a subsystem in PSC, accessible only to specific user roles, that is called "Service Designer". It can be used to build services from scratch or to edit existing services, like the one that the wizard generated for us. Just go there and select the "Hello World" service.

The user interface of the service is built on reusable elements, that are called Active Forms (one active form could be reused in many services). The wizard generated a Active Form for our service, with a name corresponding to it.

 

If you select the active form and go to the panel "Display Properties" you can change the appearance and the behavior of the order form.

 

As an example the only input field, named "person", can be transformed into a drop down list with pre-populated items. Items could even be obtained from a database query or from a call to a web service, so that the list is dynamically populated.

 

The power of the Service Designer offers many more customization options, but here we want to stay on the easiest side so we'll stop here   :-)

Order the customized service   

Go to the home page of the Service Catalog. Browse the categories (did you create a custom category or just put the Hello World service in one of the existing categories?). You can also search for it using the search function, accessed via the magnifier glass icon.

In this picture you also see a review made by one of the users of the catalog that has already used the service. You can add your own after you've ordered it at least once.

You will be asked to provide the required input:

 

When you submit a request, your order is tracked in My Stuff -> Open Orders.

This is also used for audit activities.

Check order status on PSC side  

You will see the progress of the delivery process for your order: in general it has different phases including, if needed, the approval by specific users.

Check order status and outcome in UCSD     

If you go back to the admin view in UCSD (Organizations -> Service Requests) you will see that a new service request has been generated: double click on it to see the status.

if you click on the Log tab you can check the result of the execution of the service: the hello message has been delivered!

Now that you appreciated how easy is to build new services with PSC+UCSD you're ready to use all the features provided by the products and the pre-built integration that makes it very quick.

All your data center infrastructure is managed by UCSD, so that you can automate provisioning and configuration of servers, network and storage (of course, from any vendor and both physical and virtual). Once you've the automation done, offering services in the self service catalog takes just few minutes.

References

Cisco Enterprise Cloud Suite
and its individual components:
- Cisco PSC - Prime Service Catalog 
- Cisco UCSD - UCS Director

Become a cloud provider in 3 months

This is the story of a company that decided to become a Cloud Service Provider.
They were already a successful IT outsourcer in the financial industry, with many customers' environments running in their data center.
Outsourcing was a healthy business but they started having some challenges, due to slow and inefficient provisioning processes and operations.
Any new request from a customer started a new project, so their customers started exploring public cloud services to get more flexibility and speed.
For this reason, the company decided to adopt the cloud delivery model and to offer their customers a self service catalog.

Of course a cloud project cannot be done in one night, so they were cautious in their approach.
Both technology and operational processes needed to be proven before embarking in such a challenge, but the traditional waterfall methodology made the expected return appear uncertain and distant.
To make things worse, they had tried to implement a PaaS project with a different vendor and they had spent a lot of money without tangible return.

I was engaged to support the evaluation of a new IaaS catalog that could evolve to PaaS and to self service applications management.
I made sure that the Business and IT strategy were in sync and I proposed to start with small steps to validate the approach. I also invited them to qualify the quick wins that they would expect to justify the investment and show the stakeholders an immediate return, so that the project lived enough to reach the expected success.
As you know well, many projects last too much and die before showing any business return.

We analyzed the current situation and defined a future vision. This was the driver for a gap analysis and for the prioritization of user stories, that we decided to implement in short iterations (sprints of 2 weeks, according to the Agile Scrum methodology).
Their data center was mainly based on Cisco networks and servers, but this was not the main reason for selecting the Cisco software stack for the cloud project.
After the initial workshops, some product demo and talks about other projects they understood that our people - and our partner company that implemented the project with them - were experienced enough to plan the project seriously and to chase the quick wins that we all considered so important.

The Cloud Management Platform chosen for the project was the Cisco ONE Enterprise Cloud Suite (aka ECS).

One of the most important features considered in the decision was the possibility to create flexible templates, later exposed as self service options in the end user catalog, for the deployment of complex applications. A set of servers with different roles, and all the networks needed to make them work, can be provisioned as a dedicated and virtually separated environment (multi tenancy in a shared infrastructure that offers economy of scale).

As an example, the following picture shows a environment that could be ordered - fully configured - with a single click. It is based on a component of the ECS architecture that is named VACS (virtual application cloud segmentation):

 

It was easy to engage the SME (subject matter experts) for the servers, the network, the storage and the virtualization in the customer organization and to ask them to define the basic policies that we would use as building blocks for all the services to be offered.
This model-based implementation is quicker to build and easier to maintain, and it can be exposed to the end users in a way that they understand and trust soon.

The automation that we built was considered useful by the SME (after winning their initial suspicion, because every good craftsman loves manual work) because it set them free from the manual operations that previously made their work tedious and error prone.
Delegating the configuration to an automated service gave their customers a faster service and a higher quality (no rework needed because of manual errors or missing information).

One more component in the architecture is the Stack Designer.
It is a tool provided by the Cisco ECS to create templates for application provisioning. It takes IaaS templates - made in the infrastructure management layer, that in our case is UCSD, to deploy a topology of servers and networks - and layers the software stack on top of them.

You can decide what software products (or custom applications) must be installed - and configured based on the input parameters provided by the end user - including monitoring agents and backup agents, and save this new template in the repository.
The integration with Puppet, an open source solution used to provision software applications, is leveraged to install and configure the entire software stack from the images in the repository.

The new template can now be offered as a self service option in the catalog, so that the end users don't need to install and configure the software stack themselves. A end-to-end solution is provided, up and running and ready to be used.
All the components of the ECS solution are pre-integrated and this makes the project faster than you would expect. But, since they communicate through standard protocols and open API, every component of the architecture could be replaced by an alternative product (from a different vendor or from the open source community). You should not be afraid of vendor lock in  :-)

Agile Delivery

In terms of project delivery, the following table shows the different iterations that allowed to complete the delivery in only 3 months.
But the amazing result is that at every sprint (i.e. every 2 weeks) new use cases were available in a usable environment.
The first demo to a real customer (a customer of my customer) was done after 2 months from the start of the project, and the first customer was onboarded after the 5th sprint (i.e. 2.5 months).

Conclusion

This quick win demonstrates that even complex projects like building a public cloud platform can be done in a reasonable amount of time.
The era of endless projects, based on complex technology and measured in function points, has passed forever.
There are simple solutions (like ECS) that make your work easier, but a good organization and the right methodology allow for incremental building and refinement of the solution. Every iteration of the project delivers a usable result in the production environment, and you don't need to wait the completion of the entire project to start using the solution.
If you are a service provider, you can start selling your services soon and produce a ROI.
More services will be added incrementally and the catalog will be richer at every iteration.

References

Cisco Enterprise Cloud Suite
and its individual components:
- Cisco PSC - Prime Service Catalog 
- Cisco UCSD - UCS Director
- Cisco VACS - Virtual Application Cloud Segmentation

Fast IT
Cisco Prime Service Catalog in action: Cisco eStore

Scrum (agile development)