A Description of a BPM Architecture

In the course of my work as an architect, working as I do for a consultancy specialising in integration and business process management (BPM), I am often called upon to design solution architectures in the BPM space and also to lead development teams delivering these solutions.  I thought I'd take the opportunity here to give a brief introduction to BPM architectures, the elements contained within them and illustrate this with an example architecture.

What is BPM?

Without trying to copy the Wikipedia entry, Business Process Management systems are those that are implemented to support business processes in their wider sense, as opposed to "Line of Business" systems that are traditionally more functionally aligned and therefore operate as "silos" of data.  Whilst BPM systems do not remove the need for or relevance of line of business systems, the aim is to combine such systems into a cohesive whole.

Business processes will be started, either manually or automatically, and require a certain trigger information set to kick them off.  This information set is then added to and manipulated by the business process until the process completes.  Sometimes the data itself is the objective of the business process, sometimes it just refers to physical things created as a result of the business process (order information, serial numbers od widgets, courier tracking IDs) and sometimes just records of things that have occurred during the business process (notes on phone conversations, emails).

Essential elements of BPM solutions

What is common to BPM solutions is that these activities are co-ordinated into a process.  Some activities are automated by software systems, some of them are manual, but the timing of the activities and the information presented to the activities is co-ordinated.  In order to achieve this some form of process logic (e.g. workflow in its wider sense) is required.  At this point it is not necessary to specify any specific technologies but we do need to have the process driven by our system.

When we engage people in out activities, we will need to have a user interface for them to interact with.  Human activities will usually be triggered from some sort of notification that points to items in a task list, but once the people engaged in the process have started work on their task they will usually be presented with information on (a) what their task is, (b) the input data relating to that task and (c) form fields where they record the output data for the task.   This is all achieved by the user interface.

As mentioned above, there is a set of data that encompasses the overall business process, and we need to manage the superset of this information quite closely with our process.  This may require us to maintain pointers to records in other systems, it may require us to store duplicate information and it may require us to store master information, but a key feature of a BPM solution is that somewhere there is a central point from which we can reconstruct the data for the process.

Also, mentioned above, we said that line-of-business systems may still exist, but that they operate within functional boundaries in an organisation.  Our BPM solution crosses functional boundaries, but that does not mean that an activity relating to a business function cannot interact with a line of business system.  Interactions with line of business systems come in two main forms.  One is human interaction.  A task may be to update or manipulate data in a separate system, the so-called "swivel-chair integration".  These types of integrations with line of business systems are the cheapest to implement because they are just co-ordinating manual activity, but they are the most expensive to run because the costs scale with usage as they require human activity.  In order to improve efficiency we must, wherever cost-effective to do so, make the interactions with line-of-business systems automated.  This requires us to have an integration capability.

Putting it all together

It's now time to take a look at what such a system might look like in practice from a very high-level point of view:

You will probably see straight away from my diagram that my example BPM architecture is build upon Service Oriented Architecture (SOA) principles.  It is impractical in a modern software ecosystem to achieve a BPM architecture without SOA.  Also, as I design architectures on the Microsoft .Net platform, my services these days are typically WCF rather than basic .asmx SOAP services.  

Let's take a look at some of the elements in the diagram and fill in some of the details.

Application Services

In a BPM architecture, the application services are the heart of the system.  This is where the custom data access resides, to store our superset of the process data.  The application services also act as an event processor, because (drawing on the concepts of event-driven architectures or EDA) we can view our interactions with the other parts of the systems in an event-style context (although in this case "events" really correlate to messages).

Note also that the main elements of the system are decoupled through their service interfaces.  None of the components of the system needs to have any knowledge of the other parts except for how to exchange messages with them.  Note also that some of the service interactions are bi-directional.  No-one ever said that SOA needed you to have separate publishers and consumers of web services.  In may ecosystems there may be a conversation of messages and this requires systems to both publish to and receive messages from other systems.  [Note:  If this type of interaction is complex or involves a high number of systems, look at implementing an enterprise service bus (ESB) pattern.]

Something that you may think of when you see the diagram for application services and see its three separate service endpoints is "you must have to duplicate a lot of service calls to support those three service endpoints".  Well yes, and no.  Remember that we will still in effect only have one business logic layer, and any of the service endpoints may expose some of that logic.  If we have to duplicate an operation on a service endpoint then it's not a problem because the underlying code is shared.  By keeping the service endpoints separate however we are forced to think more carefully about what operations each service needs to support, and we can make the security on the system more granular.

Workflow Services

This part of the system is responsible for the process flow in the system.  It will also contain the task lists for the human workflow and will instruct any automated tasks, usually by making calls back to the application services.  These calls will then either be routed to other services for fulfilment or handed off to the integration services.

There are a number of technologies that may host the workflow services.  For highly automated systems this may be performed using the orchestration engine in BizTalk server.  For simple applications you may write and host WF workflows, or use one of the workflow systems built on top of WF such as K2 Black Pearl.

The key to this being a success lies in correctly decoupling the workflow logic from the application logic.  It is very easy to end up with a horrific merge between the two.  This is why I prefer to provide an endpoint on the application services for the workflow to use, so that the workflow contains the minimum amount of data possible (e.g. keys), and the workflow maintains the process state whereas the application services maintain the business data state.  

Presentation Services

The UI for the system can take many forms, but in current systems it is usually a web UI.  Note that some workflow systems provide a UI to view task lists and these are sometimes used out of convenience (hence the dotted line in the diagram), but in a more purist view of the architecture the worflow will be an "engine" that is not accessed directly by the UI.  The task lists will be retrieved by the application services and rendered into the UI separately.  

As for technologies, it is now becoming common for BPM presentation services to be hosted within SharePoint as this technology can handle authentication, authorisation and end-user customisation out of the box.  Alternatively, a custom ASP.Net website or portal may be used. 

Integration Services

This part of the system abstracts the interaction with other systems and associated automated processes, and presents the application services with a clean interface to use.  Interactions with other systems can require long-running asynchronous processes that require a separate runtime, and they may require the data in the original message to be transformed significantly.  

In the world of .Net, the technology of choice for this part of the ecosystem is BizTalk Server, and the new versions of BizTalk (2006 R2 and 2009) have excellent WCF support, along with the traditional BizTalk strengths of orchestration and message transformation.  It is rare to find a back-end system that cannot be integrated in some form using the BizTalk toolkit.

A walk-through BPM scenario

I am going to illustrate BPM in practice by overlaying a simple order fulfilment process on top of the architecture described above.

Imagine that we have a business that sells widgets to consumers.  We may have a warehouse system, a financial system and a CRM system to interact with.

The process starts when a customer goes onto our website and orders something.  In this model our website is actually just another line-of business system.  This results in a message being sent to our Integration Services which receives it reliably and forwards it on to the Application Services which then creates a new header record for the order and generates a uniqe key.  The application services then immediately fires up an instance of the appropriate workflow via a call to WorkFlow Services.  

The activities that need to happen after this are several, and can be executed in parallel.  The order needs to be entered into the CRM system (via Integration Services) and the financial system  (via Integration Services) , and the warehouse system needs to be queried as to whether it has stock to fulfil the process  (via Integration Services) .  If not we may need to trigger an ordering process and wait for delivery, or else we may want to instruct the warehouse system to pick the stock and dispatch  (via Integration Services) . 

On dispatch we may then create a manual task  (within Workflow Services)  for the warehouseman to enter the courier's reference into the system for tracking (using Presentation Services).  We may then launch a long-running process that queries the courier's system every 2 hours until delivery notification is achieved (combination of Workflow Services calling through to Integration Services via Application Services). 

Summary

This simple example is not intended as a definitive design for BPM, and there are whole areas that we have not touched upon that are important factors in a BPM solution such as business rules, business activity monitoring (BAM) and feedback control loops.  We haven't even mentioned anything about process design, simulation and modelling either.  All I have been trying to achieve is to provide a small illustration of the sort of architecture you might find in a BPM solution and to fill in some of the conceptual gaps in the model.  

Hopefully before too long I will get some time to go into more detail about this important and emerging architectural topic.