I am just playing around with a new image at the moment that has BizTalk 2009, SQL 2008 and Visual Studio 2008.
I thought that in order to try out some of the features of BizTalk 2009 that I would make a sample solution that contained a WCF service, BizTalk orchestration and associated schemas for calling the service and mapping the result.
I just happened to have forgotten a field in a data table behind my service. Not to worry I thought, just add the column and then regenerate the LINQ .dbml. However, my table has timestamps and audit stuff like that in the last columns and I wanted to put my new data field further up, so using the UI on SQL Management Studio 2008 I duly inserted a column, set the properties and clicked on save. What I got was the following error in a dialogue:
"Saving changes is not permitted. The changes you have made require the following tables to be dropped and re-created. You have either made changes to a table that can't be re-created or enabled the option Prevent saving changes that require the table to be re-created."
Bugger. What can the matter be? Why can't I even save a table? It turns out that there is a setting in SQL Server Managemet Studio that won't allow table changes if they result in a recreation of the table. Note that I inserted a column between existing columns in the database. For years now, behind the scenes SQL Server has generated a script that removes constraints, renames the current table, creates the new version of the table and then bulk loads the data into it, finally dropping the original (and now renamed) table. Of course, when you have a production system this sort of script can be a very dodgy thing to run while the database is operating as you lose referential integrity while the script is running. Hence it is a "good thing" for it to be disabled by default. However, for a dev environment, especially one where I am doing a proof of concept....nah. No thanks.
In the course of my work as an architect, working as I do for a consultancy specialising in integration and business process management (BPM), I am often called upon to design solution architectures in the BPM space and also to lead development teams delivering these solutions. I thought I'd take the opportunity here to give a brief introduction to BPM architectures, the elements contained within them and illustrate this with an example architecture.
What is BPM?
Without trying to copy the Wikipedia entry, Business Process Management systems are those that are implemented to support business processes in their wider sense, as opposed to "Line of Business" systems that are traditionally more functionally aligned and therefore operate as "silos" of data. Whilst BPM systems do not remove the need for or relevance of line of business systems, the aim is to combine such systems into a cohesive whole.
Business processes will be started, either manually or automatically, and require a certain trigger information set to kick them off. This information set is then added to and manipulated by the business process until the process completes. Sometimes the data itself is the objective of the business process, sometimes it just refers to physical things created as a result of the business process (order information, serial numbers od widgets, courier tracking IDs) and sometimes just records of things that have occurred during the business process (notes on phone conversations, emails).
Essential elements of BPM solutions
What is common to BPM solutions is that these activities are co-ordinated into a process. Some activities are automated by software systems, some of them are manual, but the timing of the activities and the information presented to the activities is co-ordinated. In order to achieve this some form of process logic (e.g. workflow in its wider sense) is required. At this point it is not necessary to specify any specific technologies but we do need to have the process driven by our system.
When we engage people in out activities, we will need to have a user interface for them to interact with. Human activities will usually be triggered from some sort of notification that points to items in a task list, but once the people engaged in the process have started work on their task they will usually be presented with information on (a) what their task is, (b) the input data relating to that task and (c) form fields where they record the output data for the task. This is all achieved by the user interface.
As mentioned above, there is a set of data that encompasses the overall business process, and we need to manage the superset of this information quite closely with our process. This may require us to maintain pointers to records in other systems, it may require us to store duplicate information and it may require us to store master information, but a key feature of a BPM solution is that somewhere there is a central point from which we can reconstruct the data for the process.
Also, mentioned above, we said that line-of-business systems may still exist, but that they operate within functional boundaries in an organisation. Our BPM solution crosses functional boundaries, but that does not mean that an activity relating to a business function cannot interact with a line of business system. Interactions with line of business systems come in two main forms. One is human interaction. A task may be to update or manipulate data in a separate system, the so-called "swivel-chair integration". These types of integrations with line of business systems are the cheapest to implement because they are just co-ordinating manual activity, but they are the most expensive to run because the costs scale with usage as they require human activity. In order to improve efficiency we must, wherever cost-effective to do so, make the interactions with line-of-business systems automated. This requires us to have an integration capability.
Putting it all together
It's now time to take a look at what such a system might look like in practice from a very high-level point of view:
You will probably see straight away from my diagram that my example BPM architecture is build upon Service Oriented Architecture (SOA) principles. It is impractical in a modern software ecosystem to achieve a BPM architecture without SOA. Also, as I design architectures on the Microsoft .Net platform, my services these days are typically WCF rather than basic .asmx SOAP services.
Let's take a look at some of the elements in the diagram and fill in some of the details.
In a BPM architecture, the application services are the heart of the system. This is where the custom data access resides, to store our superset of the process data. The application services also act as an event processor, because (drawing on the concepts of event-driven architectures or EDA) we can view our interactions with the other parts of the systems in an event-style context (although in this case "events" really correlate to messages).
Note also that the main elements of the system are decoupled through their service interfaces. None of the components of the system needs to have any knowledge of the other parts except for how to exchange messages with them. Note also that some of the service interactions are bi-directional. No-one ever said that SOA needed you to have separate publishers and consumers of web services. In may ecosystems there may be a conversation of messages and this requires systems to both publish to and receive messages from other systems. [Note: If this type of interaction is complex or involves a high number of systems, look at implementing an enterprise service bus (ESB) pattern.]
Something that you may think of when you see the diagram for application services and see its three separate service endpoints is "you must have to duplicate a lot of service calls to support those three service endpoints". Well yes, and no. Remember that we will still in effect only have one business logic layer, and any of the service endpoints may expose some of that logic. If we have to duplicate an operation on a service endpoint then it's not a problem because the underlying code is shared. By keeping the service endpoints separate however we are forced to think more carefully about what operations each service needs to support, and we can make the security on the system more granular.
This part of the system is responsible for the process flow in the system. It will also contain the task lists for the human workflow and will instruct any automated tasks, usually by making calls back to the application services. These calls will then either be routed to other services for fulfilment or handed off to the integration services.
There are a number of technologies that may host the workflow services. For highly automated systems this may be performed using the orchestration engine in BizTalk server. For simple applications you may write and host WFworkflows, or use one of the workflow systems built on top of WF such as K2 Black Pearl.
The key to this being a success lies in correctly decoupling the workflow logic from the application logic. It is very easy to end up with a horrific merge between the two. This is why I prefer to provide an endpoint on the application services for the workflow to use, so that the workflow contains the minimum amount of data possible (e.g. keys), and the workflow maintains the process state whereas the application services maintain the business data state.
The UI for the system can take many forms, but in current systems it is usually a web UI. Note that some workflow systems provide a UI to view task lists and these are sometimes used out of convenience (hence the dotted line in the diagram), but in a more purist view of the architecture the worflow will be an "engine" that is not accessed directly by the UI. The task lists will be retrieved by the application services and rendered into the UI separately.
As for technologies, it is now becoming common for BPM presentation services to be hosted within SharePoint as this technology can handle authentication, authorisation and end-user customisation out of the box. Alternatively, a custom ASP.Net website or portal may be used.
This part of the system abstracts the interaction with other systems and associated automated processes, and presents the application services with a clean interface to use. Interactions with other systems can require long-running asynchronous processes that require a separate runtime, and they may require the data in the original message to be transformed significantly.
In the world of .Net, the technology of choice for this part of the ecosystem is BizTalk Server, and the new versions of BizTalk (2006 R2 and 2009) have excellent WCF support, along with the traditional BizTalk strengths of orchestration and message transformation. It is rare to find a back-end system that cannot be integrated in some form using the BizTalk toolkit.
A walk-through BPM scenario
I am going to illustrate BPM in practice by overlaying a simple order fulfilment process on top of the architecture described above.
Imagine that we have a business that sells widgets to consumers. We may have a warehouse system, a financial system and a CRM system to interact with.
The process starts when a customer goes onto our website and orders something. In this model our website is actually just another line-of business system. This results in a message being sent to our Integration Services which receives it reliably and forwards it on to the Application Services which then creates a new header record for the order and generates a uniqe key. The application services then immediately fires up an instance of the appropriate workflow via a call to WorkFlow Services.
The activities that need to happen after this are several, and can be executed in parallel. The order needs to be entered into the CRM system (via Integration Services) and the financial system (via Integration Services) , and the warehouse system needs to be queried as to whether it has stock to fulfil the process (via Integration Services) . If not we may need to trigger an ordering process and wait for delivery, or else we may want to instruct the warehouse system to pick the stock and dispatch (via Integration Services) .
On dispatch we may then create a manual task (within Workflow Services) for the warehouseman to enter the courier's reference into the system for tracking (using Presentation Services). We may then launch a long-running process that queries the courier's system every 2 hours until delivery notification is achieved (combination of Workflow Services calling through to Integration Services via Application Services).
This simple example is not intended as a definitive design for BPM, and there are whole areas that we have not touched upon that are important factors in a BPM solution such as business rules, business activity monitoring (BAM) and feedback control loops. We haven't even mentioned anything about process design, simulation and modelling either. All I have been trying to achieve is to provide a small illustration of the sort of architecture you might find in a BPM solution and to fill in some of the conceptual gaps in the model.
Hopefully before too long I will get some time to go into more detail about this important and emerging architectural topic.
I'm back from my easter break and just spending a couple of days getting to grips with BizTalk 2009. I have created a new virtual image with the following on it:
Windows Server 2008
SQL Server 2008
Visual Studio 2008
BizTalk Server 2009 Developer Edition
I have encountered an issue when configuring the BizTalk BAM Portal. All of the usual features of BizTalk, including BAM itself, have configured without any problems, but when I try to configure the BAM Portal on the default web site I get the following error message when I double-click the error icon:
Failed to validate BAM Portal Web Site (BAMPortal)
For help, click: http://go.microsoft.com/fwlink/events.asp?ProdName=Microsoft+BizTalk+Server+2009&ProdVer=3.8.368.0&EvtSrc=BAMPortal&EvtID=idsFailedToValidateWebsite
The BAM Portal website Default Web Site is not valid. (Microsoft.BizTalk.Bam.CfgExtHelper.PortalHelper)
For help, click: http://go.microsoft.com/fwlink/events.asp?ProdName=Microsoft+BizTalk+Server+2009&ProdVer=3.8.368.0&EvtSrc=Microsoft.BizTalk.Bam.CfgExtHelper.PortalHelper&EvtID=error_InvalidPortal
This is just a short post to say that I ran into a problem last week (caused by the same application that I am posting my Joys of Legacy Software series about).
The web application we have been upgrading uses the Microsoft UIProcess Application Block. When we develop we usually use local IIS rather than the Visual Studio web server because this flushes out permissions problems more quickly, and in a large development team you don't end up with people inadvertently changing the server ports on the local web sites and thus screwing up everyone's service references. In turn, using IIS means that all of your web projects tend to be in separate virtual directories under the web root, and so we never ran the application as a rot website.
Now, one of my team then went to deploy the application for testing on the client's site and the client requested that it was deployed in a separate website on a different port number. So far so good, we have to perform the install and then manually change the root website's home directory but no great shakes here. Then on testing it was found that the UIProcess was failing. This is because of a bug in the way that it is coded, where the code assumes that there is a virtual directory and does not handle being in the root website.
We didn't actually implement the fix line for line, because I did not want to have to change the config as per the fix, but it was enough of a pointer to get started.
It's amazing really how these things come back to bite you. Working for a Microsoft Gold Partner, I usually feel that I am working on fairly cutting-edge stuff most of the time, and occasionally a project like this comes along with some old technology that's too deeply embedded to strip out, and all sorts of probems arise. Not least because no-one is using these application blocks any more (are they? are you?).