Workflows demystified

If you have ever worked on a project that involves building an application for a business process, then you definitely have heard of or used a workflow. A workflow is nothing but a set of steps - with rules that define which step comes after the current step, until you complete all the steps and the result is achieved. So, if you said go to the shop, get a notepad, do your homework on it, and then turn it in for review - that is a simple workflow. This one stops at submitting for review, but if I said that once you get the review result, then go and do something else - then we have added a step in between which requires someone to come in and manually review your work. This is still a workflow, but one that involves a manual approval step.

Business Process (EPC) in ARIS Express 3

Workflows are used in any business critical application where one or more persons need to review or approve a certain business outcome that is generated by a software, or maybe something entered by the user and which needs to go into the application for processing or record keeping. Workflows are facilitated by software known as workflow engines like TIBCO iProcess etc.

In this post we will try to understand how these things work internally.

So what is a workflow and what does it look like?

Like I said before, workflow is a set of steps that need to be executed to achieve a business process. These steps can be automatic steps that get or set data in an application or a manual step which involves review and approval or user input.

The key components of a workflow are

  • A set of independent and well defined steps or tasks that can be executed based on just the input parameters and no additional dependencies
  • A set of rules that define the transitions from one step to another within the workflow
  • An initial input that starts the fist step and a final place holder for the result.
  • If there is a manual task, then clear instructions on how that task can be completed
Once we define a workflow with these in mind, there needs to be some way of executing these workflows - execute them correctly, again and again and each time complete the flow. This is facilitated by a workflow engine.
The key components of a workflow engine are -
  • A way to define and store a workflow with both automatic and manual steps.
  • A way to inspect and act upon the manual steps
  • Logic to understand and implement the state transition between the steps
  • A way to capture the output of each step and pass that as input to the next step
  • In case of manual steps, a way to queue the tasks and pause execution until this manual step is executed.
  • A state management mechanism which can tell at any given point the number of instances of that workflow being executed and the state of each flow - which step they are executing or which manual step the are waiting for to complete
These are just a few of the many features of a workflow and workflow engine.

So how are these things implemented anyway?

Commercial and Open Source workflow engines are implemented in different programming languages with varying complexities. Many provide authentication and other additional features and some provide integration into other enterprise tools built by that vendor.

But generally, these do the following

  • Provide a GUI to define and store workflows - these workflows are stored as XML or BPEL or some other proprietary formats
  • They also provide the ability to choose and name how the intermediate state and results are stored
  • They provide the ability to assign queues where tasks waiting for manual approval are put
  • They also provide a GUI which can be used to inspect the manual tasks and review or approve them.
  • They provide a way to define a hierarchy and assign tasks as per roles
  • Some provide escalation mechanisms where if a certain task does not get completed in a certain time, then it gets flagged and raised to the next person in the hierarchy.

Apart from this these tools provide error handling to recover from issues and trigger correction flows and also admin functionalities to let technology teams manage the workflows better.

State management is generally achieved using a database.

What about runtime?

At runtime, it is just a matter of starting threads that monitor each input or queue and work through the tasks. The tasks need to be executed in the most efficient manner. There is generally no need for workflows to work super fast - as long as they correctly implement the business process its good enough.

Also, workflows work in co-ordination with different other enterprise systems - so it might not be possible to get things done unless that system responds.

What next?

This post has just been a theoretical explanation of workflows - in the next post we will try to build a simple workflow engine and get more insight into how it works.

Written on October 8, 2011