During the early days of VisibleThread in Spring 2008, I used to carry around a positioning slide deck. This, I shared with interested parties; potential investors and early adopters mostly, outlining the VisibleThread value proposition. In it, I would outline high profile, troubled programs.

I would pinpoint how automated defect identification in documents would have allowed program managers not only identify, but reign in under-scoped & creeping projects & put in place effective project controls. The net effect was to avert out of control scenarios, avoiding programs being placed in jeopardy. Interestingly, two years later this remains our core proposition.

One of the examples I tended to cite was the US Census Bureau with its FDCA (Field Data Collection Automation) program aimed towards automating household data collection for the 2010 US census. The program was initiated in 2006 and by 2008 was in deep trouble. FDCA was all about equipping agents with handhelds that would automate the collection of data at the doorstep, cutting out the big expense associated with manual collection and the subsequent analysis of the data.

The quote on my slide deck came from the US Census Bureau’s Steve H. Murdock, when testifying before the Subcommittee on Commerce, Justice, Science, and Related Agencies Committee on Appropriations, U.S. House of Representatives in April 2008, some two years after the contract for implementation was originally put out for tender and awarded in 2006.

“From the beginning, we did not effectively convey to the contractor the complexity of census operations, and the detailed requirements that needed to be fulfilled in order to complete the operations that FDCA covers.  Once these detailed requirements were completely delineated, we had serious concerns about rising costs and our ability to complete a successful 2010 Census if we continued developing the FDCA program as planned.”

At the time, I remember digging into the details of his testimony along with some general research into the program itself. It became apparent that this specific program failure accounted for between $2 to $3 billion of spend.

So, recently I was travelling on the subway in New York and spotted an ad encouraging participation in the census. It brought back memories of my research and it prompted me to wonder how the census was going from a back office or IT program perspective. Did they ever automate any of it I wondered?

I went to the census website to discover the following little statement at the bottom left of the ‘How it works’ section on the 2010 home page: ‘Can I fill out my form online?’ The answer being ‘No, not at this time. We are experimenting with internet response for the future.’

One wonders if back in 2006, had prioritization been given to offering a secure online capability and more importantly, if program management had leveraged an effective PMO & review process with the vendor, could this failure have been averted?

My guess is that; had the collection of program documents for each project including project charter, BRD, vision etc. been analyzed by VisibleThread, very early warning signals would have been raised. This relates particularly to automated ‘discovery’ and ‘concept tracking’.

Now whether anybody at executive level would have paid attention is a whole other matter entirely.

At least, I can say that our current crop of Fortune 500 customers, mostly in the Financial Services sector, are seeing the merit of early warning signals for their programs and projects. They are equipped with the information that keeps programs on track scope wise. ‘Preventative medicine’ can often be a hard sell at executive level, but enlighted and battle weary stakeholders see the merit. Maybe with a few more $2 billion fiascos, we might make headway as an industry. We can but live in hope.

Until next time,
Fergal.

PS: For more context on the whole affair check out: http://spectrum.ieee.org/riskfactor/computing/it/census_going_back_to_paper_due

At VisibleThread, we are fortunate to have the opportunity to observe PMOs (Project Management Offices) at first hand, particularly so as we work with many of the leading players in the FS (Financial Services) sector.

I recently came across a nice analogy on the PMI site at: www.pmi.org/Pages/PMO-Growing-Pains.aspx comparing a PMO with the human lifecycle. The basic assertion is that PMOs are born, grow up and hopefully end up as mature, ‘adult’ PMOs.

Applying the analogy to the PMO highlights the importance of ‘good parenting’ in terms of how a PMO is set up (i.e. born) and how it is nurtured and evolves. In fact, many PMOs stay at infant stage, not progressing much beyond the basic idea of a group of project managers with no particular strategic imperitive.

Effective PMOs serve to chaperone the program to success, avoiding the inevitable speedbumps along the way to program delivery.

Is this your PMO?

On the other hand, immature and ‘infantile’ PMOs are weak & the first likely to get pruned if not eliminated altogether when the organisation is looking to achieve efficiencies. In this context, they lack the maturity to make an effective business case to the wider family and so are extremely vulnerable.

So, if a PMO struggles to mature, can we blame the PMO itself or is it lack of supports in the broader family? In a sense, this is the ‘nature versus nurture’ argument. This is a bigger question that touches on the degree of executive level support for the PMO along with the PMO leadership itself. The better the support structures, the more likely you will have a mature PMO. Of course talented PMO leadership is the second clear contributory factor to PMO success.

So, what does a mature PMO look like and do you belong to one?

Well, in our experience, aside from operation aspects of vendor management etc. mature IT oriented PMOs fulfill three fundamental objectives:

A.) they contribute to successful project management in measuable ways

B.) they ensure projects align with strategic goals articulated at program level

C.) they set effective and appropriate standards for processes and methodologies within the program

To achieve these objectives arriving at the a good level of maturity requires quantifiable & systematic metrics. Here is the rub! PMOs can easily sink in deep documentation and unduely onerous processes attempting to achieve maturity.

Since projects and program goals are typically expressed in terms of vision docs and BRDs, there are pretty simple concrete steps that can tangibly help a PMO out of the infant stage to a more mature level avoiding the documentation overload problem. The answer: more consistent project documentation supported by automation using solutions like VisibleThread. Automated vetting of documents for poor quality is helping drive the PMO from ‘infant’ to ‘mature adult’ with a lightweight approach.

Well, it’s been a very busy last few months. Between getting our 1.2.1 release out and working closely with our key accounts it’s been full on.

So, I’ve come up for air in the last few days and I wanted to take the opportunity to share some thoughts relating to medium to large programs in IT and some challenges I’ve being seeing at first hand.

Many of the people we work with are involved in programs typically coordinating between 5 to 15 projects. These roles are variously referred to as ‘program managers’ or sometimes ‘program PMOs’ (Program/Project Management Offices). They look to VisibleThread to help assess the quality of documents for these initiatives.

For instance, one of the large programs comprises 15 separate projects spanning 3 separate locations globally. In this case, we have 15 vision statements, 15 BRDs (Business Requirements Definitions) and will in due course have multiples of 15 for other types of documents including FRDs, Arch specs, Test Plans etc.

This program may well ultimately yield 15 x 5 (1 for each type) documents, i.e. 75 separate docs. If we assume that each doc averages 24 pages, we’re looking at 1800 pages of content. This is, believe it or not, roughly the size of the King James bible! (well at least this edition: http://www.christianbook.com/21st-century-king-james-bible-hardcover/9780963051233/pd/53495 )

Consider this from an individual author’s perspective. A project manager authoring her own project BRD (assuming it’s one of the 15) will likely be conscious of the considerations involved in that specific project but less conscious of sibling document sets for adjacent projects that are part of the program.

The challenge of assessing risk within the content is obvious; different authors, different audiences, different levels of detail, potentially large volumes of content making gap analysis and interdependency analysis very difficult indeed.

Working with people on the ground, this has been one of the biggest issues at the overall PMO or Program Management role level. One customer mentioned an example where the far east team had a definition of ‘client’ that was subtly different from the US version. Unravelling this interdependency did not occur until into the UAT phase and the consequent fallout was material in the context of the overall program, resulting in cost due to rework and a significant delay in the delivery timeline.

Dependencies typically go in two directions, horizontally across sibling documents (eg: Proj ‘A’ BRD relating to Proj ‘B’ BRD) and vertically, down to different document types (eg: Project ‘A’ Vision doc relating to Project ‘A’ FRD) and different sections in these docs. These are clearly non-trivial challenges and very difficult in the context of medium to large program initiatives.

Can’t we use a Traceability Matrix?

Many people within the business analysis community tend to suggest using  a Trace Matrix as one way to tackle these issues. A fairly simple image of such a matrix (taken from wikipedia) is shown to the right. Effectively a trace matrix seeks to identify the connections between numbered requirements.

In my experience, trying to use a trace matrix for the above issue is near to impossible. Three factors make this so: 1.) the effort to create and maintain an up to date trace matrix in light of changing underlying document content is unduly onerous, 2.) a trace matrix is not scalable visually beyond the simplest of initiatives, certainly not with 1800 pages of content and finally and perhaps most importantly 3.) expecting senior stakeholders such as program managers to go to the effort of maintaining such a trace matrix is ‘a bridge too far’ in most organisations and will simply not happen.

So, enter the ‘discovery’ view within VisibleThread. Discovery takes a different approach. Rather than forcing an explicit marking of ‘traceable’ relationships, it instead automatically scans a body of documents and calculates the occurrence of ‘nouns’ in the context of where they occur within document content.

How does this work?

The VT Server uses NLP (natural language processing) to automatically scan raw documents and store occurrences. The dashboard then displays these occurrences cross referencing the document content.

Let’s say we are looking at documents concerning an extension to an existing trading system that will affect business rules around how accounts are validated for trading thresholds. In this case you would expect to see a reasonable distribution of certain key domain concepts like ‘trade’, ‘dealer’, ‘letter of credit’, ‘account’ etc. spread across certain documents in certain distribution frequencies. In particular I would expect to see quite a bit of content and rules relating to ‘account’. The BRD, FRD and use cases that touch account manipulation eg: ‘Update Acount’ and associated test cases would all be expected to have relatively high occurrences of the concept ‘account’.

Looking at this screenshot, we see the list of ‘discovered’ nouns.  The grid in the center shows the correlation between concepts (nouns) like ‘account’ and in what documents they occur. Each document is represented by a column adjacent to the checkbox. Here we see 5 columns representing; a BRD doc, a tech spec doc and 3 use case docs.

In our case, only 2 occurrences of ‘account’ are encountered indicated by a green dot in the specific document column. Those occurrences are *only* in the BRD doc meaning we have no reference to ‘account’ in the detailed tech spec doc or in any of the use case docs, a potentially serious gap. We can spot gaps/issues without having to trudge through the entire set of documents in this way. For program managers at high levels, this ‘surgical’ view focussing on inherently important concepts top down, means risky gaps and defects can be identified and eliminated.

Were these types of gaps to remain undiscovered in our ‘bible’, (as they were in the above ‘client’ example) we would be in for a rude awakening downstream. Thankfully, with VisibleThread we are seeing how we can avoid these issues in very practical and intuitive ways, marrying NLP with strong visualisation techniques.

all the best,

Fergal.

PS: Over the years I have encountered some insightful people attempting to use mind maps to understand dependencies relationships. It is certainly a good idea on the surface but just like the trace matrix challenge, the mind map itself (albeit electronic in nature) must be separately maintained outside of the content. Just like trace matrices, this is hard to do when the body of content becomes in any way large.