How to Perform an Architectural Review

So, here I sit in the terminal at O'Hare International Airport, waiting for my flight back home to Texas.  I am here on business to perform an architectural review of an existing system that manages what we call the "pay and chase" process.  The two parties involved in "pay and chase" are health care providers and health care payers.  Sometimes payers overpay providers for services, and want that money back.  My company offers products and services that make this possible, and I just spent three days learning all about it.

The reason I didn't know about it before is because the "pay and chase" system is brand new to our company, having just recently been acquired.  Of course the big issue now is determining how to integrate it into our overall suite of products and services, and for us on the I.T. side, how to homogenize and leverage it's functionality to the rest of our offerings.  So, guess who does that kind of work?

So there I was, tasked with developing a "road map" for homogenizing several disparate systems to maximize our economies of scale and scope, with nary a clue on what those systems do, let alone what they are composed of.  So I decided the best course of action was to perform an architectural review. This way I could begin to get my head around just how I might approach its integration with the overall enterprise, how it might leverage existing systems, and be leveraged by those existing systems.  Who knows, if I do my job well, I just might create some new business possibilities..

So, if you ever find yourself in a situation like this, and need to learn what you have to work with, I recommend the following items, all tailored to explore the breadth of a system; depth can come later.  No point in trying to add flesh to something you don't have a skeleton for.  Without a framework, details just "float" meaningless...

• Identify the system's major components and their roles
• Identify the relationships/dependencies between those components
• Identify the inputs into, and outputs from those components
• Determine the major business objects of the system
• Identify any process workflows in the system.  Are they synchronous?  Asynchronous?
• Explore the security model of the system
• Explore how the system scales

And again, only after gathering these critical "big picture" items, can you then begin to decompose them into smaller and smaller parts.

Finally, one last word of advice.  DO NOT DISPARAGE THE SYSTEM!!  Do not, under any circumstance, utter words that make it sound like you do not approve.  All this will accomplish is to keep the folks you are dependent upon for information, silent, fearful and defensive, which is not the environment you need to create to get the best information.

Let's face it, it is almost cliche that we always believe that everyone else's software sucks, and that ours is awesome.  Even though the architecture you reveal may be very poor, you do not know the circumstances and environment in which it was designed and developed.  I've seen countless systems that I believed were sub-par, and the vast majority of them were designed and developed under the pressure of an artificial sales deadline (see my post "You Sold What?"), which we all know puts intense pressure on IT team to cut corners.

If anything, acknowledge this reality, and watch those you are working with become your best friend, eager to help you understand the "as is", so you can work towards a better "go forward".

You Sold WHAT?!?

So there you are, a sales person in a high-pressure sales meeting, demonstrating to the customer all of the awesome features of the system, and showing them stunning presentations about ROI and scalability.  Everything is cooking right along, and you can almost smell the big fat commission coming from this sale.  After you're done, the customer takes it all in, leans back in their chair and asks, "Can it do...."?

Can it do...?  The moment of truth arrives.  If you say "no", you risk losing the sale and your commission.  If you say "yes", then your development team is going to have some work to do.  So, you take the easy and profitable path and say "Yes, we can make it do that by..."  The customer is thrilled and signs the deal.  Your business has made another sale, acquired another prestigious customer and you get paid.  Awesome!!

But not so fast.  I'm NOT pleased.

You did what?!?  You told them we could do that?  Are you out of your freakin' mind?  Don't you have any clue how hard that will be?  Don't you have any idea how crappy the quality will be if we deliver in the time frame you promised?  You just single-handedly created a nightmare support situation, and made getting the sale much worse than not.  I would have been toasting with you, but now I just want to strangle you.  Why can't you sales people sell what we have?  You're not a sales person, because they actually sell something that is real.  You, my friend, are a promise maker, counting on someone else to deliver.  That takes no skill.  Want to impress me: use sales skills to sell something, instead of throwing my nights and weekends under the bus so you can pocket the coin.  Anyone can make a promise you jack wagon.

(Calming down now..  Deep breaths..  Finding my happy place..)

Two terrible mistakes were made in this scenario.  First, the sales person made a promise that assumes the development team has the ability to deliver the desired feature.  How does the sales person know that the developers have the knowledge and skill needed to "pull it off"?

(WARNING:  If you are a sales person, and ever find yourself thinking or speaking the phrase "pull it off", then you are entering very risky territory.  Good software is never "pulled off".)

Second, the sales person gave the customer a time-frame for delivery.  Once again, how does the sales person know that the feature can be done in the time promised?  Does the sales person have the knowledge, skills and clairvoyance to know all of these things?  If not, then who does?  Who IS qualified to address these issues?  Why... the architect, of course.

So, I want to tweak the sales process a little here, and introduce the one party that can reliably speak to feature requests and time-frames.  The next time a customer asks "Can it do.....", before answering, contact your architect and run it by them.  Give them a little time to determine what to do and how to do it.  (If you really want to impress me, you'll make an architect a member of your sales team.)  After they've had a chance to determine the best way to deliver, only then provide an answer to the customer.  First, the answer will be one with some actual delivery power behind it.  Second, it will be done in a way as to not create bigger problems in the future.  Third, the solution will be designed in a way to add to your company's list of leveraged assets.

At one place I worked (which will remain nameless to protect the guilty) the promissory sales technique was a really big problem, and I had my fill of it.  I used to get angry in management meetings because the executives continued to allow it.  One day I proposed what I believed was an equitable plan.

It went something like this.. Each time a salesman sold something our products did not do, that their sales commission be split with the developers.  I saw this as a way to incent the sales team to sell what we had, thus keeping 100% of their commission; but also a way to make sure the developers enjoyed the fruits of constantly delivering under the gun, keeping their growing malcontent at bay.  The idea got tossed around some, but never enacted, which is a shame.  Over time, my developers grew more and more discontent as their nights and weekends became more of a distant memory, and the quality of their work and commitment diminished right along with it.

In conclusion, I hope you understand that there are very real dangers in not involving an architect in the sales process.  Only the architect can speak to your ability to deliver a desired function.  Only the architect has any business estimating a date for that delivery.  And only the architect has the vision to make sure the function is done in a way that adds to your strategic assets, allowing it to be leveraged over time.  By not involving the architect, you may very well end up with a one-off implementation that cannot be reused, a significant and costly support headache, an unhappy and less productive development team, and ultimately unhappy customers.

A Tale of Two Companies

For just about my entire career as a software architect, I have been troubled by the number of people who do not really understand what it is I do.  Most people think I am a super-senior software developer, others an advanced form of business analyst.  Others still hear the word “architect” and think that I am a designer of software, something akin to the traditional architect that takes requirements and turns them into blueprints and plans that are then used by the team of folks who construct some sort of building; which is getting much closer to the mark.  Now I’ll readily admit that these perceptions do cause some degree of angst, because I know that each engagement I see will involve an initial period of education, which cuts into the time I should be working.  But this angst is easily and quickly tempered by the realization that because this profession is still in its nascent stage, and has so poorly penetrated the industry, that folks are allowed to be confused.  The best approach for me is to be patient, and to do a better job of communicating.

When I’m asked to explain what architecture is, and its benefits, I typically use the following story of two extremes, in order to illustrate what architecture is all about.  I call it the “Tale of Two Companies”, and it goes something like this…

Company A – Chaos & Caged Systems, Inc.

Imagine that Chaos & Caged just sold a software system to a customer and now needs to develop that system.  (Someone selling software that doesn't exist yet?  Surely that never happens.) The business team, working with a team of business analysts (maybe) and the customer, develops a list of features that the system should support.  With this list of features in hand, the business team goes directly to the developer and says something like, “I need a system developed that delivers these features, and I need it by this pre-arranged due date.”  The developer, always eager to show off her mad coding skills (Because developers naturally take a lot of pride in their skills, and who doesn’t like to show off?), eagerly jumps into the project with vigor, her Hero Coder flag flying proudly in the wind.

One later day the system is complete.  Because of the developer’s commitment and determined heroics, she worked nights and weekend to “pull it off” by the promised date.  The business team takes the system to the customer and they love it.  Everyone is happy!

A couple of weeks pass and the customer calls.  After using the system for a while, they determine that it could do a lot more things for them.  They would like to not only introduce a handful of features, but also like to see it integrate with a back-office system they’ve been using to run their business for years.  The business team loves it and can already count the commission dollars and expansion opportunities.  Like before, they write up the new list of features and take them to the same developer, presenting her with the list, and another pre-arranged due date.  Same as before, the developer opens her development environment and gets to coding, her flag a flying.

But something odd happens on the way to the Forum.  One late night on a Saturday, when the developer is working on the system, it occurs to her that a couple of the new features are really, really difficult, and that she will not be able to deliver them on time.  For these features, she thinks it through and realizes that if she just cheats a little bit by violating some established best practices, that she can, once again, “pull it off”.  But the integration is another matter altogether.  For this she has to change the design of the original system, losing fully 30% of the work she had already performed.  She informs the business team of the dilemma, and expectations with the customer are reset.  The customer is not happy, and the relationship between all parties takes on an uncomfortable tenseness.  Communication stops flowing as freely as it had, and by the time the update is delivered, no one is really happy.  Worse yet, the customer finds that the integration is very limited, and that some of the features that had worked before, now have issues.

But time marches on, and sales numbers must be met, and the business team sells another system to a different customer.  Excited by the prospects of the new customer, the business team dutifully gathers the features and takes them to another developer, who, in his eagerness to be successful, unfurls his Hero Developer flag and starts cracking on the code.  In time this project plays out the same as the first.

While all this is going on, other business teams are out selling systems to other customers.  After a couple of years many systems have been developed by many developers, each dedicated to maintaining those systems and adding new features in a seemingly endless cycle.  Since those development resources are committed to their systems, Chaos & Caged must hire additional developers each time a new initiative comes along, and the process repeats itself time and again.  A stable business is born.

But one day disaster strikes!  One of their more important hero developers leaves Chaos & Caged because of stress and general unhappiness with the conditions.  In the wake remains a huge backlog of features and bugs.  With the hero developer gone, no one can maintain the system, and a crisis is created.  It’s going to take a long time to find a developer to take the other’s place, and a lot longer for them to get up to productive speed.  In the meantime, work just piles up, more deadlines are missed and more revenues go unrealized.  Worse yet, Chaos & Caged keeps missing their profit goals quarter after quarter, as the cost of development and support continue to eat up a greater portion of revenues.
Chaos & Caged has a steady customer base and plenty of work, but they seem to have a systemic problem that keeps them from growing and being profitable.  Something is missing.  Some concern is not being represented in the process.

Company B – Ordered & Growing Systems, Inc.

Imagine that Ordered & Growing just sold a software system to a customer and now needs to develop that system.  The business team, working with a team of business analysts (maybe) and the customer, develops a list of features that the system should support.  With this list of features in hand, the business team goes directly to the architect and says something like, “I need a system developed that delivers these features, and I need it by this pre-arranged due date.”  The architect reviews the list of features and informs the business team that the date they provided cannot be met with a quality result.  A more accurate estimate is produced, and it is determined that the architect should become part of the business team, so that mistakes like providing due dates devoid of any reality are avoided in the future.

The architect then sits down and begins to more closely analyze the requirements.  In particular he breaks the requirements down into distinct functionality, a process known as decomposition.  Once a list of distinct functionality is produced, the architect sets about assembling this functionality into logical components; where each component is modularized, and kept purposely separate from the others.   Now that an inventory of needed components is determined, the architect learns from his business team colleagues if there are any restrictions against using third-party items.  As it turns out, he is free to use outside items and begins to look for third-party components that would fill-in nicely for those in his inventory.  Out of his component inventory, he discovers that roughly 40% of the components can be licensed from third-party libraries or services, far more inexpensively that they can be developed in-house.  Plus, components from third-parties tend to be less expensive over the long-run, as ongoing licensing is typically less costly than maintaining and testing internal components.

With 40% of the components inventory determined, the architect draws up plans for the remaining 60% and submits them to the developer, who then writes the necessary code.  While the developer is developing the code, the architect reviews the progress to make sure that things are going according to plan.  Occasionally the developer may run across something in the architecture that seems odd, or unworkable.  The developer brings it up with the architect, which may result in a change. (The importance of this developer feedback cannot be understated.)

One later day the system is complete.  The business team takes the system to the customer and they love it.  Everyone is happy!

A couple of weeks pass and the customer calls.  After using the system for a while, they determine that it could do a lot more things for them.  They would like to not only introduce a handful of features, but also like to see it integrate with a back-office system they’ve been using to run their business for years.  The business team loves it and can already count the commission dollars and expansion opportunities.  Like before, they write up the new list of features and it is analyzed by the team’s architect.  The architect determines that the changes involved will affect only 15% of the components; the other 85% will be unaffected, and therefore will not need to be regression tested.  He also determines that the integration would best be done by developing a completely separate integration module, which would have no impact on either system.

Like before, the architect draws up the plans for the developer and they work together to make the needed changes to the components needing to be changed, and developing the integration module.
One later day the system is complete.  The business team takes the system to the customer and they love it.  Everyone is happy, and Ordered & Growing is well on its way to building a library/infrastructure of reusable components and services.

Time marches on, and sales numbers must be met, and the business team sells another system to a different customer.  Excited by the prospects of the new customer, the business team dutifully gathers the features and they are analyzed by the architect, who determines that roughly 60% of the new system’s functionality can be provided by the components already found in the library.  He determines how to best provide the remaining 40%, again a buy vs. build analysis, and new components are both licensed and developed in-house. In time this project plays out the same as the others, and the inventory of component assets continues to grow.

While all this is going on, other business teams are out selling systems to other customers.  After a couple of years many systems have been developed, each adding more and more reusable components to the library in a seemingly endless cycle.  As additional systems are developed, Ordered & Growing begins to experience economies of scale and scope; allowing them to more rapidly, reliably and predictably deliver quality systems without having to hire proportionate numbers of developers, testers and support personnel.  A repeatable business is born.

Naturally, developers come and go.  New developers are brought in to maintain the components of the developers who left, as well as to develop new components for the library.  Because of Ordered & Growing’s excellent architectural model, no one developer “owns” all the components used in a particular system, so the impact of the departure what not as severe.   As the library/infrastructure of components and services begins to grow and mature, developing new systems for new and existing markets becomes more of an assembly operation, rather than a development operation, as these assets are combined in new and different ways to offer more and greater capabilities.

The End.

Now I know that the story of Chaos & Caged  and Ordered & Growing is over-simplistic, but it serves the point of illustrating the invaluable role architects play, and what it looks like when they don’t.  Architects are responsible for creating and enforcing a process where all development efforts go towards constructing and maturing an infrastructure of capabilities, which can be leveraged throughout the business to deliver economies of scale and scope, allowing the business to grow and profit in a way that is reliable, repeatable and predictable.  Without an architect’s steady hand on the wheel, development efforts inevitably become fragmented, resulting in higher costs and a diminished ability to expand, which over time reduces the company’s ability to compete and profit.

An architect is a hard-to-find professional; someone who feels comfortable representing both the business and technical interests.  This person is a hybrid individual; one part technologist, one part economist.  There are different types of I.T. architects representing specialized parts of a company’s operations (enterprise, software, security, infrastructure, etc.), but they all look to establish and champion a vision of how things will get done.

Without one, you might get some early wins, sure.
But with one, you will continue to win in the long term.