My assessment of the assessments of

From Paul Krugman (Obamacare Success – to Ezra Klein (Ezra Klein: Thus Far, Obamacare a ‘Big Failure’ | National Review Online) to the NYtimes (From the Start, Signs of Trouble at Health Portal – to Alex Howard at BuzzFeed (How The First Internet President Produced The Government’s Biggest, Highest-Stakes Internet Failure) there has been more and more assessments coming in for, and most of them have been negative. How good are these assessments?

I would argue that at this point, the assessments of are of limited value. For example, the article has a good run down on the background of the project and the politics involved, but the analysis of the system is mostly based on insider and second hand information. The Buzzfeed article has a great analysis of the challenges of IT procurement in the U.S. government, but again, it doesn’t deal directly with the system itself in question. That doesn’t mean those stories are bad, for there is alot of interesting background information in them. But it doesn’t tell you much about the actual system that makes up

There have been some good attempts at an assessment from an IT perspective from the CTO of Huffington Post (Why The Experts Are Probably Wrong About The Crack-Up | John Pavley), Paul Smith over at TPM (A Programmer’s Perspective On And ACA Marketplaces), as well as from individual bloggers with IT knowledge (e.g., Too Big To Succeed and Is There A Problem Here?). Someone wanting a better idea of the technology and the design of the system would be better off reading those.

In all cases, the individuals doing these assessments have very little to work with. A proper assessment of an IT system can take a team weeks if not months with full access to the system and all the artifacts and deliverables that went into making the system. Most of the assessments I have read so far have been based on having little if any data and few if any artifacts. This isn’t a criticism of the assessors: it’s all they have to work with. (The only fault I see is with some assessors displaying slight arrogance in thinking they have nailed it in their assessment as to what is wrong with the system.)

Given the little to go with, the people who are assessing the system a success or a failure are basing it on a number of assumptions that they have which may or may not be true. I don’t see much value in those assumptions. For example, most of the assessments I have read so far seem to assume the system should be up and running with few problems, given the importance of this site and the money invested in it. (Klein in particular seems to be certain of how an IT project should go, which I find remarkable in someone with an non-IT background like his.) There is nothing wrong with that assumption, but that’s all it is. You may think it is a valid assumption, but that is besides the point.

At this point in time, the only ones that can assess if the project is a success or a failure are the key stakeholders for the project. If you are someone who could never get healthcare because of preexisting conditions and now, even with difficulty, you are able register for a get healthcare, you likely consider the site a success. Conversely, if you are an insurer who expected to get alot of applicants from the site and are getting none, you may consider the site a failure. Right now it is too early to weigh any of that: it will take time and further analysis.

The government seems to have a longer term view of the site than most of the analysts that I have seen so far.  As the story says, “Administration officials have said there is plenty of time to resolve the problems before the mid-December deadline to sign up for coverage that begins Jan. 1 and the March 31 deadline for coverage that starts later”. There is actually some benefit in launching the site now, well in advance of the December deadline. Sites with deadlines often experience the most traffic around the time of the deadline, and I expect will be no exception. They have two months to resolve performance issues, better model usage patterns, fix critical bugs in the software, enhance the infrastructure, and improve the integration with other systems. Two months is a short timeframe, but it is feasible that they can resolve many of the obvious problems that the site is suffering now. As well, the proponents of the site should have enough data and analysis of the data to argue the success of the site.

Regardless of how the site is perceived then, anyone doing these assessments should have alot more to work with. In the future, if you are reading future assessments of the sites, things to consider are:

  • how much information about the site is the writer using in the assessment? More is better. Skip the ones based solely on anecdotes, or that ignores key stakeholders.
  • what is the criteria the writer is using for determine whether or not the site is successful? Is that criteria a valid one? Comparing it to other government or large scale commercial IT project is a good criteria. Comparing it to the roll-out of the latest iPhone is a poor one.
  • is the writer assessing the IT aspects of the site? How much IT experience does the writer have? You don’t have to be an IT expert to write about IT, but if you are talking about IT, you should have a basis for why your analysis is valid. If you are saying the architecture is faulty, you should be able to represent the architecture diagrammatically and say the architecture is faulty at points A, B, and C and here are the reasons why.

I am excited to see people discussing IT architecture with general audiences. I have been building and assessing IT architectures for decades, and it is a topic dear to me. I also know it is hard to assess the validity of what people are saying about it. That’s why I decided to write this. I appreciate any constructive feedback, and I will try and answer any that I receive.

(The above Flow Chart: How Health Insurance Exchanges Work is a representation of a health insurance exchange. I’ve included it to represent the complexity of any IT system that has to provide this type of capability.)

3 responses to “My assessment of the assessments of

  1. I’ve been thrilled to see the attention this is getting, at least among IT folks. And it is an interesting puzzle, though I suspect in the final analysis it’s too little time for as project that was too big to begin with. I don’t care how much money you throw at a project like that, it’s going to have problems.

    What has bothered me greatly, though, are some of the silly issues people are finding. For example, the URL for the “error” page directed to a missing page. I’m sure the folks running this thing are harried, but I really question what kind of release process they have in place. There’s being terribly busy, and then there’s negligence. Hopefully someone on this project will come forward.

    • smartpeopleiknow

      I agree 100%.

      I hope to revisit the site and the issues around it in a few months. I expect that may of them will become non-issues by then.

  2. Unfortunately most architectures I see still have this notion of an “administrator” interface. This fabled administrator tends to have more power than he is supposed to because that’s the easiest way to build systems.

    Paper processes do not guarantee protection from rogue administrators as Edward Snowden had shown us.