From Paul Krugman (Obamacare Success – NYTimes.com) to Ezra Klein (Ezra Klein: Thus Far, Obamacare a ‘Big Failure’ | National Review Online) to the NYtimes (From the Start, Signs of Trouble at Health Portal – NYTimes.com) to Alex Howard at BuzzFeed (How The First Internet President Produced The Government’s Biggest, Highest-Stakes Internet Failure) there has been more and more assessments coming in for healthcare.gov, and most of them have been negative. How good are these assessments?
I would argue that at this point, the assessments of healthcare.gov are of limited value. For example, the NYTimes.com article has a good run down on the background of the project and the politics involved, but the analysis of the system is mostly based on insider and second hand information. The Buzzfeed article has a great analysis of the challenges of IT procurement in the U.S. government, but again, it doesn’t deal directly with the system itself in question. That doesn’t mean those stories are bad, for there is alot of interesting background information in them. But it doesn’t tell you much about the actual system that makes up healthcare.gov.
There have been some good attempts at an assessment from an IT perspective from the CTO of Huffington Post (Why The Experts Are Probably Wrong About The Healthcare.gov Crack-Up | John Pavley), Paul Smith over at TPM (A Programmer’s Perspective On Healthcare.gov And ACA Marketplaces), as well as from individual bloggers with IT knowledge (e.g., Too Big To Succeed and Is There A Problem Here?). Someone wanting a better idea of the technology and the design of the system would be better off reading those.
In all cases, the individuals doing these assessments have very little to work with. A proper assessment of an IT system can take a team weeks if not months with full access to the system and all the artifacts and deliverables that went into making the system. Most of the assessments I have read so far have been based on having little if any data and few if any artifacts. This isn’t a criticism of the assessors: it’s all they have to work with. (The only fault I see is with some assessors displaying slight arrogance in thinking they have nailed it in their assessment as to what is wrong with the system.)
Given the little to go with, the people who are assessing the system a success or a failure are basing it on a number of assumptions that they have which may or may not be true. I don’t see much value in those assumptions. For example, most of the assessments I have read so far seem to assume the system should be up and running with few problems, given the importance of this site and the money invested in it. (Klein in particular seems to be certain of how an IT project should go, which I find remarkable in someone with an non-IT background like his.) There is nothing wrong with that assumption, but that’s all it is. You may think it is a valid assumption, but that is besides the point.
At this point in time, the only ones that can assess if the project is a success or a failure are the key stakeholders for the project. If you are someone who could never get healthcare because of preexisting conditions and now, even with difficulty, you are able register for a get healthcare, you likely consider the site a success. Conversely, if you are an insurer who expected to get alot of applicants from the site and are getting none, you may consider the site a failure. Right now it is too early to weigh any of that: it will take time and further analysis.
The government seems to have a longer term view of the site than most of the analysts that I have seen so far. As the NYTimes.com story says, “Administration officials have said there is plenty of time to resolve the problems before the mid-December deadline to sign up for coverage that begins Jan. 1 and the March 31 deadline for coverage that starts later”. There is actually some benefit in launching the site now, well in advance of the December deadline. Sites with deadlines often experience the most traffic around the time of the deadline, and I expect healthcare.gov will be no exception. They have two months to resolve performance issues, better model usage patterns, fix critical bugs in the software, enhance the infrastructure, and improve the integration with other systems. Two months is a short timeframe, but it is feasible that they can resolve many of the obvious problems that the site is suffering now. As well, the proponents of the site should have enough data and analysis of the data to argue the success of the site.
Regardless of how the site is perceived then, anyone doing these assessments should have alot more to work with. In the future, if you are reading future assessments of the sites, things to consider are:
- how much information about the site is the writer using in the assessment? More is better. Skip the ones based solely on anecdotes, or that ignores key stakeholders.
- what is the criteria the writer is using for determine whether or not the site is successful? Is that criteria a valid one? Comparing it to other government or large scale commercial IT project is a good criteria. Comparing it to the roll-out of the latest iPhone is a poor one.
- is the writer assessing the IT aspects of the site? How much IT experience does the writer have? You don’t have to be an IT expert to write about IT, but if you are talking about IT, you should have a basis for why your analysis is valid. If you are saying the architecture is faulty, you should be able to represent the architecture diagrammatically and say the architecture is faulty at points A, B, and C and here are the reasons why.
I am excited to see people discussing IT architecture with general audiences. I have been building and assessing IT architectures for decades, and it is a topic dear to me. I also know it is hard to assess the validity of what people are saying about it. That’s why I decided to write this. I appreciate any constructive feedback, and I will try and answer any that I receive.
(The above Flow Chart: How Health Insurance Exchanges Work is a representation of a health insurance exchange. I’ve included it to represent the complexity of any IT system that has to provide this type of capability.)
Smart People I Know 