Tag Archives: architecture

A lot of people have very strong opinions about the IT that has been rolled out for Ontario’s vaccine distribution system. I understand that: it has been very challenging for people to get a vaccine here in this province. People look at other provinces like Nova Scotia with their centralized system and ask why didn’t the province do that. They look at this site some very smart guys hacked together in four hours that allows you to text it and get back nearby vaccination sites and they say the government should be more like that. They attribute the government with being cheap, racist and other things, and say they didn’t build a good IT system because of that.

I have strong opinions about the vaccine IT that has been built out for the province too.  The difference is that my opinions are based on working on several large scale projects with the province. It’s also based on working on emerging IT for several decades. I’d like to share it in the hope it helps people gain some perspective as to what is involved.

Building IT systems for a large organization, private or public, is difficult. There are many stakeholders involved and many users involved and often many existing IT systems involved. You have to meet the needs of all of them, and you often have to go through many reviews with internal reviewers to demonstrate your new IT system meets their standards before you can start to build anything. Even then, with all of that, the IT system you are about to build could still fail. Big organizations are very sensitive to this and work diligently to prevent it. You can’t just hack together a proof of concept one day and then the next day have it go live on some banking or government site. Not in my experience.

Failure is a big concern. Another big concern is the needs of the stakeholders. For government systems there are many of them. There were over 50 other organizations that I had to work with external to the government on the projects I was leading. Each had their own IT systems and their own way of doing things. We could not just come in and say throw out your existing IT and use this new thing. There was a whole onboarding process that had to be developed to bring them into the new way of doing things. And that didn’t include all the people in the province or the country who will use the IT system and are therefore are also big stakeholders: they were taken into account and consulted separately.

A third big concern is systems integration. Not only do you need to work with the external IT systems of the stakeholders mentioned above, you have to work with internal IT systems to get data or send them data. In all cases that means not only do you need to understand what the government needs the new IT  system to do, but it means you have to have some understanding of how all these other systems work. Your new IT system can not be effective if you don’t know how to work with the existing systems. It’s a lot harder than scraping existing web sites and calling it done.

It is one thing to develop an IT system to provide new functionality; you also have to make sure it satisfies a number of non-functional requirements (NFRs). Reliability, performance, security, maintainability, data integrity, accuracy are just some of the NFRs that must be determined and met. Even cost and speed to market (i.e., the time it takes to develop a working system) are important requirements. Then there are regulatory requirements you need to meet, from SOC 2 to HIPAA, depending on the type of system you are building.

In addition to all that, there may be technical or design constraints that you must meet. The organization you are working with may require that you use certain suppliers or certain technology for anything you build. You may want to use Mongo and Node in a GCP region in the US for your IT system, but your client might say it has to run on Azure in Canada using Java/Springboot and Postgres, so your new IT system will have to accommodate that.

Once you have taken all that into account, the organization may have some other requirements, including dates, that must be met. In the case of systems like the vaccine IT, that date is “yesterday”. That will force some decisions on how you build your system.

All that said, my educated guess – and it is a guess, because it is based on my experience and not inside knowledge on how the system was built – was that Ontario decided the quickest way to roll out the vaccine IT was to build on the basis of what already exists. For example, many of the individual pharmacies in Ontario have their own systems for working with their patients. And several hospitals I checked use other software like Verto to manage their patients. The integration of all those systems is on “the glass”. By that I mean you can go to the government web site (“the glass”) and then you are redirected to other systems (e.g. a Verto system for a hospital) to book an appointment.

There are benefits to going with this approach versus building a new centralized IT system. It’s cheaper, for one. But it’s also faster to rollout than a new system. It’s less prone to failure than a new system. If you assume people are going to sign up for COVID vaccines like they do flu vaccines, then you know this approach will work, and reliability is a key NFR. If you are designing IT systems, you have to make assumptions to proceed, and that one is based on things you know, which is usually good.

Unfortunately, it turned out to be a bad assumption. Unlike the flu, where uptake is around 30% and spread out, people are scrambling to get the COVID vaccine. This has led to the downfall of the current approach in Ontario as people try all sorts of ways to get a shot asap.

You might say: well that was a dumb assumption, why would anyone make it? In my experience, with new IT systems, it is hard to predict how people will behave. My colleagues once built a system for a government agency that allowed people to weekly update their status on their workplace. The system was available 24/7, but they had to put in their information by Sunday night, 23:59. All week no one would use the system, and then at 11 pm on Sunday it would get hammered with users trying to send in their information. We did not predict that. We assumed there would be peak usage then, but almost all the traffic was at that point.

To mitigate the risk of bad assumptions, IT projects will often do a gradual rollout. However that was never going to be an option here: people wanted the vaccine IT system “yesterday”.

Nova Scotia chose to develop a centralized system and people are saying Ontario should have done that. Possibly. It’s also possible that conditions in Ontario could have resulted in delays in rolling out a centralized system. Or the system could have been on time but failed often. Many IT systems and programs (e.g. Obamacare) have this result. Or some of the big hospitals and independent small pharmacies could have opted out. Then people would have been complaining about not being able to get a vaccine at all and that would have been much worse.

I am happy for Nova Scotia that theirs works well (although people are bypassing it and just showing up in Nova Scotia, so it’s not all roses there either). It’s fair to compare Ontario to them to some degree. And when all this is over, there should be an audit done by objective third parties to see what worked and what didn’t and what Ontario should do next.

I hope after reading this you have a better understanding of what goes into building IT systems for large organizations. I wish they could be built in a day or a week or a two week sprint even. I do know that large organizations are becoming more nimble and are working to getting out IT capability to their clients and citizens faster than ever before. But as you see, there are many things to take into account, and even with many people working on a new IT system, it does take time. Time measured in weeks and months and even years, not hours.

So the next time you hear someone say “they had all this time to figure this out”, take this into account. And thank you for reading this. I hope it helps.

Finally, these thoughts expressed here are mine and not those of my employer.

(Image is a link to the wikipedia page on system context diagrams, a diagram often used to determine how a new IT system fits in with existing IT systems).