it

Last week (Thu-Fri Feb 25-27, tens of thousands of DC residents (and some non-DC residents) were frustrated and angry after 3 straight days of DC”s COVID vaccine registration portal was plagued with crashes and errors. Last night (Thu March 4), the DC Council’s Committee on Health held a marathon oversight hearing, with over two hours devoted to the vaccine rollout to get some answers about what went wrong and the pre-registration portal coming soon. I encourage you to watch the hearing yourself https://www.youtube.com/watch?v=URIc6lhNJvs&feature=youtu.be if you’re interested, starting at the 9 hour mark, with relatively detailed testimony from DC’s Office of the Chief Technology Officer (OCTO) Chief, Lindsey Parker, starting at 9:30.00 or so, but here’s a quick paraphrase of what we learned:

Portal to Date

• DC Health approached OCTO in September 2020 to build a vaccine scheduling/administration portal. OCTO decided to contract this out instead of using limited manpower to build out self, and was already familiar with Microsoft’s recent COVID-work since they had helped launch a public COVID testing site in March in very short order. Contract was an emergency procurement based on the public health emergency, so no competition required and can be done quickly. DC was the first jurisdiction in the country to sign on to Microsoft’s product; at the time that occurred, Microsoft hadn’t yet created a pre-registration product. There were no budget constraints on OCTO/DC Health to create the portal and they expect to be reimbursed by Federal government once the new bill is passed later this month.
  • [Author’s Note: I think a lot of people will be angry about the fact that this project didn’t start until September 2020. I expect the media and Council to press more about this. Based on my sense, I don’t think the product would have been any better if it had started in March or not; also Federal govt was developing its own system and no one knew exactly how it would be administered until later on so creating a system just on a guess would cause just as many problems as it would solve. But I could be wrong]
• Portal launched in December 2020. As part of the contract, Microsoft is responsible for testing. Since the portal launched in December, there has been 16 versions created. There has not been any public testing by Joe Schmoe or seniors prior to launching, so they have been reactive to feedback.
• On February 18th portal was open for priority zip codes for seniors and other essential workers (including grocery store workers). The peak had 62 users with 2000 requests (clicks/refreshes) during the peak 5 minute window. 100,000 clicks/refreshes in the hour it was open. No errors occurred.
• On Feb 25th, the portal was now supposedly open for priority zip codes, not only seniors but with 18-64 with chronic conditions. The number of distinct users during peak went from 62 the week before to 8,780. The number of requests (clicks/refreshes) during the peak 5 minute window went up 27400% to 550,000 and 6500% for the full hour to 1.32 million attempts (up 6500%). Microsoft was not expecting this increase and there was an extreme bottleneck. System worked to identify if traffic was legitimate which slowed system tremendously. Once traffic speed was restored, they realized Microsoft forgot to update the workflow to include 18-64 (“Tired eyes missed that workflow”). This was updated at 9:51 and appointments were closed by 9:54.
• Microsoft then assured OCTO that they improved their infrastructure for Friday’s wider rollout. They were wrong.  The number of distinct users went up to 20,202, with 1.2 million clicks/refreshes during peak 5 minutes and 7.38 million during the hour (459% increase from Thursday). Again there was extreme bottleneck as service protection limits again were invoked (to protect from bot that trying to take system down) which then triggered slow site as well as errors for 13% of clicks.
• For 3rd time on Saturday, there was a new registration opportunity for 18-64 with conditions only in priority zip codes with next week’s appointments. Once again Microsoft reassured they had enough capacity. Again they were wrong. Apparently they had fixed the issue but Microsoft’s development team “cleared the cache” right before it opened and with the heavy traffic, the portal froze again, and had to restart the system. It was down for 15 minutes. Peak 11,247 users, 800,000 clicks/refreshes in 5 minutes and 3.8 million in the hour. Whopping 1.6 million errors (44% of all clicks/refreshes).
• Finally, Microsoft higher level folks fixed the portal this week and yesterday a waiting room was instituted and the captcha was removed (other security features on backend to replace it). The site handled roughly the same number of unique users and number of hits as the previous Thursday with no errors and no issues with the site (all appointments taken in less than 7 minutes)
• OCTO and Microsoft seemed to have had correctly anticipated a huge number of new sign-ups (it wasn’t clear but it seemed system could handle 100,000 unique users at once), but per Parker, they did not expect “millennials” to use every browser on their computer/phone/tablet at once continuously refreshing, as well as asking friends from across the country to assist.

Pre-Registration System

• • To the shock of everyone on the call, the representative from the City Administrator’s office (Jay Melder) said that both Microsoft and Accenture have been asked to develop the pre-registration portal and they have yet to pick which one has been through the right testing, has the right features. This is despite fact that Mayor demonstrated what the portal looked like today at a press conference apparently the Microsoft version though Accenture’s would look similar.
    • [My read: DC’s is making sure Microsoft doesn’t screw up on Friday and want to put their feet to the fire by bringing in a second vendor. I entirely expect Microsoft to be chosen.]
  • It’s not guaranteed that there will be a pre-registration portal Monday or even next week at all (note that originally the pre-registration wasn’t supposed to open until mid March).
  • As noted by others, pre-registration will allow DC residents or workers to submit their information and then wait for DC Health to invite them to make an appointment. Invitation will come through email, text, and/or phone call. Will be able to list a 3rd party who can register on your behalf.
  • Dr. Nesbitt botched an answer at the press conference-all DC residents and DC workers will be able to register when the site opens live even if they’re not eligible. Ineligible workers will not be advantaged with higher priority for booking early though it seems.
  • Once you have the invitation (which will likely be issued in batches on a rolling basis), you’ll have a certain amount of time to book an available appointment. Likely 48 hours. Even if you miss that window or the appointments don’t work for you, you still are kept in the system so you have another bite of the apple.
  • Won’t be based on first come, first serve. The way it seems that it work is that there will be certain % dedicated to each group, and this % will constantly change based on whether groups uptake on the vaccine % changes. There will continue to be a priority for zip codes (or more targeted neighborhood clusters) whose vaccine uptake % is low relative to rest of the city. Within each group there will be some randomization so that person who registers minute 1 doesn’t necessarily get it before the person who registers on Day 3, though obviously the appointment invitations are only filled based on who has registered so far, so it still pays not to delay too much on pre-registering.
    • [Here’s an example I made up that could be the way it works but is not a real example: 10,000 vaccines through portal, (10,000 with hospitals and other places not through portal). 2000 to seniors in priority zip codes/neighborhoods, 2000 to 18-64 with chronic conditions in priority areas, 1000 to eligible essential workers in priority areas. 2000 to seniors in rest of city, 2000 to 18-64 with chronic conditions in rest of city, 1000 to eligible essential workers (DC or not DC resident]
  • OCTO/DC Health didn’t seem to commit to announcing each time the breakdown of what the doses are going but seemed open to it. They said that the current vaccine data would show the prioritization.
    • Note: The system will be pretty opaque which is why DC Health originally preferred the free-for-all system of registration.

Another random note, DC Health this week essentially confirmed my analysis/reporting that the reason why Bloomberg/NYT/Post/Time trackers (which all use the same CDC data show that DC has way more doses than DC Health has is that 40-50,000 doses out of the 280k+ are for Federal entities which DC has no control over (and they’re more likely to stockpile). He said the true % of adminstered/delivered is closer to 80%. Issues with data is lack of reporting, hospitals are holding onto doses a bit longer while community health organizations have reserved more shots for 2nd doses just to be safe.

OCTO also posted some slides about the challenges of government IT.