Continuous Validation: unlocking accurate mass testing

We are not short on technological solutions for testing [1][2][3]. What’s blocking us is fear of making a mistake. Dr Michael Ryan, Executive Director, WHO Health Emergencies:

anyone involved in emergency response will know this. If you need to be right before you move, you will never win. Perfection is the enemy of the good, … and the problem we have in society at the moment, is everyone is afraid of making a mistake… But the greatest error… is to be paralysed by the fear of failure.

https://www.youtube.com/watch?v=NpUOG8pkDYI&t=43 (abbreviated)

It is a balance, we don’t want to test someone who has the virus and tell them they don’t, letting them spread it and kill people. Or tell someone who is not infected that they do have the virus and they commit suicide or other significant downsides.

These false negative and false positive results can come from a wide range of sources, from cross contamination chemicals, incorrect temperature of samples, software bugs etc.

Standards, and accreditation, and the things that come with this such as negative pressure labs, appropriate PPE, cleaning rotas, lab access control, documenting chemical batches for tracing, improving protocols, etc help keep lab staff safe whilst reducing these false negatives and false positives. They will never eliminate them. But this is as good as we can get and the test manufacturer and testing lab can say everything has been done. But I believe we can do better, and faster.

The current situation

Imagine you’re an NHS lab with skilled staff, necessary facilities and machines with no reagents [4][5]. Or you are an academic or private lab who are ready to repurpose your facilities and equipment. You don’t have staff who are trained in molecular diagnostic pathology but you’ve run hundreds of molecular biology experiments. You have extensive experience of running robust scientific experiments and research. You read all the regulations, implement them and existing trained pathology lab staff have kindly given you extensive advice from cleaning protocols, sources of sample contamination, good practices for sample handling, avoiding long shifts etc.

Right now you could look to various open-source / generic designs [2][3].

You have facilities for handling patient samples, so you order the components, assemble it “DIY” style, and validate it using commercially available controls [6][7]. The results come back. Your LOD (Limit Of Detection) is comparable with the gold standards. You repeat the samples 10 times. Your false positives stay at 0. You’ve got an accurate, safe and ethical testing facility.

But the tests are running on a new system that’s unaccredited. Or if you’re not an existing pathology lab your lab and staff are also unaccredited. So then they’ll need to be validated by MHRA, process 20 directives for CE marking, 4 ISO standards, UKAS accreditation, HSE accreditation, HSL accreditation. By 2021 you might receive your first sample and test it in the lab.

Continuous Validation: How to quickly approve 100,000 accurate tests per day

(and how other options like EUA can be made more robust)

Test the testers. Not through accreditation. But through correctly identifying control samples in every batch of samples every day.

Let’s call this Continuous Validation. To pass the test the testers will still need to read all the standards and implement them. The tests are cheap enough though (£3-5 each [8]) that we could afford to run two or three tests per sample. In reality you might only need to run 10 tests per batch of 96 to have a good assurance that if those are correctly identified, that the other samples are also correct.

With the USA’s FDA’s EUA policy [9], a larger number of tests have come to market very quickly [1] but as no test is better than a bad test they’ve likely done more harm than good. Continuous Validation of labs avoids that. If we had 20 new labs, capable of processing 5,000 samples per day each, that would need 200 tubes (20 labs x 10 different controls), containing 50 ml (5000 samples per day / 100 samples per batch) of the different control samples. A single technician could make that in an afternoon for under $1,000, put on dry ice and send to each lab. You’d also ship one set back to the central lab and everyone would run their tests, and report all their results.

Continuous Validation in a distributed testing system

This inverts the majority of the necessary quality control effort from a centralised point to become distributed. It scales proportionally with both the number of labs and the number of tests. It distributes the risk of non-compliance to individual labs. And it does this whilst retaining the central point of control over quality.

Trust

Untrusted and semi-trusted systems were also explored by the authors. The former likely requiring various degrees of integrating central testing authority with patient sample collection sites and patient record systems. Semi-trusted systems could include the use of smaller one use samples per batch along with live streaming video from labs to increase transparency and provide an additional form of documentation.

Interlab comparisons for free

A component of ISO 17025 is the valuable interlab comparisons. We’d get that for free.

What’s still a bottleneck? HSE to protecting the staff and the environment

If we move to lysing samples at source, prove they’re inactivated and disinfect them before entry to the lab then we remove a significant risk factor, reducing the lab from BSL3 to BSL2+.

We need your help

A bad test is worse than no test, and this post is not about cutting corners. It’s about challenging ourselves to see if there are faster ways to increase testing capacity. If you’ve got a better idea, you think this is not necessary or you think the current situation is the best we can do please let me know by commenting on the live version of this document or contacting bioajp@gmail.com or https://twitter.com/AJamesPhillips . Thank you.

If you’re a pathology lab manager, what do you think of this approach to getting our testing increased in a responsible and timely manner?

If you’re in policy, what would need to make this happen?

If you’re an accreditation body, in a non-binding and confidential way, would you support this approach?

If you’re in biotech or business and debating setting up to help run tests, does this resonate with you?

Acknowledgements

In reality this is a very long list but: OpenCell, Bomb.bio, OpenTrons, Crick for all being inspiring. Tom at OpenCell for patiently answering questions. For valuable feedback: Charlotte Roach, Ian Robertson.

One thought on “Continuous Validation: unlocking accurate mass testing

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s