A pragmatic trial of an Artificial intelligence DRiven appOInTment maNagEment SyStem (ADROITNESS) in NHS outpatient departments

DrDoctor is an artificial intelligence system that seeks to predict the probability of outpatient not attending their appointment (phase 1) predicting linked clinics (phase2a) and predicting out of sequence linked appointments (phase 2b). LSBU were commissioned by NHS England to evaluate the use of the DrDoctor system in two NHS trusts to see if it is accurate, safe, effective provides value and explore person centred aspects. Phase 1 of the evaluation examines the accuracy of the model in predicting those most likely not to attend outpatient appointments – Do Not Attend (DNA) probability. Report covers approach to evaluation detailed findings, conclusions and recommendations. Phase 1 work stream 3 examined How effective is the DrDoctor AI system? covering rationale, objectives, hypotheses, methods and results for this work stream. The Phase 1 technology appeared to be accurate in terms of DNA prediction and was safe and mostly acceptable to patients. Phase 2a of the evaluation examines the process adopted in identifying clinics with the same specialisms so that patients can be offered greater appointment choice. It is recognized that some trusts have built digital hierarchies that show this but other trusts have not and it is for the latter category that this phase provides potential benefit. Report covers approach to evaluation detailed findings, conclusions and recommendations. The Phase 2a report was not able to evaluate outcomes (and therefore overall accuracy) due to the delays in the DrDoctor system development. A key recommendation is that a clinical review of accuracy of the predicted linked clinics is undertaken. Phase 2b evaluated the process adopted for identifying out of sequence linked appointments. For example a diagnostic test appointment occurring after the clinician review. Report covers approach to evaluation detailed findings, conclusions and recommendations. This report considered the process adopted for phase2b. Due to delays in DrDoctor Phase 2b development the report excluded consideration of outcomes. A key recommendation was the clinical review of its predicted linked appointments out of sequence for their accuracy and also consideration of missed out of sequence linked appointments.