The NIHR Community Healthcare MIC is pleased to offer a series of workshops for 2018 that will build on the educational programme previously delivered by NIHR DEC Oxford. Running sequentially, the workshops will provide an introduction to evaluation of diagnostics (Workshop 1), cost-effectiveness analysis of diagnostic tests for development and adoption (Workshop 2), and statistical methods for diagnostic accuracy in medical research (Workshop 3). These short courses have been developed as workshops which can be taken individually or combined depending on the requirements of the student. The workshops will be taught by front-line MIC researchers together with colleagues from industry, NICE and the NIHR.
Aimed at all professionals working on diagnostic tests including those working in industry, academia, research funding and regulatory affairs, these workshops will provide delegates with a variety of information on what evidence is needed to obtain regulatory approval, how NICE evaluates new diagnostic technology, how to collect evidence to support adoption in routine clinical practice and different study designs, cost effectiveness analysis and statistical methods for evaluating diagnostic accuracy.
Introduction to Evaluation of Diagnostics
The opening workshop of the series will be taught by frontline diagnostics researchers and will provide participants with a basic overview of the components which one might employ for the evaluation of diagnostic tests, including basic health economics and statistics, searching for evidence, and pragmatic study design. Colleagues from the diagnostics industry and the NIHR will discuss some of the challenges faced by developers of diagnostic tests and potential sources of funding for development and evaluation projects respectively.
There are no prerequisites for attendance. The workshop has been formulated so that it is accessible to all who have an interest in gaining a basic understanding of the evaluation of diagnostic tests.
The course will consist of three sessions of seminars separated by lunch and coffee breaks. Some of the seminars will incorporate short practical exercises.
Course content (subject to minor changes)
1. Tests as part of a clinical pathway
2. Introduction to study design for pragmatic evaluation of diagnostics
3. Basic statistics for diagnostic medicine
4. Basic health economics for diagnostic evaluation
5. Challenges of diagnostic test development – the industry perspective
6. Searching for existing evidence to support regulatory approval and other purposes
7. NIHR funding streams
Cost-effectiveness analysis of diagnostic tests for development and adoption.
This one day course aims to explain how cost-effectiveness analysis of diagnostic tests is conducted and used in the research, development and evaluation of diagnostics. It is aimed at those involved in the development and evaluation of diagnostics, in both academic and commercial settings.
Participants should have a basic understanding of measures of diagnostic accuracy and clinical effectiveness, as well as basic descriptive statistics (means, ranges). Previous knowledge of cost-effectiveness concepts would be an advantage, but these are not necessary as they will be revised in the first session. For the practical session, participants will need a laptop with Microsoft Excel installed.
By taking this course
Participants will revise the core concepts underlying cost-effectiveness analysis, including incremental cost-effectiveness ratios. They will learn how diagnostic tests can be evaluated using cost-effectiveness methods, and will get practical experience of working with a pre-built decision model and performing sensitivity analysis to understand how cost-effective a diagnostic would be in different settings. They will also learn about the diagnostic evaluation evidence requirements used by NICE, and consider contemporary issues related to cost-effectiveness analysis of diagnostics.
- Introduction to cost-effectiveness for diagnostics
- Evaluating the cost-consequences of testing
- Quality-adjusted life years
- Incremental cost-effectiveness ratios
- Linking test results to outcomes
- Introduction to decision models
2. Generating and using cost-effectiveness evidence for diagnostics
- Identifying which evidence is needed
- Revising evidence sources
- Selecting parameters
- Model assumptions
3. Evidence for implementation in routine clinical practice – NICE evaluations
4. Practical: Identifying evidence, decision modelling, and evaluating determinants of cost-effectiveness
- Search strategies
- Performing deterministic sensitivity analysis
- Identifying threshold values in early evaluations
5. Outstanding issues in cost-effectiveness of diagnostics: AMR, diagnostic delay and going beyond the QALY.
Statistical methods for diagnostic accuracy in medical research
This one day course is predominately designed for researchers who want to analyse their own diagnostic accuracy data. Typically, this will be analysis on results from research carried out in an early development/exploratory phase but the methods covered will also be applicable to research in an advanced stage where tests are studied in a clinical setting. The course content is geared towards teaching “how to” rather than “why” and will make extensive use of practical sessions so that participants can gain hands-on experience of analysing diagnostic accuracy data. In the practical sessions, participants will use the free statistical software R (https://cran.r-project.org/) for completing the analyses.
Participants should have a basic understanding of statistics up to the level of confidence intervals and p values. A rudimentary knowledge of the basics of diagnostic accuracy would be advantageous but it is not necessary as we will revise the basics in the first session. Participants will be expected to bring their own laptops and have R and RStudio installed.
By taking this course:
The course will consist of four sessions with demonstrations and practical exercises. Participants will revise their understanding of common measures of diagnostic accuracy, learn how to calculate measures of diagnostic accuracy from data and quantify uncertainty in their estimates. Then learn when and how you should use ROC analysis, how to compare two diagnostic tests and finish with methods for finding optimal thresholds.
Course content in detail
1. Single summary estimates of diagnostic accuracy.
- Common measures of DA - sensitivity, specificity,
- Other summary measures- Diagnostic odds ratio, Youden’s index. Etc.
- LR+ and LR-
- Bayes theorem, PPV and NPV and prevalence
- Simple sample sizes calculations.
2. Receiver operating characteristic
- ROC space and ROC curves (pros and cons)
- Area under the curve measures (partial area under the curve)
- Smooth (parametric) ROC curves
3. Hypothesis tests for comparing the accuracy of two tests
- Tests based on either sensitivity or specificity
- Comparing two ROC AUC’s curves
- Paired and non-paired designs
4. Methods for finding optimal thresholds.
- Methods based on sensitivity and specificity alone
- Optimal thresholds based on costs (weighting for false pos and false negs)
- Graphical summaries (Net-benefit measure etc)