Guest blog post by David Byrne

Models have been widely deployed in scientific discussion of the likely course of the COVID-19 pandemic to explore the potential impact of different policy interventions. However, any model is a necessary simplification of the system it describes.

Covid-19 is a biological intervention in the complex social systems – the plural is very important – which include human social interactions and policy interventions within existing social relations and institutional structures. These systems have emergent properties. That is not to say that modelling is useless but its use is necessarily limited.

Existing models are basically modifications of traditional epidemiological models of infectious disease transmission with parameters changed to reflect different timings and degrees of social interaction in response to social distancing and lock down regulatory interventions. There is a real problem of scale. Have these models got the scale of the system right before any attempt at description, let alone prediction, is attempted?

Communicable disease public health doctors have consistently made the point that what we have is not one nationwide outbreak but a set of local outbreaks which is why isolation of cases, tracing of contacts and isolation of those contacts is such an important part of the public health armoury. Most modelling seemed to deal only with the national scale and looked at the impact of policies like lockdown at that level although I am aware that local modelling is being attempted. Few models have examined in detail the impact of case isolation, tracing and contact isolation despite this having been a successful strategy in South Korea and elsewhere. An exception is  Kretzschmar, et al (2020). Plainly this is a very important set of interventions to consider.

Although an alternative form of approach has no immediate predictive capacity it is absolutely necessary to develop it in order to learn from this experience, for similar outbreaks will happen again. That approach is case-based process tracing and systematic case-comparison to establish what has worked better. That needs setting up now and whilst data is essential for developing it, and modelling can play a retrospective role if done at the right scale and with full incorporation of structural elements, it is not the only or even the best way to establish wha has worked where.

How might we  learn from this first wave of COVID-19 in order to find out what approaches have worked or not and in what contexts they have worked or not? Note the emphasized plural. Interventions have been interventions in different local complex systems and have themselves been complex. At national or even sub-national scales (where sub-national governments as with provinces in Canada have had appropriate powers) they have combined public health regulatory regimes (again note the plural) – different regimes in different places – with different levels of curative intervention depending on resources and even perhaps (on some limited evidence) different curative approaches, particularly in relation to the diagnostic anticipation and intervening prevention of cytokine storms. On this see the interviews with Chinese Intensive Care Physicians here: https://www.newscientist.com/article/mg24632783-600-wuhans-covid-19-crisis-intensive-care-doctors-share-their-stories/.

The first thing we need to know is just what has been done in different places alongside descriptions of the spatial and temporal contexts in which those things were done. We need careful process tracing and that means that we need good recording of what things were done in reasonable detail. This is a norm of any complex engineering production process but health systems are weak at full case recording other than those insurance based systems which generate financial records for costing. There have been attempts to improve this in non-insurance based systems but at present they are not fully developed. That kind of recording might be useful at the level of the individual patient and might provide the basis of a new wave of learning algorithm-based data mining to guide intervention, but it does not take account of institutional interventions at higher levels. It will be useful, indeed essential, in establishing treatment protocols – the sheer uselessness and inappropriateness of Randomized Controlled Trials other than for vaccine testing in a pandemic is obvious. It will not guide overall health system management.

There are well established tools in evaluation which can deal with the issue of post hoc exploration of what has worked in different contexts this time to guide policy and practice for next time. These are inherently mixed method in that they require the construction of narratives of what has been done, a mix of descriptive quantitative and qualitative specification of the contexts in which things have been done, and the use of data generated from those account to establish the multiple forms of intervention which have worked to different degrees. Equifinality rules OK!  The same outcome – control over the impact of the disease – can be generated in different and multiple ways. We need comparative process tracing based exploration of the multiple and complex ways in which systems have generated different outcomes.

This is precisely the set of problems addressed by  CECAN – a multi research council and UK government department funded investigation into the problems of Evaluating Complex Interventions Across the NEXUS (food, environment, water and energy). A range of approaches have been developed for this purpose:  CECAN’s website provides a full listing, https://www.cecan.ac.uk/.

Developed outside CECAN but interacting with it has been the very interesting approach of Dynamic Pattern Synthesis devised by Phil Haynes (see Haynes 2019).

This combines exploratory cluster analyses with Qualitative Comparative Analysis to explore how policy and practice systems have come to the outcomes they have reached.

Fundamental to this way of finding “what works” is a combination of qualitative materials and quantitative data. QCA – which is one tool but a good one with an established literature of effective use – requires the interpretation of qualitative narrative accounts of process to yield quantitative descriptions of interventions alongside quantitative descriptions of context. For an example of how this can be done see: Blackman et al.(2013).  Note that the level of measurement is often simply binary or at best ordinal specification of the attributes of the systems and of the interventions made within them.

Demanding documentation during crises is a hard thing to do but the construction of narratives, preferably on an ongoing real time basis but if necessary by careful historical investigation is absolutely necessary. We must always be able to say what has been done and if we can’t then we won’t learn what needs to be done.

References

Blackman et al.(2013). “Using Qualitative Comparative Analysis to understand complex policy problems.” Evaluation 19(2):126-140.

Haynes, Phil (2019). Social Synthesis – Finding Dynamic Patterns in Complex Social Systems. Abingdon: Routledge.