When 'dependent' expert reports might be more informative than independent ones

Whether it's Government ministers deciding if it is safe to end Covid-19 lockdown, journal editors deciding if a research paper is worthy of publication, or just consumers deciding which kettle is best value, we have to rely on evaluating information from multiple 'experts' who may or may not agree on their conclusion. In determining which conclusion is most probable we have to take account of not just which experts we trust most, but also the extent to which the experts may or may not have collaborated.  This problem is especially pertinent in intelligence analysis work, and was addressed as part of a recent project funded by IARPA (Intelligence Advanced Research Projects Activity)*. While it is always assumed intutively that 'independence' among experts is advantageous, it turns out - as shown in a paper by Pilditch et al (researchers at UCL, Queen Mary and Birkbeck) just accepted for publication in Cognition, that this is not always the case.

Consider the following scenario:

A plane has crashed, and you must determine whether it was sabotage. You await the crash site reports from two investigators, Bailey and Campbell. They have separately assessed the various pieces of wreckage before leaving to write up their conclusions. Both investigators are equally accurate in their conclusions, seldom making mistakes. Now consider two alternative cases:

i. Bailey provides a report in which she concludes the plane was sabotaged, but she has also seen Campbell’s report, in which Campbell likewise concluded that the plane was sabotaged.

ii. Bailey provides a report in which she concludes the plane was sabotaged, based on her assessment alone. Campbell then separately provides a report (based on his assessment alone), likewise concluding that the plane was sabotaged.

Here, i) is a case of corroborating reports with a directional dependence from Campbell to Bailey (i.e., Bailey has seen Campbell’s report, thus Bailey’s report may depend upon Campbell’s, but not vice-versa), and ii) is a case of corroborating reports coming from independent sources. Given the two reports in each case, it would be right to conclude that more support for the conclusion that the plane was sabotaged is provided in the independent case.  However, now consider the same scenario, with two slight alterations:

1. Bailey reports to you the plane was sabotaged, having seen Campbell’s report, but you do not know what Campbell concluded (case i), versus you only know Bailey’s independent conclusion of sabotage (case ii).

2. Bailey reports to you the plane was sabotaged, having seen Campbell’s report, but you know that Campbell concluded the opposite (case i), versus you only know that Bailey and Campbell have independently provided contradictory conclusions (case ii).

1) is an instance of partial information, and 2) an instance of contradicting information. In both these instances, it is less clear whether case i) or ii) provides more support for the sabotage hypothesis.

The paper demonstrates that for partial or contradicting information, the dependent case (i) is, in fact, superior (i.e., there is a dependency advantage, in that more evidential support is provided to the hypothesis when a report is the result of a structural dependency (i) than when independent (ii)) – at least given reasonable assumptions.

Full reference:

Pilditch, T., Hahn, U., Fenton, N. E., & Lagnado, D. A. (2020). "Dependencies in evidential reports: The case for informational advantages". Cognition, to appear.  Accepted version (pdf)

*The research was supported in part by The Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), under Contract [2017-16122000003]. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. The research was also supported in part by the Leverhulme Trust under Grant RPG-2016-118 CAUSAL-DYNAMICS. The authors acknowledge and Agena Ltd for software support.