Many important technical decisions are made by groups of technical experts, each of whom speak their own professional language. Such committees are charged with fairly combining information from multiple perspectives to reach a decision that one person could not make alone. This work focuses on techniques for empirically assessing mechanisms for information aggregation among expert groups. Work in this project has reviewed and critiqued leading theories of decision-based design, showing that empirical and axiomatic approaches need not conflict, and demonstrated that leading axiomatic approaches to decision-based design, which suggest that combining preferences across these elements is virtually guaranteed to result in irrational outcomes, are not empirically valid. In so doing, we introduced a new technique to model and measure mental models in engineering design contexts, and showed how this technique can be used to determine the likelihood of “irrational” (i.e., cyclic) design outcomes. Simulation results show that even minimal amounts of structure can vastly reduce the likelihood of irrational outcomes at the level of the group, and that slightly stronger restrictions yield probabilities of irrational preferences that never exceed 5%. These results show how axiomatic consistency can be combined with empirical correspondence to determine the circumstances under which ‘‘dictators’’ are necessary in design decisions. We also used computational text analysis tools to identify and isolate indicators of professional specialization. We showed that these indicators may be used to test theories about how information is shared on FDA expert advisory panels that review cardiac devices. In most cases, panel members marshal their expertise to make decisions about whether devices should be approved; however, panels encountering extremely novel situations are unable to do so and instead engage in search across professional specialty boundaries.
Research on this project is supported in part by the National Institute of General Medical Sciences of the National Institutes of Health under award number R01GM114771, and by the Rheumatology Research Foundation (RRF). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or RRF.
