International Early Stage Researcher Training School on Applying Expert  
Judgement Methodologies to Real Problems  

Welcome Reception at a restaurant, including introductions, a brief overview of the course, a short elicitation exercise and Dinner.

Led by Simon French.

Car transit for those staying at Erasmo residence.

David Ríos.

Lecture on Cooke’s Classical Method, including some case study material, and demonstration of the Excalibur software.

Roger Cooke.

Lecture on the behavioural background: System 1 and System 2 thinking, heuristics and biases, group behaviours, relevance to debates about the relative merits of behavioural and mathematical aggregation.

Eva Chen.

Coffee and small group discussion with mentors. Note that the course directors will provide the mentors with a series of suggestions and topics for discussion to ensure that these sessions have structure.

Groups led by mentors.

Experiential Exercise: Feedback on the previous evening’s results. Training in how to make judgements, i.e. the participants will experience the training that they would need to provide as analysts to their experts. Structured elicitation of their individual judgements on further calibration variables and quantities of interest that will become resolved by Day 2.

Initial feedback. Simon French.
Trainer: Oswaldo Morales.
Panel: Oswaldo Morales (chair), Roger Cooke, Eva Chen, Simon French, Jesus. Rios, David Rios.

Short Presentation of a Case Study.

Jesus Rios. Strategic financial planning.

Introduction to the group exercise in a plenary session. Overall description of the exercise. The participants will be divided into groups and given a topic on which several of the lecturers and mentors can act as experts – i.e. from experience of past applications they have sufficient contextual knowledge to provide judgemental inputs. Note: these groups need not necessarily be the same as the groups meeting at coffee time.

Simon French

Group work identifying key uncertainties on which they would need expert judgemental input in their exercise, potential calibration data sets, etc. Each group would be assigned a mentor/tutor to provide guidance on process, tools and methods, but not on context. Note each group will work on the same problem but independently.

Groups with mentors.

Drinks and talk by guest speaker on their experience of the use of expert judgement in an applied study or studies.

Oswaldo Morales.

Lecture on expert judgement theory (other than Cooke’s Method) covering topics such as Bayesian approaches, group decision making, meta-analysis etc.

Simon French

Short talk covering case studies or research fronts that the mentors/course lecturers are involved with.

David Rios. Aviation Safety Project.

Experiential Exercise: Session 3. Feedback and discussion of the results from the previous day exercises. Learning points, etc.

Simon French

Coffee and small group discussion with mentors.

Groups with mentors.

Design, reporting and peer review of expert judgement studies. Lecture and discussion. The aim of this session is to emphasise the softer professional skills needed in the community of expert judgement analysts to ensure that such studies are carried out rigorously and defensibly. The session will involve an exercise critiquing one or more reports of expert judgement studies. (Note: participants will have been given copies of these in their packs on arrival and asked to read them in their free study time).

Led by Roger Cooke.

Short talk covering case studies or research fronts that the mentors/course lecturers are involved with.

Eva Chen. The good judgement project.
Simon French and David Hartley. Bayesian approaches to calibration and combination.

Group Exercise. Elicitation of judgements from ‘experts’. The mentors and other course lecturers will visit the groups in rotation, role playing experts for this. Analysis and development of a group report on the results.

Groups with mentors.

School Dinner in Central Madrid.

Car transit for those staying at Erasmo residence

Group exercise. Finalising analyses, reports and developing a presentation on the results of the exercise. (Coffee to be available to the groups).

Groups with mentors.

Group presentations of the results of the exercise and discussion.

Simon French (chair).

General discussion on how they will take their skills forward in their research and professional activity.

Roger Cooke (chair).