Watching the Watchmen: Assessment-Biases in Waiting List Prioritization for the Delivery of Mental Health Services

Keywords: patient prioritization tools, illness severity assessment, rater-based effects, mental health


Purpose: While the demand for mental health services increases, supply often stagnates. Providing treatment to those most in need is an important factor in its efficient distribution. We propose and conduct a statistical procedure for detecting rater-biases in patient prioritization tools.

Design / Method / Approach: We gather real-life data from 266 illness severity assessments in an Austrian publicly funded mental health service provider, including a rich set of covariates. To ensure robustness, we merge this data with determinants of mental health and assessment identified by previous research, such as weather or seasonal indicators.

Findings: We find statistically significant effects of rater-biases. These effects are robust to a large array of controls.

Practical Implications: A back-of-the-envelope calculation reveals that the identified rater effects can translate to large changes in the waiting times for patients. Misspecified treatment allocations may lead to worsened symptoms and potentially fatal outcomes.

Originality / Value: Although a growing literature focuses on patient prioritization tools, many articles study these in synthetic contexts using “vignettes”. In comparison, our study adds external validity by considering real-life treatments in the field.

Research Limitations / Future Research: This study can be used as a starting point for deeper, causally focused studies.

Disclaimer: In accordance with publisher policies and our ethical obligations as researchers, we report that one of the authors is employed at a company that may be affected by the research reported in the enclosed paper. We have disclosed those interests fully.

Paper type: Empirical


Download data is not yet available.


Adams-Prassl, A., Boneva, T., Golin, M., & Rauh, C. (2020). The impact of the coronavirus lockdown on mental health: Evidence from the US.

Alexopoulos, E. C. (2010). Introduction to multivariate regression analysis. Hippokratia, 14(Suppl 1), 23–28. Retrieved from

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders: DSM-IV (4th ed.). American Psychiatric Association.

Arslanian-Engoren, C. (2000). Gender and age bias in triage decisions. Journal of Emergency Nursing, 26(2), 117–124.

Arslanian-Engoren, C., & Scott, L. D. (2016). Women's perceptions of biases and barriers in their myocardial infarction triage experience. Heart & Lung : The Journal of Critical Care, 45(3), 166–172.

Bakhshi, S., Kanuparthy, P., & Gilbert, E. (2014). Demographics, weather and online reviews. In C.-W. Chung, A. Broder, K. Shim, & T. Suel (Eds.), Proceedings of the 23rd international conference on World wide web - WWW '14 (pp. 443–454). ACM Press.

Bell, I., & Mellor, D. (2009). Clinical judgements: Research and practice. Australian Psychologist, 44(2), 112–121.

Berufsverband Österreichischer PsychologInnen, & Karmasin Research & Identity. (2020). Psychische Gesundheit in Österreich. Retrieved from

Bless, H., Schwarz, N., & Kemmelmeier, M. (1996). Mood and Stereotyping: Affective States and the Use of General Knowledge Structures. European Review of Social Psychology, 7(1), 63–93.

Bowes, S. M., Ammirati, R. J., Costello, T. H., Basterfield, C., & Lilienfeld, S. O. (2020). Cognitive biases, heuristics, and logical fallacies in clinical practice: A brief field guide for practicing clinicians and supervisors. Professional Psychology: Research and Practice, 51(5), 435–445.

Brooks, S. K., Webster, R. K., Smith, L. E., Woodland, L., Wessely, S., Greenberg, N., & Rubin, G. J. (2020). The psychological impact of quarantine and how to reduce it: rapid review of the evidence. The Lancet, 395(10227), 912–920.

Chan, J., & Wang, J. (2018). Hiring Preferences in Online Labor Markets: Evidence of a Female Hiring Bias. Management Science, 64(7), 2973–2994.

Christensen-Szalanski, J. J., Diehr, P. H., Bushyhead, J. B., & Wood, R. W. (1982). Two studies of good clinical judgment. Medical Decision Making, 2(3), 275–283.

Clark, D. M., Canvin, L., Green, J., Layard, R., Pilling, S., & Janecka, M. (2018). Transparency about the outcomes of mental health services (IAPT approach): an analysis of public data. The Lancet, 391(10121), 679–686.

Corrigan, P. W., Druss, B. G., & Perlick, D. A. (2014). The Impact of Mental Illness Stigma on Seeking and Participating in Mental Health Care. Psychological Science in the Public Interest, 15(2), 37–70.

Coster, C. de, McMillan, S., Brant, R., McGurran, J., & Noseworthy, T. (2007). The Western Canada Waiting List Project: Development of a priority referral score for hip and knee arthroplasty. Journal of Evaluation in Clinical Practice, 13(2), 192-6; quiz 197.

Cowden, R. G., Davis, E. B., Counted, V., Chen, Y., Rueger, S. Y., VanderWeele, T. J., Lemke, A. W., Glowiak, K. J., & Worthington, E. L. (2021). Suffering, Mental Health, and Psychological Well-being During the COVID-19 Pandemic: A Longitudinal Study of U.S. Adults With Chronic Health Conditions. Wellbeing, Space and Society, 2, 100048.

Croskerry, P. (2002). Achieving quality in clinical decision making: cognitive strategies and detection of bias. Academic Emergency Medicine, 9(11), 1184–1204.

Déry, J., Ruiz, A., Routhier, F., Bélanger, V., Côté, A., Ait-Kadi, D., Gagnon, M.‑P., Deslauriers, S., Lopes Pecora, A. T., Redondo, E., Allaire, A.‑S., & Lamontagne, M.‑E. (2020). A systematic review of patient prioritization tools in non-emergency healthcare services. Systematic Reviews, 9(1), Article 227, 1–14.

Earp, B. D., Monrad, J. T., LaFrance, M., Bargh, J. A., Cohen, L. L., & Richeson, J. A. (2019). Featured Article: Gender Bias in Pediatric Pain Assessment. Journal of Pediatric Psychology, 44(4), 403–414.

Endicott, J., Spitzer, R. L., Fleiss, J. L., & Cohen, J. (1976). The global assessment scale. A procedure for measuring overall severity of psychiatric disturbance. Archives of General Psychiatry, 33(6), 766–771.

FitzGerald, C., & Hurst, S. (2017). Implicit bias in healthcare professionals: A systematic review. BMC Medical Ethics, 18(1), 19.

Gingerich, A., Regehr, G., & Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Academic Medicine, 86(10), 1-7.

Glied, S., & Pine, D. S. (2002). Consequences and correlates of adolescent depression. Archives of Pediatrics & Adolescent Medicine, 156(10), 1009–1014.

Goetzmann, W. N., & Zhu, N. (2005). Rain or Shine: Where is the Weather Effect? European Financial Management, 11(5), 559–578.

Graaf, R. de, van Dorsselaer, S., Have, M. ten, Schoemaker, C., & Vollebergh, W. A. M. (2005). Seasonal variations in mental disorders in the general population of a country with a maritime climate: Findings from the Netherlands mental health survey and incidence study. American Journal of Epidemiology, 162(7), 654–661.

Hadorn, D. C., & Steering Committee of the Western Canada Waiting List Project (2000). Setting priorities for waiting lists: defining our terms. Cmaj, 163(7), 857–860. Retrieved from

Hairston, D. R., Gibbs, T. A., Wong, S. S., & Jordan, A. (2019). Clinician Bias in Diagnosis and Treatment. In M. M. Medlock, D. Shtasel, N.-H. T. Trinh, & D. R. Williams (Eds.), Racism and Psychiatry (pp. 105–137). Springer International Publishing.

Harding, K. E., & Taylor, N. (2013). Triage in Nonemergency Services. In R. Hall (Ed.), International Series in Operations Research & Management Science. Patient Flow (Vol. 206, pp. 229–250). Springer US.

Harries, P., & Gilhooly, K. (2011). Training Novices to Make Expert, Occupationally Focused, Community Mental Health Referral Decisions. British Journal of Occupational Therapy, 74(2), 58–65.

Hibbing, J. R., Smith, K. B., & Alford, J. R. (2014). Differences in negativity bias underlie variations in political ideology. Behavioral and Brain Sciences, 37(3), 297–307.

Hirshleifer, D., & Shumway, T. (2003). Good Day Sunshine: Stock Returns and the Weather. The Journal of Finance, 58(3), 1009–1032.

James, S. L., Abate, D., Abate, K. H., Abay, S. M., Abbafati, C., Abbasi, N., ... & Briggs, A. M. (2018). Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. The Lancet, 392(10159), 1789-1858.

Jeong, Y., & Jung, M. J. (2016). Application and Interpretation of Hierarchical Multiple Regression. Orthopedic Nursing, 35(5), 338–341.

Kassin, S. M., Dror, I. E., & Kukucka, J. (2013). The forensic confirmation bias: Problems, perspectives, and proposed solutions. Journal of Applied Research in Memory and Cognition, 2(1), 42–52.

Keithly, L. J., Samples, S. J., & Strupp, H. H. (1980). Patient motivation as a predictor of process and outcome in psychotherapy. Psychotherapy and Psychosomatics, 33(1-2), 87–97.

Kieling, C., Baker-Henningham, H., Belfer, M., Conti, G., Ertem, I., Omigbodun, O., Rohde, L. A., Srinath, S., Ulkuer, N., & Rahman, A. (2011). Child and adolescent mental health worldwide: evidence for action. The Lancet, 378(9801), 1515–1525.

Kimberlin, C. L., & Winterstein, A. G. (2008). Validity and reliability of measurement instruments used in research. American Journal of Health-System Pharmacy, 65(23), 2276–2284.

Lipson, S. K., Lattie, E. G., & Eisenberg, D. (2019). Increased Rates of Mental Health Service Utilization by U.S. College Students: 10-Year Population-Level Trends (2007-2017). Psychiatric Services, 70(1), 60–63.

Lizaur-Utrilla, A., Martinez-Mendez, D., Miralles-Muñoz, F. A., Marco-Gomez, L., & Lopez-Prats, F. A. (2016). Negative impact of waiting time for primary total knee arthroplasty on satisfaction and patient-reported outcome. International Orthopaedics, 40(11), 2303–2307.

López, S. R. (1989). Patient variable biases in clinical judgment: Conceptual overview and methodological considerations. Psychological Bulletin, 106(2), 184–203.

Luigi, S., Michael, B., & Valerie, M. (2013). OECD health policy studies waiting time policies in the health sector: What works? Oecd Publishing. Retrieved from

MacCormick, A. D., Collecutt, W. G., & Parry, B. R. (2003). Prioritizing patients for elective surgery: A systematic review. ANZ Journal of Surgery, 73(8), 633–642.

Magnusson, A. (2000). An overview of epidemiological studies on seasonal affective disorder. Acta Psychiatrica Scandinavica, 101(3), 176–184.

Magnusson, A., & Boivin, D. (2003). Seasonal affective disorder: An overview. Chronobiology International, 20(2), 189–207.

Malouff, J. (2008). Bias in Grading. College Teaching, 56(3), 191–192.

McDermott, P. A., Watkins, M. W., & Rhoad, A. M. (2014). Whose IQ is it? Assessor bias variance in high-stakes psychological assessment. Psychological Assessment, 26(1), 207–214.

McIntyre, D., & Chow, C. K. (2020). Waiting Time as an Indicator for Health Services Under Strain: A Narrative Review. Inquiry : A Journal of Medical Care Organization, Provision and Financing, 57, 46958020910305.

Meier, A. N., Schmid, L., & Stutzer, A. (2019). Rain, emotions and voting for the status quo. European Economic Review, 119, 434–451.

Mojtabai, R., Olfson, M., & Han, B. (2016). National Trends in the Prevalence and Treatment of Depression in Adolescents and Young Adults. Pediatrics, 138(6).

Murray, K. B., Di Muro, F., Finn, A., & Popkowski Leszczyc, P. (2010). The effect of weather on consumer spending. Journal of Retailing and Consumer Services, 17(6), 512–520.

Murrie, D. C., Boccaccini, M. T., Guarnera, L. A., & Rufino, K. A. (2013). Are forensic experts biased by the side that retained them? Psychological Science, 24(10), 1889–1897.

Norman, G. R., & Eva, K. W. (2010). Diagnostic error and clinical reasoning. Medical Education, 44(1), 94–100.

Nottingham, Q. J., Johnson, D. M., & Russell, R. S. (2018). The Effect of Waiting Time on Patient Perceptions of Care Quality. Quality Management Journal, 25(1), 32–45.

Patel, V. L., Kaufman, D. R., & Arocha, J. F. (2002). Emerging paradigms of cognition in medical decision-making. Journal of Biomedical Informatics, 35(1), 52–75.

Pathirana, T. I., & Jackson, C. A. (2018). Socioeconomic status and multimorbidity: A systematic review and meta-analysis. Australian and New Zealand Journal of Public Health, 42(2), 186–194.

Pieh, C., Budimir, S., & Probst, T. (2020). The effect of age, gender, income, work, and physical activity on mental health during coronavirus disease (COVID-19) lockdown in Austria. Journal of Psychosomatic Research, 136, 110186.

Platts-Mills, T. F., Travers, D., Biese, K., McCall, B., Kizer, S., LaMantia, M., Busby-Whitehead, J., & Cairns, C. B. (2010). Accuracy of the Emergency Severity Index triage instrument for identifying elder emergency department patients receiving an immediate life-saving intervention. Academic Emergency Medicine : Official Journal of the Society for Academic Emergency Medicine, 17(3), 238–243.

Raymond, M.‑H., Demers, L., & Feldman, D. E. (2017). Differences in Waiting List Prioritization Preferences of Occupational Therapists, Elderly People, and Persons With Disabilities: A Discrete Choice Experiment. Archives of Physical Medicine and Rehabilitation, 99(1), 35-42.

Rechnungshof. (2019). Bericht des Rechnungshofes: Versorgung psychisch Erkrankter durch die Sozialversicherung (BUND 2019/8). Wien. Retrieved from

Reichert, A., & Jacobs, R. (2018). The impact of waiting time on patient outcomes: Evidence from early intervention in psychosis services in England. Health Economics, 27(11), 1772–1787.

Reynolds, C. R., & Suzuki, L. A. (2013). Bias in psychological assessment: An empirical review and recommendations. In Handbook of psychology: Assessment psychology, Vol. 10, 2nd ed (pp. 82–113). John Wiley & Sons, Inc.

Roper, R. L. (2019). Does Gender Bias Still Affect Women in Science? Microbiology and Molecular Biology Reviews : MMBR, 83(3).

Rossi, R., Socci, V., Talevi, D., Mensi, S., Niolu, C., Pacitti, F., Di Marco, A., Rossi, A., Siracusano, A., & Di Lorenzo, G. (2020). Covid-19 Pandemic and Lockdown Measures Impact on Mental Health Among the General Population in Italy. Frontiers in Psychiatry, 11, 790.

Rössler, W. (2012). Stress, burnout, and job dissatisfaction in mental health workers. European Archives of Psychiatry and Clinical Neuroscience, 262(S2), S65-9.

Samelius, L., Wijma, B., Wingren, G., & Wijma, K. (2010). Lifetime history of abuse, suffering and psychological health. Nordic Journal of Psychiatry, 64(4), 227–232.

Samuel, D. B., & Bucher, M. A. (2017). Assessing the assessors: The feasibility and validity of clinicians as a source for personality disorder research. Personality Disorders, 8(2), 104–112.

Schuster, D. P., & Powers, W. J. (2005). Translational and experimental clinical research. Lippincott Williams & Wilkins.

Shor, E., van de Rijt, A., & Fotouhi, B. (2019). A Large-Scale Test of Gender Bias in the Media. Sociological Science, 6, 526–550.

Sifneos, P. E. (1978). Motivation for Change A Prognostic Guide for Successful Psychotherapy. Psychotherapy and Psychosomatics, 29(1/4), 293–298.

Slaunwhite, A. K., Ronis, S. T., Peters, P. A., & Miller, D. (2019). Seasonal variations in psychiatric admissions to hospital. Canadian Psychology/Psychologie Canadienne, 60(3), 155–164.

Suresh, S. (2014). Nursing Research and Statistics (2nd ed.). Elsevier Health Sciences APAC. Retrieved from

Talevi, D., Socci, V., Carai, M., Carnaghi, G., Faleri, S., Trebbi, E., Di Bernardo, A., Capelli, F., & Pacitti, F. (2020). Mental health outcomes of the CoViD-19 pandemic. Rivista Di Psichiatria, 55(3), 137–144.

Thomas, O. (2018). Two decades of cognitive bias research in entrepreneurship: What do we know and where do we go from here? Management Review Quarterly, 68(2), 107–143.

Twenge, J. M., Cooper, A. B., Joiner, T. E., Duffy, M. E., & Binau, S. G. (2019). Age, period, and cohort trends in mood disorder indicators and suicide-related outcomes in a nationally representative dataset, 2005-2017. Journal of Abnormal Psychology, 128(3), 185–199.

Ulasi, I. (2008). Gender bias in access to healthcare in Nigeria: A study of end-stage renal disease. Tropical Doctor, 38(1), 50–52.

van Ryn, M., & Burke, J. (2000). The effect of patient race and socio-economic status on physicians' perceptions of patients. Social Science & Medicine, 50(6), 813–828.

Vries, M. de, Holland, R. W., Corneille, O., Rondeel, E., & Witteman, C. L. (2012). Mood effects on dominated choices: Positive mood induces departures from logical rules. Journal of Behavioral Decision Making, 25(1), 74–81.

West, R. F., Toplak, M. E., & Stanovich, K. E. (2008). Heuristics and biases as measures of critical thinking: Associations with cognitive ability and thinking dispositions. Journal of Educational Psychology, 100(4), 930–941.

Wolfson, A. M., Doctor, J. N., & Burns, S. P. (2000). Clinician judgments of functional outcomes: How bias and perceived accuracy affect rating. Archives of Physical Medicine and Rehabilitation, 81(12), 1567–1574.

Wynn, A. T., & Correll, S. J. (2018). Combating Gender Bias in Modern Workplaces. In B. J. Risman, C. M. Froyum, & W. J. Scarborough (Eds.), Handbooks of Sociology and Social Research. Handbook of the Sociology of Gender (pp. 509–521). Springer International Publishing.

Yourstone, J., Lindholm, T., Grann, M., & Svenson, O. (2008). Evidence of gender bias in legal insanity evaluations: A case vignette study of clinicians, judges and students. Nordic Journal of Psychiatry, 62(4), 273–278.

How to Cite
Kreiseder, F., & Mosenhauer, M. (2022). Watching the Watchmen: Assessment-Biases in Waiting List Prioritization for the Delivery of Mental Health Services. European Journal of Management Issues, 30(1), 3-16.