Does Generative AI Exhibit Human-Like Cognitive Biases? A Comprehensive Evaluation

Tomer Geva

Tel Aviv University

Does Generative AI Exhibit Human-Like Cognitive Biases?
A Comprehensive Evaluation

Identifying cognitive biases in Large Language Models (LLMs) is essential given the growing reliance on generative AI in sensitive, high-stake contexts such as healthcare, law enforcement, and finance. Prior studies investigating human-like cognitive biases in LLMs have reported mixed and contradictory findings. These conflicting findings may be due to methodological shortcomings shared by many related studies including small sample sizes, unstructured experimental design selection procedure, and insufficient control for multiple variables during modeling. To address these gaps, this paper provides, to our knowledge, one of the most comprehensive assessments of cognitive biases in LLMs to date. We employ a new structured and unbiased selection process to test 20 randomly chosen cognitive biases across multiple prominent LLMs, and we utilize meta-regression to evaluate and control for diverse factors simultaneously. Our contributions include the proposal of a rigorous experimental selection protocol and novel findings about factors associated with LLM biases. Our findings notably reveal that recognition of cognitive biases and prompt characteristics are significantly related to the magnitude of biases observed in LLM outputs, offering important insights into biases in generative AI systems.