Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

Abstract

Chain-of-thought (CoT) prompting encourages large language models (LLMs) to provide step-by-step explanations for their predictions, potentially making their decisions more interpretable. However, these explanations may not faithfully represent the factors that actually influence model outputs. In this paper, the authors investigate the faithfulness of CoT explanations generated by GPT-3.5 and Claude 1.0. They introduce two biasing features in tasks from BIG-Bench Hard: arranging few-shot examples so that the correct answer is always (A), and suggesting a particular answer in the prompt. They also examine how social stereotypes influence explanations using the Bias Benchmark for QA (BBQ). Across 13 BIG-Bench Hard tasks, prompting biases reduce accuracy by as much as 36.3 percentage points, while model explanations generally fail to acknowledge these influences and often rationalize incorrect answers. In the social-bias experiments, models sometimes weigh evidence inconsistently to support stereotype-aligned predictions. The findings highlight that plausible CoT explanations are not necessarily faithful and underscore the need for more reliable methods to evaluate and improve LLM interpretability.

Pegah Khorasani

Graduate Research Associate

As a doctoral candidate at Arizona State University (ASU), I am currently conducting research under the guidance of Professor Hassan Ghasemzadeh at the Embedded Machine Intelligence Lab (EMIL). My research interests encompass a range of topics, including machine learning, health monitoring system development, and mobile health.