As large language models (LLMs) are increasingly deployed in bias-sensitive contexts, concerns have emerged regarding whether training data distributions may reproduce or amplify existing social value biases. Many recent bias evaluation frameworks rely on model-generated judgments to improve scalability and efficiency, thereby reducing direct expert involvement. This shift raises concerns about whether model-based evaluation can adequately capture implicit, culturally embedded, and context-dependent forms of bias. Thus, this research constructs a Traditional Chinese gender bias corpus derived from contemporary social media discourse reflecting real-world gender debates. After extensive data cleaning and curation, we developed a LLM corpus containing 150 evaluation items. Each item is paired with model-generated justification and expert-provided justification. Semantic distance is computed using transformer-based sentence embeddings to measure reasoning divergence. A topic modeling framework is applied across all justifications to identify latent thematic structures and compare reasoning patterns. Results reveal systematic interpretive divergence. Model justifications predominantly emphasize abstract principles such as equality, neutrality, and anti-discrimination. In contrast, expert reasoning foregrounds relational roles, contextual interpretation, and culturally embedded value hierarchies. Topic transition analysis shows consistent shifts from principle-based framing in model outputs to context-dependent relational reasoning in expert assessments. The findings indicate that misalignment extends beyond lexical differences to structural disparities in interpretive framing. The study underscores the importance of reasoning-level evaluation in low-resource cultural contexts, where bias judgments often require integrating layered social meanings rather than identifying a single correct answer.

Authors:
Chia-Lee Yang, National Center for High-Performance Computing, Taiwan
Poning An, Yonsei University Linguistic and Informatic, South Korea
Hung-Hsun Chen, Fu Jen Catholic University, Taiwan
Yi Hao Hsiao, National Center for High-Performance Computing, Taiwan

About the Presenter(s)
Dr. Chia-Lee Yang is a Principal Engineer at Taiwan’s National Center for High-Performance Computing (NCHC).She has also led multiple gender-related research projects, advancing evidence-based approaches to gender equity in science and technology.

See this presentation on the full schedule – Monday Schedule

Bookmark