GENRES: From Individuals to Interactions

The question

What changes when characters interact?

A model can portray an individual fairly, yet distribute roles unequally when two people share a story.

Beyond isolated individuals

Most evaluations examine characters in isolation. GENRES places a male and a female character in the same social scenario, then asks a model to generate their profiles and a shared narrative.

Who takes action? Who is given higher status? Whose personality is developed? These relative portrayals reveal biases that individual descriptions can miss.

The preliminary single- versus dual-character comparison illustrates larger gaps in warmth-related descriptors and character depth in interpersonal settings.

Original paper figure comparing warmth-related words and stereoscopicity in single- and dual-character generation for Qwen-3B and Janus-7B. — S. = single-character; D. = dual-character. Original Figure 1. Click any figure to enlarge.

The paper

Abstract

Ensuring the trustworthiness of Multimodal Large Language Models (MLLMs) requires rigorous evaluation of gender bias, particularly in socially sensitive domains. Existing benchmarks primarily assess bias in isolated contexts and are less suited to capturing biases that emerge through interpersonal interactions. We introduce GENRES, a benchmark for evaluating gender bias in MLLMs through the lens of social relationships in narrative generation. Grounded in Fiske's relational models theory, GENRES comprises 1,440 multimodal narrative elicitation pairs spanning four fundamental relationship types, diverse social scenarios, and three age groups. Its dual-character generation tasks elicit socially situated interpersonal narratives and enable fine-grained assessment across multiple dimensions of gender bias. Experiments on six recent MLLMs reveal persistent yet strongly relationship- and model-dependent gender disparities, with some patterns becoming more pronounced in interpersonal settings. We further evaluate two complementary generation-stage mitigation strategies. Our findings demonstrate the value of relationship-aware evaluation for uncovering subtle, interaction-driven gender bias in MLLMs.

Benchmark design

Social relationships as an evaluation lens

Grounded in Fiske’s relational models theory, GENRES pairs visual scenes with prompts for socially situated narrative generation.

GENRES pipeline: narrative element design, image and prompt generation, model response collection, and multidimensional gender bias evaluation. — The GENRES development pipeline. Narrative elements become image–text pairs, which elicit character profiles and narratives for evaluation.

Design the context

288 narrative-element entries span four relationships and three age groups, with conflict and cooperation scenarios.

Curate image–text pairs

GPT-4o prompts and SDXL images undergo CLIP screening and independent human verification. Five qualified images per entry form 1,440 NEPs.

Elicit and evaluate

Models generate two profiles and a shared story. Lexicon analysis, grammatical analysis and LLM judges measure relative gender portrayals.

Explore the four relationships

One real narrative elicitation pair from each relationship in the released dataset.

Group unity and belonging

Communal Sharing

Relationships built on a shared group identity and a sense of belonging.

Characters
Age group

Scenario

Read the complete generation prompt

View source record on Hugging Face ↗

Evaluation framework

Four dimensions. Eight complementary metrics.

Gender disparities appear in different forms. GENRES measures each dimension separately rather than combining them into a single fairness score.

Profile Assignment Bias

Which personality traits does the model give each character?

M1 Warmth Ratio: The proportion of warmth words among matched warmth and competence descriptors.
M2 Positive Score: The number of positive personality descriptors minus negative descriptors.

Agency and Role Bias

Who acts, and who occupies the higher-status role?

M3 Subject Ratio: Character-specific subject counts from subject–verb–object analysis.
M4 Status Allocation: Higher-status assignments, measured on unequal-status cases.

Emotional Expression Bias

How are emotion and positive sentiment distributed?

M5 Emotion Words: Emotion-related word counts from the NRC Emotion Lexicon.
M6 Positive Sentence: Positive-sentence ratios using a fine-tuned DistilBERT classifier.

Narrative Framing Bias

Who receives psychological depth and drives the narrative?

M7 Stereoscopicity: The presence of both positive and negative personality traits.
M8 Main Character: The character who addresses the conflict or leads cooperation.

Reading the scores. M1–M6 are male–female contrasts. M7 and M8 are Pearson correlations with gender (male = 1), and M8 includes only narratives with an identifiable protagonist. Positive values indicate male-favoring bias; negative values indicate female-favoring bias. Units differ across metrics.

Experimental findings

Bias depends on the model and the relationship

Six open- and closed-source MLLMs show persistent disparities. Greater model capability does not consistently imply smaller gender gaps.

Signed gender bias metrics across six MLLMs
Model	Profile assignment		Agency and role		Emotional expression		Narrative framing
Model	Warmth Ratio M1, %	Positive Score M2	Subject Ratio M3	Status Allocation M4, %	Emotion Words M5	Positive Sentence M6, %	Stereoscopicity M7 (p-value)	Main Character M8 (p-value)
Qwen-3B	-6.575	0.012	0.097	2.952	-3.172	-1.272	0.014(0.134)	-0.102(0.000)
Phi4-5.6B	-7.277	0.019	-0.312	-2.632	-3.030	-0.010	0.007(0.433)	-0.131(0.000)
Qwen-7B	-8.281	-0.011	0.092	0.905	-3.010	-2.876	0.010(0.286)	-0.126(0.000)
Janus-7B	-7.869	-0.017	-0.092	-1.491	-0.453	-0.012	0.015(0.109)	-0.133(0.000)
Gemini	2.861	0.007	0.968	-3.917	4.567	-4.196	0.008(0.369)	-0.163(0.000)
GPT-4o	-3.864	0.001	0.025	-13.066	-1.547	-4.088	0.023(0.012)	-0.257(0.000)

+ Male-favoring− Female-favoring Parentheses report p-values for M7/M8, displayed at the paper’s precision. M1–M8 have different units; compare models within a metric.

Status allocation · M4

Models introduce hierarchies even in role-neutral contexts

Smaller models more often assign higher status to men, while GPT-4o and Gemini favor women. In Communal Sharing and Market Pricing, models introduce unequal status even though the relationships do not specify a hierarchy.

The figure retains samples with at least two-thirds agreement among evaluators. Gray marks equal-status assignments.

Status Allocation proportions for six models across Authority Ranking, Market Pricing, Equality Matching and Communal Sharing, separated into male, equal and female status assignments. — Original Figure 4. High-status assignment proportions by gender.

Narrative framing · M7 and M8

Being the protagonist does not guarantee a richer portrayal

Female characters more often drive the narrative across all six models. Meanwhile, Qwen-7B and GPT-4o more often give male characters both positive and negative traits, especially in Equality Matching and Market Pricing.

Mistral-Small-24B evaluator. Larger bubbles correspond to lower p-values; positive correlations indicate male-favoring bias.

Pearson correlations between gender and Stereoscopicity on the left, and Main Character assignment on the right, across four relationships. — Original Figure 5. Narrative centrality and psychological depth can diverge.

Mitigation

Partial improvements, with room to go further

Fairness-Aware Prompting adds explicit generation-time fairness instructions. The Critique-and-Revision Framework uses iterative feedback to revise narratives. Gains depend on the model and metric; agency and emotional expression remain difficult to control through prompting alone.

Read the full evaluation ↗

Start exploring

The GENRES dataset

1,440 image–text pairs are available on Hugging Face, together with the original query files and narrative-element specifications.

Split: test · 1,440 examples
Language: English
License: CC BY-NC 4.0

Explore on Hugging Face

Python · Hugging Face Datasets

from datasets import load_dataset

ds = load_dataset(
    "Savannah-y7/Genres",
    split="test"
)

sample = ds[0]
image = sample["image"]
prompt = sample["text_prompt"]

Each pair includes an image, prompt, age group, relationship, scenario and character names.

Scope and limitations

GENRES uses paired male and female characters for controlled directional comparisons. It does not represent non-binary and gender-diverse identities. Names and scenarios primarily reflect Western/U.S. contexts. Results should be interpreted within this scope, and the benchmark should support broader analysis rather than serve as a standalone measure of fairness.

Use in your research

Citation

If GENRES supports your research, please cite our paper.

BibTeX · COLM 2026

@inproceedings{xu2026genres,
  title = {From Individuals to Interactions: Benchmarking Gender Bias in
           Multimodal Large Language Models Through the lens of Social Relationship},
  author = {Xu, Yue and Wang, Wenjie},
  booktitle = {Conference on Language Modeling},
  year = {2026},
  url = {https://arxiv.org/abs/2506.23101}
}

The Paper button links to the COLM 2026 camera-ready version. The arXiv link points to the public preprint.