GeoDiv: Framework For Measuring Geographical Diversity In Text-To-Image Models

AI summary

Overview

Research area: Computer vision / generative model evaluation, specifically auditing geographic and socio-economic representation in text-to-image (T2I) diffusion models using large language models (LLMs) and vision-language models (VLMs).

Technical level: Intermediate. The paper assumes familiarity with text-to-image diffusion models, entropy-based diversity metrics, and LLM/VLM prompting pipelines, but the core ideas are explained in accessible terms.

Scope: The paper introduces GeoDiv, an interpretable, reference-free framework that scores geographic diversity of generated images along four dimensions, and applies it to 160,000 images from four T2I models across 10 entities and 16 countries.

What This Paper Is About

Text-to-image models are trained on internet-scale data and inherit the biases of that data, so prompts like "a photo of a house in Nigeria" often produce images that misrepresent regions or collapse into stereotypes. Existing diversity metrics either require curated reference datasets or only measure low-level visual similarity, and neither can tell you what about the depiction is biased. The authors build GeoDiv, a framework that decomposes geographic diversity into separate, human-interpretable axes and quantifies each one using the world knowledge of LLMs and VLMs, then use it to audit four open-source diffusion models.

Key Contributions

1. GeoDiv, an interpretable two-axis evaluation framework. It measures geographic diversity through the Socio-Economic Visual Index (SEVI)—covering Affluence and Maintenance on a 1–5 scale—and the Visual Diversity Index (VDI)—covering Entity-Appearance and Background-Appearance. Diversity is quantified per question with a Normalized Hill Number (the exponential of Shannon entropy, rescaled to range 0 to 1).

2. Released structured attribute-value sets for 10 entities. LLMs generate entity-specific question-answer sets (consolidated by an aggregator LLM), covering both SEVI and VDI. The paper also provides the full prompts and filtering mechanisms needed to extend the framework to new entities.

3. A 160,000-image synthetic dataset plus human annotations. Images were generated with four open-source diffusion models across 16 countries and 10 entities (250 images per entity-country pair, 40,000 per model). For a subset, the authors collected VDI attribute annotations and country-level SEVI ratings from crowdworkers, then used these to evaluate multiple LLM-VLM combinations.

4. A diagnostic audit revealing regional biases. GeoDiv surfaces specific, fine-grained defaults in current generative models and is compared against existing diversity measurement baselines, with diversity scores released for all GeoDiv dimensions.

Main Findings

• Socio-economic stereotyping by country. India, Nigeria, and Colombia are consistently portrayed as the poorest (average Affluence 2.31, Maintenance 3.34), while Japan, UAE, and UK appear as the wealthiest (Affluence 3.53, Maintenance 4.30). Per-country mean SEVI diversity scores are low: 0.36 (Affluence) and 0.38 (Maintenance).

• No model covers the socio-economic spectrum. The average Affluence Diversity-Score is similar across T2I models (0.35 ± 0.01), and average Maintenance Diversity-Score is 0.34 ± 0.12. FLUX.1 shows a severe lack of variation in physical condition, scoring only 0.15 on Maintenance diversity.

• FLUX.1 trades polish for diversity. FLUX.1 achieves the highest mean ratings—Affluence 3.82 and Maintenance 4.73 on the 1–5 scale—but the lowest overall geo-diversity, while SD2.1 scores lowest on mean ratings (Affluence 2.41, Maintenance 3.23) but highest on diversity. Affluence and Maintenance correlate moderately overall (ρ = 0.5), yet this pattern reverses for some entities.

• Entity diversity beats background diversity, and both are weak. Entity-Appearance Diversity-Scores are 0.51 (SD2.1), 0.45 (SD3m), 0.42 (FLUX.1), and 0.40 (SD3v.5); the mean across countries is 0.47. Background-Appearance diversity is lower still, averaging 0.31 across datasets and 0.33 across countries.

• Concrete country-specific attribute defaults. SD3.5 generates 99% Egyptian houses made of stone and 88% UK houses made of bricks. Across models, 77% of car images from Nigeria show dirt/gravel roads, compared to the US, which generates paved roads 85% of the time. 57% of Indian images show dense buildings in the background, versus 17% for the UAE.

• Underrepresented natural and social context. Mountains and hills appear in only 12% of images on average—least in Nigeria (1.1%), most in Turkey (24%). Waterbodies appear in only 3.4%. Backgrounds are mostly quiet and empty regardless of country or model.

• Country-level ranking. Egypt, Colombia, Turkey, and Spain are the most geo-diverse countries, while Japan, the UK, the US, and India rank lowest. The mean GeoDiv score per country is 0.39. There is a weak negative correlation between GeoDiv scores and GDP nominal (ρ = -0.27) and GDP per capita (-0.28).

• Newer model versions tend to be less geo-diverse. Averaged across SEVI and VDI, FLUX.1 shows the lowest scores and SD2.1 ranks highest, consistent with prior findings.

• GeoDiv exceeds Vendi-Score in interpretability. Correlations with Vendi-Score are high only for Entity Diversity (Pearson's ρ = 0.56) and low elsewhere (ρ = 0.06 for Maintenance), showing Vendi-Score misses multidimensional geo-diversity.

• Real-world data is still more diverse. Evaluated with GeoDiv, the real-world GeoDE dataset reaches Entity-Appearance diversity of 0.60 versus 0.44 for synthetic images, Background-Appearance of 0.42 versus 0.31, and Maintenance diversity of 0.61.

Methodology in Plain English

The authors start by defining what geographic diversity should mean, breaking it into four questions a human could answer about an image: how affluent does the scene look, how well maintained are the objects, how much do the main entities vary, and how much do the backgrounds vary.

To make this automatic, they generate natural-language questions and answer choices using LLMs (an ensemble of five LLMs for entity attributes, consolidated by an aggregator LLM, with a fixed set of background questions). A vision-language model then looks at each image and picks answers from those choices. Because the answers selected across a set of images form a distribution, the authors measure how evenly spread that distribution is using the Hill Number—effectively the "effective number of answers" present—normalized to a 0-to-1 range so questions with different numbers of options are comparable.

For the socio-economic side, the VLM directly rates each image on the two 1–5 scales. The authors deliberately treat the full spectrum of conditions, not just affluence, as the ideal, so that diversity rather than polish is rewarded.

They add three controls against VLM hallucination: a visibility step that drops images where the asked-about attribute cannot be seen, multi-select responses so the model is not forced into one choice, and a "None Of The Above" option (chosen for only 2.6% of image-question pairs). They then validate the whole pipeline against human annotators, and finally apply it to 160,000 generated images and to the real-world GeoDE dataset as a comparison point.

Why This Matters

GeoDiv shifts bias auditing from "are these images different?" to "how exactly are these images wrong, and for whom?" By decomposing geographic diversity into named, human-readable dimensions, it lets developers point to specific failures—say, a model that only produces paved roads for the US—rather than reporting a single opaque number. The authors release their diversity scores, annotations, and codebase so others can benchmark against them.

Real-world applications:

• Auditing T2I models before deployment in consumer or enterprise products, to document geographic representation gaps.
• Guiding dataset curation and fine-tuning by flagging which attributes are systematically missing for which entity-country pairs.
• Evaluating synthetic training data used to augment image datasets, where unmeasured bias propagates into downstream models.
• Informing policy and content-integrity work by quantifying how generative tools portray different countries and economic conditions.

Industry relevance: Generative image tools are deployed globally, and their portrayal of countries shapes how users perceive places and people. The paper reports detailed cost figures for running the pipeline—approximately $58.64 per entity-country-question combination for the VDI component and $9.46 for SEVI, totaling $68.10 per complete assessment across 250 images—making the trade-off between audit depth and expense concrete for teams deciding whether to adopt it.

Future Directions

• Broadening coverage. The study covers 16 countries and 10 entities; the authors note that extending to more regions and entity types may reveal additional patterns and biases, and release prompts and distributions to make this possible.

• Measuring cultural representation. The paper describes a Cultural Localization score computed through the VQA pipeline, with higher disagreement between VLM and human annotators for the USA and UK and better alignment for Japan and Colombia. This axis is left for fuller treatment.

• Reducing dependence on closed-source models. GeoDiv relies on Gemini-2.5-Flash, and budget constraints limit large-scale evaluation. The open-source Qwen2.5-VL is shown to be a practical alternative, agreeing with Gemini at an average correlation of ρ = 0.83 across two datasets and six entities.

• Closing the gap to real data. Because GeoDE consistently outperforms synthetic datasets on Entity-Appearance and Maintenance diversity, a natural question is what training or sampling changes would move generative models toward the crowd-collected diversity of real-world imagery.

Target Audience

This paper is most useful to researchers and practitioners working on fairness, bias, and evaluation of generative models; dataset curators and teams fine-tuning diffusion models for global audiences; and computer vision researchers interested in using LLMs and VLMs as automated evaluators. It is also relevant to those studying socio-economic representation in AI systems, since the SEVI scale offers a concrete, human-validated instrument for measuring wealth and physical-condition cues that earlier diversity metrics do not capture.

Authors’ abstract

Text-to-image (T2I) models are rapidly gaining popularity, yet their outputs often lack geographical diversity, reinforce stereotypes, and misrepresent regions. Given their broad reach, it is critical to rigorously evaluate how these models portray the world. Existing diversity metrics either rely on curated datasets or focus on surface-level visual similarity, limiting interpretability. We introduce GeoDiv, a framework leveraging large language and vision-language models to assess geographical diversity along two complementary axes: the Socio-Economic Visual Index (SEVI), capturing economic and condition-related cues, and the Visual Diversity Index (VDI), measuring variation in primary entities and backgrounds. Applied to images generated by models such as Stable Diffusion and FLUX.1-dev across $10$ entities and $16$ countries, GeoDiv reveals a consistent lack of diversity and identifies fine-grained attributes where models default to biased portrayals. Strikingly, depictions of countries like India, Nigeria, and Colombia are disproportionately impoverished and worn, reflecting underlying socio-economic biases. These results highlight the need for greater geographical nuance in generative models. GeoDiv provides the first systematic, interpretable framework for measuring such biases, marking a step toward fairer and more inclusive generative systems. Project page: https://abhipsabasu.github.io/geodiv