Culture Affordance Atlas: Reconciling Object Diversity Through Functional Mapping

AI summary

Overview

Research area: AI safety and ethics, specifically cultural and socioeconomic representation bias in vision-language (VL) datasets and models (arXiv category cs.CY), spanning dataset construction, computer vision, and NLP.

Technical level: Intermediate. The paper's central framework is conceptual and accessible, while the empirical evaluation assumes familiarity with contrastive vision-language models (CLIP), embedding similarity, and recall-based retrieval metrics.

Scope: The paper proposes a function-centric annotation framework and applies it to the Dollar Street image collection to build the Culture Affordance Atlas, then tests whether function-based labels reduce income-related performance disparities in VL models.

What This Paper Is About

Mainstream vision-language datasets are object-centric and skewed toward objects common in higher-income, Western settings, so culturally distinct objects and object uses from lower-income or non-Western regions are often missing, mislabeled, or collapsed under a single Western label (the paper's example: "clay pot," "cooler box," and "refrigerator" all filed under "refrigerator" in Dollar Street). The authors argue that object-centric labeling alone cannot fix this, and instead ask what an object is used for rather than what it is called, organizing objects by shared human functions. Their goal is to build a culturally grounded knowledge base of function-to-object mappings and to test empirically whether function-centric labels narrow the performance gap between high- and low-income image contexts.

Key Contributions

1. A function-centric framework for constructing culturally diverse VL datasets, organizing objects by universal human functions alongside contextual usage rather than by appearance or Western taxonomies.
2. The Culture Affordance Atlas, a publicly available knowledge base created by re-annotating the Dollar Street dataset into culturally grounded function categories, comprising 367 object-function entries with representative images and metadata, each backed by at least one ethnographic citation. The initial edition spans 7 categories, 46 functions, and 288 unique objects.
3. Empirical evidence that function-centric labeling reduces socioeconomic performance disparities in VL models. Using CLIP, the authors report that function-based labels reduce the gap between high- and low-income groups by a median of 6 percentage points, which is statistically significant, and confirm the direction of the effect on the siglip2-so400m-patch14-384 model.
4. Identification of culturally relevant long-tail objects that are absent or under-represented in major VL datasets, revealing representational gaps that current taxonomies do not capture.

Main Findings

• Function labels flatten the income-related performance gap in CLIP. CLIP alignment scores generally increase with income, indicating a gap favoring higher-income contexts, but function-based scores produce a flatter trajectory than topic-based scores, with linear regression slopes of 0.002 for functions versus 0.004 for topics.
• The per-topic gap reduction is statistically significant. Across all 270 topics, the difference in recall between high-income (rich and up-mid) and low-income (poor and low-mid) image sets was computed under topic-only and function-based prompts. A Wilcoxon signed-rank test shows the gap reduction is significantly greater than zero, with a median of 0.06 (6 percentage points) and p = 1.62e-17.
• Recall varies less across income quartiles with functions, but overall recall is lower. Function-based retrieval shows less variation across the poor, low-mid, upper-mid, and rich quartiles than topic-based retrieval, yet overall recall scores for functions are lower than for topics, indicating a trade-off.
• Combining topics with functions eliminates the income slope. Merging a topic with its function into a single prompt (for example, "A dishwasher that cleans dishes and utensils") produced a near-zero slope of -0.0002 in CLIP association tests, with notably larger gains for lower-income bins and markedly improved recall for lower-income quartiles, alongside slight reductions in higher-income quartiles.
• Misalignment between Dollar Street topics and actual objects is common. The re-annotation pipeline identified mismatches between original topic labels and identified objects, averaging 38.25% misalignment across topics.
• Long-tail objects are largely missing from popular VL datasets. Using the 100 least frequent items from the Atlas, the authors report coverage in official class lists: OpenImages 61% (61 of 100, out of 19,868 classes), LVIS 28% (out of 1,230 classes), ImageNet 8% (out of 1,000 classes), Dollar Street 5% (out of 291 classes), COCO 2% (out of 80 classes), YFCC100M Entity-Artifacts 1% (out of 325 classes), and GeoDE 0% (out of 40 classes).
• 34 objects are universally absent, and 33 appear in only one dataset. The paper states that 34% of the curated long-tail objects are entirely absent from the seven widely used VL datasets examined.
• Objects often serve multiple functions. For instance, "bowls" appear in contexts including clothes washing, hand washing, food serving, and food preparation; "charcoal" is primarily used as fuel but also serves dental hygiene functions. The authors call such items "contextually niche," with dominant uses alongside lesser-known but culturally significant secondary functions.
• Generalization to a second model holds. Repeating the experiments on siglip2-so400m-patch14-384, function prompts significantly reduce the high–low income performance gap by 11% (p = 5.9e-11, Wilcoxon signed-rank test). The Lessons Learned section describes the same result as 11 percentage points in SigLIP2.
• Quality control on functional descriptions passed. An assessment with 21 participants across 7 countries (China, India, Ethiopia, Nigeria, United States, Romania, and Russia), with 30 questions per country and three native participants judging each, produced an overall validation score of 90%, with every country above 85% (China 87.78, India 90.00, Ethiopia 91.11, Nigeria 86.67, United States 87.78, Romania 96.67, Russia 90.00).
• Function prompts introduce some false positives. Function prompts can retrieve contextually incorrect objects; for example, retrieving dishware and cutlery instead of "objects that clean dishes," or retrieving images related to general toilet cleaning while missing the intended personal-hygiene context. The authors attribute this partly to CLIP emphasizing nouns independently rather than interpreting full sentence contexts.

Methodology in Plain English

The researchers started from the Dollar Street dataset, which contains 38,479 images from 63 countries covering everyday household objects and activities, organized into 291 topics, with household incomes ranging from $26.9 to $19,671.0 per month adjusted for purchasing power parity. Following prior work, they removed 21 subjective topics, leaving 270 objective topics for analysis.

For every one of the 270 topic labels, they used GPT-4o to generate a short functional description—what the object does—and then stripped the topic label out of the description, leaving a function such as "object that provides a place to sleep." Separately, they used LLaVA to caption each image, then GPT-4 to extract the name(s) of the object in the caption that matched the functional description, producing an identified object label such as "straw mat" where the original topic was "bed." They manually re-annotated 1,458 identified object names, including entries previously listed as NaN, None, or Not identifiable, and ran a cross-country validation study of the functional descriptions.

To build the Atlas, they mapped functional descriptions and object names into seven function categories drawn from Brown's anthropological list of human universals—practicing hygiene, beautifying, cooking, maintaining a diurnal cycle, adjusting to the environment, healing, and traveling—and grounded each entry with at least one published source (approximately 98% from the eHRAF World Cultures database and 2% from Google Scholar).

For evaluation, they ran two experiments with CLIP. The CLIP association test computed cosine similarities between image embeddings and text embeddings for topics versus functions, with images grouped into geometric income bins. The retrieval experiment ranked images by CLIP score against each topic and each function caption and measured Recall by retrieving a top-N set where N equals the number of ground-truth images. Statistical significance was assessed with the Wilcoxon signed-rank test. Long-tail analysis compared 100 infrequent Atlas objects against the official class lists of seven VL datasets after normalization (lowercasing, lemmatization, fuzzy matching with a threshold of 0.8).

Why This Matters

Impact on research. The paper reframes dataset bias mitigation as a labeling-schema problem rather than only a data-collection problem. Its finding that 38.25% of Dollar Street topics show label misalignment, and that 34% of curated long-tail objects are absent from seven prominent VL datasets, quantifies gaps that prior work described mostly qualitatively. It also supplies a reusable artifact—the Atlas—that other researchers can both evaluate against and extend, and it shows that a simple prompt change (function versus topic) can measurably shift equity outcomes across income groups without retraining.

Real-world applications:

• Content moderation, search, and recommendation systems that serve users worldwide can misfire when culturally specific household items are unrecognized or mislabeled, leading to incorrect filters and poor recommendations.
• Assistive and accessibility technologies, and any consumer product that relies on object recognition in the home (for example, smart appliances or home robotics), benefit from recognizing objects by function rather than by a single Western canonical form.
• International development and humanitarian organizations can use function-based taxonomies to inventory household assets across regions without imposing a preset list of object names.
• E-commerce and logistics cataloging can reduce the mismatch between local product forms and generic category tags, improving discoverability for regionally specific goods.

Industry relevance. Any organization that builds or fine-tunes multimodal models on crawled web data inherits the biases documented here. The paper's demonstration that function prompts improve outcomes for low-income contexts—and that combining topic and function prompts removes the income-related slope in CLIP association tests—offers a low-cost intervention for teams curating training and evaluation data. The identified trade-off (better equity, lower overall retrieval accuracy with function-only prompts) is directly relevant to product teams choosing between recall and fairness trade-offs.

Future Directions

• Deepen the ethnographic grounding. Most Atlas entries (89%) include one reference, the first reliably identified one, while 11% contain multiple sources. The authors position the Atlas as a foundational layer to be expanded through additional eHRAF searches, museum and archival materials, and community contributions.
• Broaden the validation base. The quality control study covered 21 participants across seven countries; the authors state that broader participation would further strengthen robustness. Extending validation to more countries and languages is a natural next step.
• Resolve the precision-versus-inclusivity tension in prompts. Function prompts reduce disparities but introduce false positives, sometimes because CLIP emphasizes nouns independently of full sentence context. Designing prompts that are precise yet sufficiently inclusive—for example, preferring "object that cleans dishes" over "machine that cleans dishes"—remains an open problem.
• Extend beyond the tested setup. The experiments focus on Dollar Street images with CLIP and siglip2-so400m-patch14-384, using CLIP association and Recall metrics. Whether the function-centric framework generalizes to other datasets, tasks such as captioning or detection, and other modalities is not reported in this paper.

Target Audience

This paper is most useful to researchers and practitioners working on dataset curation, multimodal model evaluation, and AI fairness, particularly those concerned with geographic and socioeconomic representation. It also speaks to anthropologists and social scientists interested in how ethnographic concepts such as affordances and human universals can be operationalized for machine learning, and to product and policy teams at organizations deploying vision-language systems across global markets who need concrete evidence that labeling choices affect equity outcomes.

Authors’ abstract

Culture shapes the objects people use and for what purposes, yet mainstream Vision-Language (VL) datasets frequently exhibit cultural biases, disproportionately favoring higher-income, Western contexts. This imbalance reduces model generalizability and perpetuates performance disparities, especially impacting lower-income and non-Western communities. To address these disparities, we propose a novel function-centric framework that categorizes objects by the functions they fulfill, across diverse cultural and economic contexts. We implement this framework by creating the Culture Affordance Atlas, a re-annotated and culturally grounded restructuring of the Dollar Street dataset spanning 46 functions and 288 objects publicly available at https://lit.eecs.umich.edu/CultureAffordance-Atlas/index.html. Through extensive empirical analyses using the CLIP model, we demonstrate that function-centric labels substantially reduce socioeconomic performance gaps between high- and low-income groups by a median of 6 pp (statistically significant), improving model effectiveness for lower-income contexts. Furthermore, our analyses reveals numerous culturally essential objects that are frequently overlooked in prominent VL datasets. Our contributions offer a scalable pathway toward building inclusive VL datasets and equitable AI systems.