Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content?

AI summary

Overview

Research area: AI safety, legal compliance, and intellectual property governance for large vision-language models (LVLMs), sitting at the intersection of multimodal machine learning, retrieval-augmented generation, and copyright law.

Technical level: Intermediate. Readers should be comfortable with LVLM architectures, standard evaluation metrics (ROUGE, BERTScore, embedding similarity), and the basics of retrieval-augmented pipelines. No deep legal expertise is required.

Scope: The paper builds the first large-scale benchmark for measuring whether vision-language models recognize and respect copyrighted content presented as images, then proposes a tool-augmented guardrail called CopyGuard to reduce infringement.

What This Paper Is About

Vision-language models are increasingly deployed in multimodal RAG systems, web agents, and search engines, where they ingest images that often contain copyrighted text such as book pages, news articles, song lyrics, or API documentation. Prior work studied this problem for text-only LLMs, but no systematic benchmark existed for models that must read copyright cues from pixels. The paper asks three questions: how well LVLMs comply with copyright when copyrighted material appears in a multimodal context, whether explicit copyright notices change that behavior, and how compliance can be improved without breaking legitimate use.

Key Contributions

1. The first multimodal copyright-compliance benchmark. A dataset of 50,000 query-content pairs spanning four copyrighted content types (book excerpts, news articles, music lyrics, code documentation) and four infringement task types (repetition, extraction, paraphrasing, translation), with matched versions that include or omit copyright notices.

2. A systematic evaluation of 12 LVLMs across three model families. The study covers API-based models (GPT-4o, GPT-4o-mini, Gemini-2.0, Claude-3.7), open-weight models (LLaVA-1.5 7B/13B, LLaVA-NeXT 13B, Qwen2.5-VL 3B/7B, DeepSeek-VL 7B, GLM-4v 9B), and a unified model (Janus-Pro 7B), measuring both content similarity to the source and refusal rates.

3. CopyGuard, a tool-augmented defense framework. A four-component pipeline that detects copyright notices via OCR, verifies copyright status through live web search, analyzes query-level infringement risk, and feeds a structured reminder back to the model before generation.

4. Evidence that the defense works without harming utility. CopyGuard raises refusal rates dramatically across all models and scenarios while producing zero refusals on non-copyrighted content and leaving general benchmark performance (MMMU, MMBench, MathVista, MM-Vet) intact.

Main Findings

• Most models fail at copyright compliance. Eleven of twelve evaluated LVLMs struggle to recognize or respect copyrighted content, even when an explicit notice is embedded in the image or appended as text. Gemini-2.0, Qwen2.5-VL (3B and 7B), GLM-4v 9B, and Janus-Pro 7B show refusal rates below 2 percent on repetition tasks and ROUGE-L scores above 90 percent, indicating near-verbatim reproduction.

• GPT-4o is the outlier, not the norm. It achieves the highest refusal rates (about 91 percent on repetition, over 95 percent on translation) and the lowest similarity scores, making it the only model in the set that meaningfully resists redistribution requests.

• Task type strongly affects compliance. Repetition is the task models handle best, with the lowest reproduction scores and highest refusal rates. Extraction and paraphrasing are the weakest areas: most models refuse fewer than 12 percent of extraction requests, and paraphrasing refusal stays below 30 percent even with a notice present. This reflects a tendency to treat "selecting" or "rephrasing" as benign rather than as derivative reproduction.

• Detailed notices outperform generic ones. Notices naming the specific work produce lower ROUGE-L scores and higher refusal rates than a uniform "All Rights Reserved" statement, suggesting some models have an emergent sensitivity to richer copyright language. However, no single notice format works universally, and three models (Gemini-2.0, LLaVA-1.5 7B, LLaVA-1.5 13B) show near-identical score distributions regardless of notice type, indicating they have not learned to prioritize notices at all.

• Notice modality matters inconsistently. GPT-4o and Claude-3.7 respond more strongly to notices presented as text, while Qwen2.5-VL, DeepSeek-VL 7B, and Janus-Pro 7B respond more strongly to notices embedded in the image. This points to a broader weakness in transferring information between the visual and textual modalities.

• Architecture beats scale. Models of the same parameter count (Qwen2.5-VL 7B, DeepSeek-VL 7B, Janus-Pro 7B) behave very differently, and API-based models generally outperform open-weight ones. Scaling from 3B to 7B in the Qwen family improves instruction-following around notices but does not produce robust intrinsic copyright understanding. LLaVA-NeXT 13B benefits from notices more than LLaVA-1.5 13B, yet its baseline awareness without a notice is no better than its predecessor.

• Fine-tuning alone causes over-refusal. When models are tuned for copyright awareness, they tend to reject legitimate queries about copyrighted material as well, which is why the authors pursue an external guardrail rather than a training-based fix.

• CopyGuard substantially closes the gap. Refusal rates reach 100 percent in several model-task combinations (including GPT-4o, GPT-4o-mini, Gemini-2.0, Claude-3.7, and LLaVA-1.5 7B on repetition), with all models improving to at least 82 percent on that task. Gains hold across extraction, paraphrasing, and translation, and across every notice condition.

• Fair use is preserved. CopyGuard records a zero refusal rate on transformative queries such as summarizing, commenting, asking about the author, or requesting background information, and it does not degrade performance on standard multimodal reasoning benchmarks.

• Metrics are validated. A user study of 300 randomly sampled query-content pairs found GPT-4's refusal judgments agree with human annotation 99 percent of the time, and refusal rate shows strong negative correlation with reproduction metrics, particularly for extraction and translation tasks.

Methodology in Plain English

The researchers assembled real copyrighted material from four sources: best-selling books (via Goodreads), news articles (BBC and CNN), top music tracks (Spotify), and code documentation (Hugging Face Docs and PyPI). Content was deliberately spread across publication dates, genres, and popularity levels to avoid overlap with what the models may have memorized during training.

For each piece of content, they produced two versions: one with no copyright notice, presented as if public domain, and one carrying a notice. Notices varied along two axes, giving four types: either a work-specific "Original" notice or a generic "All Rights Reserved" notice, and either embedded inside the image or supplied as text alongside it.

Queries were written by three experienced LVLM users for four infringement categories and then paraphrased by GPT-4 to produce varied wording while preserving intent. This yields roughly 50,000 query-content pairs.

Evaluation used two complementary signals. For reproduction, the paper measures how much of the original text reappears in the response: ROUGE-L for extraction and repetition, BERTScore for paraphrasing, and multilingual embedding similarity for translation. Higher similarity means more infringement. For behavior, an LLM judge scores whether the model refused the request, with human validation confirming near-perfect agreement.

CopyGuard works as a pre-generation guardrail with four parts. A notice identifier runs OCR over the input image and scans for copyright markers. If no notice appears, a status verifier extracts text, searches for its source through the Serper Google Search API, and asks an internet-connected reasoning model to confirm whether the work is still under protection. A query risk analyzer evaluates whether the user's request would involve redistribution and proposes safer alternatives. If both conditions trigger, a reminder informs the model of the specific risk before it generates anything. The search and query-analysis steps run in parallel, so the added latency is the maximum of the two rather than their sum, and no extra cost is incurred when no risk is found.

Why This Matters

Research impact. This is the first systematic treatment of copyright compliance as a multimodal problem rather than a text-only one. It establishes measurement infrastructure, shows that visual copyright cues are processed inconsistently (a finding relevant to broader work on cross-modal transfer in LVLMs), and demonstrates that external guardrails can outperform fine-tuning for this class of policy.

Real-world applications.

• Multimodal RAG systems that retrieve scanned pages, PDFs, or screenshots and feed them to a model before answering user questions.
• AI web agents and browser assistants that read rendered pages, including images of paywalled articles or e-books, and summarize or extract from them.
• Enterprise document assistants processing internal or licensed materials that carry third-party copyright.
• Search engines and copilots that surface snippets, lyric panels, or code documentation alongside generative answers.

Industry relevance. Copyright litigation against generative AI developers is ongoing and expensive, and the paper shows that disclaimers and notices alone do not produce reliable compliance. The distinction the paper draws between redistributive uses (repetition, extraction, paraphrase, translation) and transformative ones (summarizing, commenting) maps closely onto how platforms already frame fair use, giving engineering teams a concrete policy boundary to implement. CopyGuard's low false-positive rate matters commercially: a guardrail that blocks legitimate summarization would be unusable in production.

Future Directions

• Reducing dependence on external services. Both the source identification and the copyright status verification rely on live web search and an internet-connected model. Cached results can become stale as works enter the public domain or change ownership, so more robust real-time verification mechanisms are needed.

• Internalizing copyright awareness during training. The paper shows that fine-tuning for copyright currently causes over-refusal. Closing that gap, so models judge copyright status internally without rejecting legitimate tasks, remains unresolved.

• Extending to other media. The benchmark covers text rendered in images. Whether the same failures appear for audio, video, or complex layouts such as sheet music, diagrams embedded in textbooks, or multi-page spreads is untested.

• Cross-jurisdictional legal grounding. The study adopts a U.S.-centered standard for what counts as infringing and avoids strict legal interpretation. Adapting the framework to the Berne Convention, EU quotation exceptions, and national fair-use variants would make it more broadly deployable.

• Cost and latency under scale. CopyGuard adds OCR, search, and an additional model invocation. Characterizing overhead at production query volumes, and reducing it, is an open engineering question.

Target Audience

AI safety and alignment researchers working on policy compliance and guardrail design; multimodal ML engineers building RAG systems, web agents, or search products that ingest real-world documents; legal and policy teams at AI companies assessing copyright exposure; and benchmark designers interested in how to construct multimodal evaluation sets with carefully controlled metadata. Graduate students entering the copyright-and-generative-AI space will find the framing and metrics especially useful as a starting point.

Authors’ abstract

Large vision-language models (LVLMs) have achieved remarkable advancements in multimodal reasoning tasks. However, their widespread accessibility raises critical concerns about potential copyright infringement. Will LVLMs accurately recognize and comply with copyright regulations when encountering copyrighted content (i.e., user input, retrieved documents) in the context? Failure to comply with copyright regulations may lead to serious legal and ethical consequences, particularly when LVLMs generate responses based on copyrighted materials (e.g., retrieved book experts, news reports). In this paper, we present a comprehensive evaluation of various LVLMs, examining how they handle copyrighted content -- such as book excerpts, news articles, music lyrics, and code documentation when they are presented as visual inputs. To systematically measure copyright compliance, we introduce a large-scale benchmark dataset comprising 50,000 multimodal query-content pairs designed to evaluate how effectively LVLMs handle queries that could lead to copyright infringement. Given that real-world copyrighted content may or may not include a copyright notice, the dataset includes query-content pairs in two distinct scenarios: with and without a copyright notice. For the former, we extensively cover four types of copyright notices to account for different cases. Our evaluation reveals that even state-of-the-art closed-source LVLMs exhibit significant deficiencies in recognizing and respecting the copyrighted content, even when presented with the copyright notice. To solve this limitation, we introduce a novel tool-augmented defense framework for copyright compliance, which reduces infringement risks in all scenarios. Our findings underscore the importance of developing copyright-aware LVLMs to ensure the responsible and lawful use of copyrighted content.