PhotoGuider: Composition-Aware Cropping and 360$^{\circ}$ View Framing with Explainable Photographic Assessments

AI summary

Overview

Research area: Computer Vision and Computational Photography — automatic image cropping, composition analysis, 360° panorama view selection, and explainable aesthetic assessment.

Technical level: Advanced. The method combines foundation vision models (DINOv3, SAM, InSPyReNet, CLIP, LAION aesthetic and FGAesQ predictors), gradient-boosted tree rankers (LightGBM) with TreeSHAP attribution, a neural additive view scorer, and a fine-tuned multimodal large language model (MLLM).

Scope (one sentence): The paper presents PhotoGuider, a single pipeline that measures four interpretable "photographic tokens" over shared features and then uses them to pick a viewing direction in a 360° panorama, refine a crop, output camera parameters for native re-rendering, and generate explanations grounded in those measurements.

What This Paper Is About

Conventional image cropping assumes the input photograph already contains a composition worth keeping, so the only question is where to place the crop box. A full 360° scene provides no such guarantee: before anything can be framed, the system must first discover which direction in the scene is worth looking at. PhotoGuider addresses both problems with one shared set of composition-aware measurements, then additionally explains its framing decisions using the same measured values.

Key Contributions

1. Composition-aware cropping with a compact, interpretable decision layer. Four photographic tokens — Subject (S), Composition (C), Integrity (I), and Aesthetics (A) — score candidate crops through learned compact rankers that hold only 63k parameters, while the MLLM is confined to a bounded "consultant" role that neither scores nor makes decisions.

2. Measurement-conditioned explanations. A fine-tuned MLLM is conditioned on token assessments and geometric measurements, so its measurement-based statements can be checked against the supplied values rather than being free-form rationales.

3. The first system (to the authors' knowledge) for automatic composition-aware framing from 360° panoramas that combines explicit view-level direction selection with camera-parameter output for re-rendering and measurement-conditioned explanations based on shared photographic feature groups.

4. A measure-then-frame architecture spanning both tasks. The same 60 measurements drive a LightGBM framing ranker and an additive direction scorer; the system selects a direction, reprojects the view, refines the crop, converts the box into pan, tilt, field of view, and aspect ratio, and re-renders at full resolution.

Main Findings

• Best average 2D cropping accuracy. On FCDB (348 test images) and FLMS (500 test images), PhotoGuider achieves the highest mean mIoU and lowest mean BDE averaged across the two benchmarks: PhotoGuider (full) reaches 0.056 BDE / 0.766 mIoU on FCDB and 0.033 / 0.856 on FLMS, for an average of 0.044 BDE / 0.811 mIoU. Fast mode averages 0.050 / 0.797. For comparison, CPIC averages 0.048 / 0.797, Venus 0.051 / 0.792, and Cropper 0.052 / 0.789. The same trained checkpoint is used on both benchmarks without dataset-specific retraining.

• Strong human preference in 360° framing. In a blind ranking study, 120 participants ranked three cropping methods over 100 panoramic scenes from the public Poly Haven HDRI library, producing 4,820 rankings. PhotoGuider placed first in 53.2% of cases versus 27.3% (Venus) and 19.5% (CACNet), with a mean rank of 1.62 versus 2.10 and 2.28, and won 59 of 100 scenes. Head-to-head, it was preferred over Venus in 65.2% and CACNet in 72.8% of comparisons.

• Subject and Composition are the most informative tokens. In the token ablation (fast mode), Subject and Composition provide the strongest individual feature groups, and combining all four tokens gives the best overall cropping. Integrity and Aesthetics receive lower ranker weights because candidate shortlisting drastically reduces their variance within pools: subject capture exceeds 0.95 for 64% of FCDB and all FLMS shortlist frames, varying within a pool at only 0.4 of its global spread (compared with 0.9 for rule-of-thirds alignment). These tokens instead act downstream, in the coverage and shrink gates and the aesthetic-delta gate.

• Fine-tuning reduces explanation contradictions. A rule-based detector checks every numeric and directional claim against the injected measurements. Fine-tuning lowered the detector-measured contradiction rate of Qwen2.5-VL-7B from 6.31% to 1.55%.

• Explanations preferred by human judges. In a double-blind three-way study with 50 participants, five test images (two from FCDB, three from FLMS), and 250 total votes, PhotoGuider's explanations captured 40.0%, ahead of Venus at 32.8% and ArtiMuse at 27.2%.

• Interpretability is built in. Token attributions come from TreeSHAP decomposition of the ranker output and are exact for the three uncalibrated heads carrying a combined weight of 0.88; the calibrated head (weight 0.12) uses a secant approximation. The view scorer is a token-grouped additive network whose output is the sum of four token-specific terms.

• Dataset bias is modest across benchmarks. Normalized by pooled standard deviation, the median-to-median displacement between crops selected on FCDB and FLMS is 0.34 SD in the S–A plane and 0.47 SD in the C–I plane.

Methodology in Plain English

The design mirrors what a photographer does on arriving at a scene: look around, settle on a direction, adjust the frame, press the shutter, and be able to explain the choice.

Measuring four things about any frame. Every candidate frame is described by 60 features grouped into four disjoint tokens. Subject (10D) asks what is being photographed, using DINOv3's self-supervised attention for focal strength, InSPyReNet for a salient-object mask, and SAM for area, placement, and agreement between the two. Composition (22D) asks how the subject is placed, combining hand-computed geometry (rule of thirds, symmetry, triangles, balance, negative space, aspect) with SAM object parsing and nine CLIP prompt similarities that turn phrases like "leading lines" into measurable numbers. Integrity (20D) asks what the frame cut off, using SAM encoder and decoder statistics about object count, area shares, boundary cleanliness, edge margins, and depth layering. Aesthetics (8D) asks whether it is pleasing, using the LAION and FGAesQ aesthetic predictors plus saturation and subject-to-background contrast.

Generating and shortlisting crops. About 300 candidate boxes come from eight sources (geometric grid, subject-mask anchors, saliency peaks, multi-scale refinement, and four composition-guided generators). A determinantal point process picks K = 10 framings that are individually strong and mutually distinct; the top three go to the main ranker.

Ranking. A LightGBM ranker learns how to weight the four questions, supervised by overlap (IoU) with human-favored framings. It blends three quality scores: two from a 17-dimensional core saliency-and-geometry set, and one from the full 60-dimensional set with isotonic calibration.

Refinement. Three gated spatial moves adjust the chosen frame: a recenter step that mixes a rule-of-thirds target with a CLIP-attention target according to a predicted IoU advantage; an expand step with four independent gates reading subject coverage, CLIP wide-versus-tight response, aesthetic-score change, and golden-section proximity; and a shrink step that fires if no expansion gate triggers, comparing shot scale via SAM embedding similarity.

Optional MLLM consultant. Venus proposes boxes but never scores or decides. Its outputs are aggregated by median over multiple decodes and paraphrased prompts (self-consistency), and each consultant-aligned candidate gets a "twin" candidate re-scored by the ranker; a switch happens only with a significant score margin.

Explaining. The explainer cites the same quantitative features the ranker used. Qwen2.5-VL-7B drafts explanations for training images, and four distillation filters admit only valid drafts: Silence (no geometric claims without valid subject masks), Consistency (no values conflicting with physical measurements), Evidence (must cite features beyond basic geometry), and Wording (no banned phrasing or wrong language). The student learns only from admitted drafts via QLoRA.

Extending to 360°. Perspective views are reprojected on an overlapping grid of pan and tilt angles and measured with the same 60 features. An additive scorer trained on the GAICD training set (whose mean opinion scores rate candidate frames) picks the best direction. That direction is reprojected into a wider working view, run through the cropping pipeline, and the resulting box is converted into camera parameters (pan, tilt, field of view, aspect ratio) for native re-rendering at full resolution.

Why This Matters

Impact on research. The paper reframes cropping as a measure-then-frame problem rather than an end-to-end regression problem, and shows that a very small interpretable decision layer (63k parameters) can compete with large MLLM-based croppers while keeping the language model in a bounded, auditable role. It also demonstrates a way to reduce hallucination in aesthetic critique by filtering training drafts against the same physical measurements the model is asked to talk about — a template applicable to any system that generates explanations about measurable image properties. Extending cropping from pre-shot photographs to raw 360° environments moves the task one step earlier in the photographic pipeline.

Real-world applications:

• Consumer and smartphone photography: suggesting a direction and framing when a user captures panoramas or 360° content, with camera parameters that allow native re-rendering at full resolution.
• Virtual and augmented reality content creation: turning HDRI-style environment captures into well-composed stills and thumbnails, with an explicitly reasoned choice of viewing direction.
• Automated photo editing and curation tools: cropping existing image libraries while flagging when a subject or object is being cut off.
• Photography education and capture-time guidance: explaining, in terms of concrete composition rules and measured values, why a particular frame was chosen.

Industry relevance. The method's use of a compact tree-based decision layer and a bounded MLLM is attractive for deployment: it is cheap to run relative to end-to-end MLLM croppers, produces per-feature attributions, and does not require retraining per dataset (one checkpoint was used on both FCDB and FLMS). The authors are affiliated with MediaTek Inc. and two universities, suggesting relevance to mobile imaging and camera-pipeline products.

Future Directions

• Continuous refinement instead of discrete steps. The authors note that framing adjustments rely on discrete step fractions, approximating targets that fall between sampled intervals; continuous refinement is left to future work.
• Multiple preferred views per scene. View selection uses a single argmax, returning one frame even in scenes with several strong regions, regardless of user preference; user-guided or multi-hypothesis view selection is proposed as future work.
• Broader evaluation of explanation faithfulness. The current verification is a rule-based detector over numeric and directional claims, tested on the fifth-scale studies; scaling faithfulness checks and human evaluation to more images and domains remains open.
• Rethinking the role and calibration of the MLLM consultant. Since the calibrated head's attributions require a secant approximation, and the consultant operates under bounded influence, further work could examine how much proposal quality the consultant can contribute without compromising the interpretability of the decision layer.

Target Audience

• Computer vision researchers working on image cropping, composition understanding, aesthetic assessment, and 360°/omnidirectional vision.
• Practitioners building camera and imaging pipelines who need a lightweight, inspectable framing component rather than a large end-to-end model.
• Researchers interested in explainable MLLM systems, particularly the use of measurement-conditioned generation and rejection-sampling-style filtering to control hallucination.
• Graduate students and advanced undergraduates with background in deep learning and vision-language models who want a clear example of combining foundation models with classical, interpretable decision layers.

Reading level guidance: the main paper is approachable at an intermediate level, but full appreciation of the token definitions, the LightGBM/isotonic-calibration/TreeSHAP details, and the DPP shortlisting requires familiarity with modern vision models and gradient-boosted trees.

Authors’ abstract

Traditional image cropping assumes the photograph already contains the composition worth keeping, and asks only where to crop. A full 360$^{\circ}$ scene offers no such guarantee: the viewing direction in a scene has to be discovered before anything can be framed. We present PhotoGuider, a measure-then-frame method that answers both questions with compact learned scorers over one shared set of composition-aware features. The designed features are extracted using foundation and task-specific vision models and structured into four photographic tokens: Subject, Composition, Aesthetics, and Integrity. Our MLLM explainer is fine-tuned to generate composition-related explanations conditioned on images and measured token assessments. To our knowledge, PhotoGuider is the first method to combine explicit composition-based direction selection in 360$^{\circ}$ panoramas with refined framing, camera-parameter output for native re-rendering, and measurement-conditioned explanations. On two standard 2D cropping benchmarks, PhotoGuider achieves the highest average mIoU performance against ground-truth crops. In a blind ranking study, 120 participants ranked three cropping methods over 100 panoramic scenes from the public Poly Haven HDRI library, placing our framings first in 53.2% of cases, against 27.3% and 19.5% for two strong alternatives. Project page: https://ntuneillee.github.io/research/photoguider/