The Pulse
Property Finder Finds 96.8% of Its AI Labels Came From the Old Model
Property Finder found that 96.8% of labels in its property-image evaluation set came from its incumbent vision model, biasing comparisons with alternatives. After independent annotation of 356 UAE images, the company selected Llama 4 Maveri
AI.info Team · Written with AI · fact-checked ·
On October 27, AWS and Property Finder disclosed that 96.8% of the labels in the real-estate platform’s vision-model evaluation set came from predictions made by the model it was considering replacing. That meant the test favored the incumbent’s answers, even when they were wrong, and could make competing systems look less accurate simply because they disagreed. Property Finder uncovered the problem while evaluating lower-cost options for checking listing images.
The issue affected a system that processes tens of thousands of property images a day across the Middle East and North Africa. It assesses 11 attributes, including room type, furnishing, watermarks, photo quality and construction indicators. The company says its listings average 15 images, each requiring quality assessment before publication.
The answer key repeated the incumbent’s mistakes
When an evaluation set adopts outputs from the model under review as its answer key, it measures agreement with that model rather than correctness. AWS and Property Finder found that watermark-text labels marked character-level errors as correct, including missing characters and invented brand names. The existing single-label setup also forced one room category onto pictures that showed multiple room types.
The remaining 3.2% of labels had been written by people, but they were not consistently reliable either: some included categories outside the original specification, such as “Office.” The authors describe the broader problem directly: “But this creates a circular evaluation where any alternative gets judged by how closely it matches the incumbent’s behavior, not by actual correctness.” Khaled Thabit, Meriem Boughrara, Mina Metias, Mohamed Desouky, Mohamed Noordeen Alaudeen and Vadim Selyutin
Fresh labels changed the model comparison
A real-estate strategy and analytics expert independently annotated 356 images from UAE listings, without access to the old model’s predictions. The first 256 images formed the development set used to tune prompts; 100 more were annotated after those runs for the final comparison. AWS says a separate set of 230 images from Egypt served as a regional check, but the post does not report results for it.
The team also switched room and photo classification to multi-label output, so an image showing a bedroom through a doorway could receive both room labels. It iterated prompts from version zero to version 27, using error analysis and automated optimization in later stages. On the 356-image evaluation set, Llama 4 Maverick 17B Instruct scored 0.80 on Micro F1, compared with 0.56 for the proprietary incumbent, which used a different tuned prompt.
A higher score, with an important testing caveat
The comparison is not a fully independent test: the 356-image evaluation set includes the 256 development images, and both models’ prompts were tuned using those development examples. AWS therefore calls the scores optimistic. The incumbent received one tuning round, while Llama received several rounds and additional Property Finder rules for edge cases, giving the two models different amounts of prompt-tuning effort.
Property Finder selected Llama for production, and the post says it matched or exceeded the incumbent on all 11 attributes; the two models tied on multi-label photo classification. Llama’s exact-match scores varied by task: 100% for rotation detection, 94% for watermark detection and 55% for watermark-text extraction. The weaker text result shows why a single overall score can obscure the places where a system still struggles.
The cost estimate is not a like-for-like benchmark
AWS estimates Llama’s cost at $2.35 per 1,000 images, 86% below the previous setup. Those figures are estimates for the US East (Ohio) Region as of December 2025, based on 9,000 input tokens and 200 output tokens per image; AWS says the production prompt uses fewer input tokens, making the estimate an upper bound. The cost reduction compares against the previous setup using prompt version 21, while the accuracy comparison used version 22, so the two comparisons do not share an identical baseline.
Property Finder says its monitoring found 40% to 60% lower inference latency with the new pipeline. The practical finding is narrower than a claim that one model is simply better: model comparisons can inherit the incumbent’s errors when its own predictions supplied most of the labels. Here, the evaluation changed only after the company built an independently annotated set—and the reported scores still need a separate test set untouched by tuning to establish a cleaner comparison.