Capstone: Design and Defend a Computer Vision System

Each component creates evidence, dependencies, and failure modes — and component 2 is where designs quietly overclaim.

Automatic redaction of vehicle-collected street imagery is reported as a measured recall figure, and it has never been 100%. Google described its own system in 2009: a high-recall sliding-window detector followed by a false-positive post-filter. It blurred more than 89% of faces and 94–96% of license plates on evaluation sets sampled from Street View. Nine years later Mapillary published its number for the same problem: “The resulting privacy blurring model is able to detect 99.9% of all identifiable faces and license plates while wrongly blurring only 0.078% of the pixels.” Mapillary states the residual plainly — roughly one miss in a thousand.

So the quality and privacy filter does not remove privacy risk. It reduces it at a rate you have to measure and report. Across a citywide vehicle fleet, one in a thousand is a count rather than a rounding error. The retention, access, and deletion rules downstream must be written for the frames the filter missed.

A narrower scope may create a safer and more useful first release. And the narrow scope here is not an invention of this capstone. It is an existing tier of an American National Standard.

ANSI A300 (Part 9)-2011 defines Level 1 as a limited visual assessment made “from a specified perspective such as foot, vehicle, or aerial patrol”. The standard says it directly: “Level 1 assessments shall be a limited visual assessment of an individual tree or a population of trees near specified targets, such as along roadways or utility rights-of-way, to identify specified conditions or obvious defects.” A 360-degree ground inspection is reserved for Level 2. Instruments — drilling, sounding, pull testing, resistance drilling — are Level 3.

Drive-by imagery is therefore Level 1 evidence before any model touches it. Automated diagnosis is a Level 3 claim resting on Level 1 input. The standard also settles the decision rights this capstone would otherwise have to argue for: clause 93.6.4.1 leaves it to the owner, the owner's agent, or the controlling authority to determine actions and schedule further assessment. Inspection prioritization is the option that keeps the system inside the tier its evidence belongs to.

The unit of analysis is not an isolated frame. Two of these steps have measured numbers attached rather than good intentions.

Step 4, split structurally. The Auto Arborist dataset joins public tree censuses from 23 US and Canadian cities with street-level and aerial imagery: over 2.6M trees, over 9.1M images, 344 genera. Its evaluation protocol holds out one city per region. Beery and colleagues trained a single-view street-level ResNet-101 on one city and then on the full dataset, and class-averaged recall rose by 21.3% on average across cities — 18.3% for regional training alone. Their 2022 paper opens with the reason: “Generalization to novel domains is a fundamental challenge for computer vision. Near-perfect accuracy on benchmarks is common, but these models do not work as expected when deployed outside of the training distribution.” Geographic holdout is not a precaution in urban tree vision. It is the dominant effect. A random split inside one city measures far less than it appears to.

Step 3, create references. The reference a municipal system would actually be validated against is a decadal, volunteer-built inventory with a coarse subjective health field. New York City describes its own street tree census this way: “Data collection ran from May 2015 to October 2016 and the results of the census show that there are 666,134 trees planted along NYC’s streets.” The published table holds 683,788 records — 652,173 Alive, 13,961 Dead, 17,654 Stump. Its perceived-health field takes three values: Good (528,850), Fair (96,504) and Poor (26,818). Another 31,616 records carry no health value at all. Design the schema, the metrics, and the uncertainty story around a three-class field refreshed on a ten-year cycle, not around a continuous ground truth that does not exist.

Separate visible observations from expert diagnoses and actions. Then set the granularity of each visible field at what an observer can actually reproduce.

That boundary has been measured. In 2017 Roman and colleagues compared volunteer and expert street tree data in Lombard IL, Grand Rapids MI, Philadelphia PA and Malmö, Sweden. Volunteers missed 1.2% of trees and counted 1.0% extra. They were about 90% consistent with experts on site type, land use, dieback and genus, and 84.8% consistent on species within a correct genus. Their verdict on the crown fields is blunt: “Crown transparency and wood condition had the poorest performance and participants expressed concerns with these variables; we conclude that these variables should be dropped from future citizen science projects.”

The same boundary shows up independently in a 2018 study by Hallett and Hallett. Defoliation, discoloration and vigor agreed within two classes 100%, 93% and 92% of the time. Crown dieback agreed within 10% only 76% of the time, and transparency within 15% only 76% of the time. About 90% on genus and 76% on crown condition is the schema's design constraint: coarse ordinal classes, and a genuine ungradable option exactly where the humans disagree with each other.

Use an input-quality model before tree detection. Segment or localize visible regions, aggregate repeated views, and estimate uncertainty for each attribute rather than collapsing everything into one opaque risk score.

Compare strong baselines: route and maintenance rules, image-quality triage, simple classifiers, and a modular detector-segmenter pipeline. Complexity must earn its operational cost. Auto Arborist sets the honest scale of that comparison, since a ResNet-101 gained 21.3% class-averaged recall from being trained across cities rather than one. That is a data-coverage result, not an architecture result, and no stack change substitutes for it.

The surrounding cost was named in 2015. Sculley and nine colleagues argued that “it is common to incur massive ongoing maintenance costs in real-world ML systems”. Their paper on hidden technical debt lists the ML-specific risk factors: “boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns”. None of that lives in the perception stack. All of it lands in the operations plan.

Approval requires evidence across model, workflow, and city operations. Every row should be a number another team can reproduce.

Three rows already have published comparators. Capture validity includes redaction recall, and the public reference points are more than 89% of faces in 2009 and 99.9% in 2018. Report yours the same way — a rate with a denominator, not a checkbox. Perception performance must be reported across held-out cities or districts, because a within-city score is precisely what conceals the 21.3% class-averaged-recall gap. And the arborist-agreement row carries a ceiling of its own. Assessors agree only 76% of the time on crown dieback and transparency, so a model scored against those labels cannot be credited with more precision than the reference itself holds.

Do not deploy citywide on one retrospective score. Lock the geographic holdout before the first fit — Auto Arborist holds out one city per region across its 23 cities for exactly this reason. Treat each later stage as a new question rather than a confirmation of the previous one.

The retrospective stage scores against a census with three health classes and 31,616 records carrying no health value. That is why stages 2 through 5 exist. A silent field trial, a controlled assistance study, and a limited district pilot generate evidence about coverage, workflow, and disagreement that no locked historical table can supply.

A design is defendable when another team can inspect its evidence and assumptions.

Two of those deliverables have published templates. Mitchell and colleagues proposed model cards in 2019 as “short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type) and intersectional groups”. They also “disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information”. Gebru and colleagues proposed datasheets for datasets by analogy with electronics: “In the electronics industry, every component, no matter how simple or complex, is accompanied with a datasheet that describes its operating characteristics, test results, recommended uses, and other information. By analogy, we propose that every dataset be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on.” That proposal reached the Communications of the ACM in 2021.

NIST published a framework for the governance half. The Artificial Intelligence Risk Management Framework (AI RMF 1.0) was released on 26 January 2023, and it organises the work into four functions: govern, map, measure and manage. The framework “is intended for voluntary use and to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems”. On a project like this one, its value is the vocabulary rather than the authority.