Building Footprints from Aerial Imagery: SAM 3 and ESRI

Compared SAM 3 and ESRI building extraction on 10 cm aerial imagery, then evaluated frozen predictions against manually reviewed 2026 building outlines.

Why it matters

A reviewed reference helps distinguish model errors from gaps in an older building inventory. It shows which extraction results are useful for inventory review and where small structures still need attention.

My contribution

I built the Python extraction and evaluation workflow, ran SAM 3 and local ESRI processing, and compared those outputs with the city-published ESRI run. I manually reviewed the reference outlines and evaluated the frozen predictions using shared scoring rules. I did not train or fine-tune either model.

Result

The city ESRI run performed best on 468 reviewed buildings of at least 20 m²: F1 was 0.958, compared with 0.920 for local ESRI and 0.873 for SAM 3. All three missed most smaller structures. The models were unchanged; the update provides a better reference for evaluating them.

Tools

  • Python
  • PyTorch and CUDA
  • SAM 3 / SamGeo
  • ArcGIS Pro and ArcPy
  • rasterio and GDAL
  • GeoPandas and Shapely
  • SciPy
  • pytest
  • GitHub Actions

These results cover one reviewed test square and do not establish city-wide accuracy. The reference retains historical geometry and was reviewed by one person.

Approach

I compared SAM 3, local ESRI processing and the city-published ESRI output using shared cleanup and scoring rules. I then reviewed the reference outlines against the imagery and evaluated the existing predictions without rerunning or retuning either model.

Imagery preparation and model runs followed by frozen outputs, manual reference review, and separate primary and all-size scoring
The later review changes the evaluation reference, not the model outputs. Scroll horizontally to explore the diagram.

Reviewed test-area results

City ESRI had the strongest overall result. Among buildings of at least 20 m², SAM 3 found more than local ESRI but also produced more unmatched outlines, lowering its F1 score.

Small structures were the main weakness: of 132 reviewed outlines under 20 m², SAM 3 and local ESRI each matched four; city ESRI matched none. Including these smaller outlines lowered all three methods’ F1 scores. These are size groups, not confirmed building types.

Precision, recall and F1 for city ESRI, local ESRI and SAM 3, with separate all-size and at-least-20-square-metre reviewed test cohorts.
One reviewed 700 m test square, Lewisville. All sizes: 600 outlines; primary ≥20 m²: 468. Predictions and settings were unchanged. Scroll horizontally to explore the diagram.
Reviewed 2026 accuracy in one 700 m test square, Lewisville. Primary: 468 outlines ≥20 m². All sizes: 600 outlines. F1 is a score, not percent accuracy.
CohortMethodPrecisionRecallF1
Primary ≥20 m²ESRI, city run0.9650.9510.958
Primary ≥20 m²ESRI, local raw run0.9360.9040.920
Primary ≥20 m²SAM 30.8280.9230.873
All sizesESRI, city run0.9650.7420.839
All sizesESRI, local raw run0.9320.7130.808
All sizesSAM 30.8240.7270.772

Manual reference review

I added missing outlines and corrected errors against the aerial imagery. I confirmed that the test review used imagery and historical outlines without seeing model predictions. Coverage and independence are reviewer attestations; no second reviewer is recorded. Old Town was used for tuning, not as another independent test.

Completed reference review. Boundary crossings were reviewed but excluded from scoring.
AreaReviewedFully insideBoundary crossingsRole
Old Town39337419Development / tuning
Test area64060040Reviewed test evaluation

Historical inventory agreement

City ESRI also scored highest against the older city-wide inventory. These scores measure agreement with historical outlines, not current city-wide accuracy. Differences from the reviewed results reflect the changed reference and scoring rules, not improved models. Outlines without inventory matches remain candidates for review.

Historical city-wide inventory agreement: outlines found by both models, one model or neither, grouped by area
Earlier comparison of 38,882 historical outlines with city ESRI and SAM 3. Size groups are not building-type classifications. Scroll horizontally to explore the diagram.
Historical city-wide inventory agreement, outlines ≥20 m². Separate from reviewed test-area accuracy.
MethodHistorical city-wide F1, ≥20 m²
ESRI, city run0.834
ESRI, local raw run0.766
SAM 30.744

Engineering and next steps

The workflow supports resumable processing, consistent scoring and reproducible reports. SAM 3 fine-tuning for small structures is deferred and needs separate training, validation and fresh test areas. Large roofs can still split at processing boundaries.

Earlier groundwork recovered all 120,426 expected tiles from an October 2025 collection and assembled the imagery mosaic. This coverage check is separate from the May 2026 accuracy comparison.

Technical details

Reviewed baseline complete · Lewisville, Texas · Nearmap imagery, May 2026 · Approximately 10 cm resolution

Building inventories become outdated. I compared existing extraction models on the same aerial imagery, then reviewed reference outlines to separate current test-area accuracy from agreement with an older inventory. The existing city-scale imagery workflow processed 21.1 gigapixels; that processing extent is larger than the reviewed test area.

  • Prepare imagery and run or import predictions, with shared geometry cleanup and resumable processing.
  • Keep settings selected in Old Town and existing predictions frozen; review reference outlines against May 2026 imagery.
  • Evaluate the separate test square with common boundary, area and one-to-one matching rules.

Scoring and labeling rules

The reviewed baseline was completed on October 15, 2026. All methods use the same reviewed reference, study boundary, metric coordinate system (EPSG:26914) and one-to-one matching at intersection-over-union (IoU) ≥0.5. The primary run filters both predictions and references to ≥20 m² after boundary selection and before matching. The all-size run matches the 600 fully contained references separately; 40 boundary crossings are excluded. Size diagnostics cannot be substituted for the primary filter. Precision measures the proportion of predictions matched to the reference, recall measures the proportion of reference buildings found, and F1 combines both.

Labels follow visible roof outlines of permanent roofed buildings, including garages and sheds around 4 m² or larger. Open carports, canopies, pergolas, tents, trailers and vehicles are excluded. The approximate minimum retains three reviewed outlines just below 4 m². Boundary-crossing buildings were reviewed, then excluded from scoring.

The review retains historical geometry and building-splitting conventions rather than a fully independent redraw. I followed the labeling rules and reported no unresolved cases. The historical city-wide comparison used 38,882 evaluated outlines from a 38,924-record inventory, primarily dated 2015 or earlier; 33 undated records remain in the evaluated set. Inventory absence does not establish construction or demolition dates. City and local ESRI outputs have different processing histories.

The toolkit retains chunk ownership, geometry cleanup and one-to-one evaluation. Review records connect frozen references and study boundaries to aggregate results and sanitized provenance. Export checks verify source bindings and refuse accidental overwrites. Public aggregates support regeneration of charts and reports, not the private-data geometric evaluation. Imagery, reviewed labels, predictions, per-building matches and full receipts are not distributed here.

Recorded software validation spans Windows and Ubuntu with Python 3.12, 3.13 and 3.14: 282 local tests passed with two optional-dependency skips, and all eight checks passed for the published documentation revision. These checks validate software and reporting, not model accuracy.

Deferred fine-tuning must preserve the frozen test baseline and label every eligible building in separate training, validation and fresh test areas. No real training run, trained checkpoint, new labeling package or confirmed training feasibility on the 6 GB GPU exists yet. Tile overlap reduces edge losses but does not guarantee complete large-building outlines.

Image viewer

100%