The one-degree bug that almost shipped wrong frost dates

The build story behind our free US frost-date calculator — the bug that nearly shipped

The one-degree bug that almost shipped wrong frost dates

While building the precomputed frost map behind our free frost-date lookup, a one-degree difference in how two agencies name their map tiles put wrong elevation data under about 3.3 million grid cells — nearly all of it invisible to normal checks. A rule requiring overlapping map sections to match byte-for-byte caught the 8,831 cells that leaked, and saved the ship.

Some background, quickly

This week we shipped a new engine for our free frost lookup: frost odds computed for your own ~800-foot square, from 26 years of NASA daily temperature data. The launch post has the what and the why.

This post is about the bug that nearly poisoned it, and the boring, paranoid check that caught it.

If you've ever shipped software, you'll recognize the shape of this story. If you garden and don't code, stay anyway — the ending is about how much checking sits between raw government data and the frost date we hand you.

The bake

The precomputed frost map the tool reads from — we call it the frost surface — isn't computed live. It's baked: every frost number for the whole lower 48, precomputed once, so the tool answers your lookup by reading 36 bytes.

The bake ran on a MacBook Air. Roughly 490 GB of data from NASA's Daymet — a dataset that reconstructs daily weather for every square kilometer of the continent — streamed through it: 30 years of daily low temperatures for the lower 48. Almost none of that raw data ever touched the disk; each chunk was reduced in memory to the frost numbers we keep, then thrown away. What did land on disk was a cache of elevation tiles (capped at 30 GB) and the finished ~8 GB surface. The work was cut into 107 resumable tiles — independent sections of the map that each bake on their own — so a crash costs one tile instead of the whole run. The first full pass took about 14 hours.

Each tile also needs elevation data, because the whole point of this surface is correcting frost odds for elevation. Hold that thought.

The paranoid rule

Tiles bake independently, and adjacent tiles overlap along their edges. Both sides compute the overlap on their own.

So the run had one hard gate: every overlapping cell must come out byte-identical from both sides. Not "close enough" — the same bytes. 8.68 million overlap cells, zero mismatches allowed, or nothing ships.

That sounds excessive for weather data. It isn't. If the pipeline is deterministic — same inputs, same math — two computations of the same cell must agree exactly. Any drift means the inputs weren't actually the same. The gate doesn't test whether the numbers look reasonable. It tests whether the pipeline is telling the truth about its inputs.

8,831 cells out of 8.68 million

The first pass came back with 106 of 107 tiles clean. And the seam check — that byte-identical rule on the tile overlaps — flagged 8,831 mismatched cells out of 8.68 million.

That's 0.1%. Every summary statistic looked fine. The tempting call was obvious: noise, edge effects, whatever — ship it. First-frost season was weeks away and the tool was sitting there almost done.

The rule said zero. So instead: root-cause hunt.

One degree of latitude

The trail led to the elevation data.

Elevation comes from USGS 3DEP, the government's ~30-meter terrain model. 3DEP has small coverage holes, mostly coastal. At those holes the pipeline falls back to Copernicus GLO-30, the European terrain model. Both agencies chop the planet into one-degree tiles and name each tile after a corner coordinate.

Here's the bug. USGS names a tile by its north edge. Copernicus names it by its south edge.

Same-looking name, two different squares of the planet — one degree of latitude apart, which is about 69 miles. So at every 3DEP hole, the fallback fetched the Copernicus tile one degree north of where it should have, and the pipeline blended terrain from 69 miles away into the elevation model. Across 29 coastal tiles, about 3.3 million of the 250-meter cells were sitting on contaminated elevation data.

The scary part: it agreed with itself

What makes this bug worth a post is how close to invisible it was.

When two adjacent tiles both hit the same 3DEP hole, they both fetched the same wrong Copernicus tile. So along most seams, both sides agreed — on the same wrong data. Agreement isn't correctness. It's just agreement.

Statistical validation wouldn't have blinked either. Elevations from 69 miles away are still plausible elevations — nothing an outlier check would flag. Frost dates built on them would have looked completely normal on every chart we had. They'd just have been wrong, quietly, for 3.3 million cells, mostly near the coasts. How wrong per cell? We can't say precisely — and that's exactly the problem. There'd have been no error bar and no warning. Just wrong numbers that looked like right ones.

The only cells that betrayed the bug were the 8,831 where the two tiles' processing windows happened to differ, so their fetched-wrong data differed too. A tolerance-based check would have shrugged them off. Byte-identical refused.

The fix, and the receipt

The fix itself was small, the way these things usually are: shift the tile-name conversion by one degree, version the download cache so no stale wrong tiles survive, and enumerate tiles in a window-independent way so the same class of bug can't sneak back in.

Then the unglamorous part: rebuild the 29 affected elevation mosaics (the stitched-together elevation maps, one per region), verify the changes stayed confined to the known fallback zones, and re-run the whole affected bake phase. The better part of another day of quiet MacBook grinding — the Air doesn't even have a fan to complain with.

The re-run came back: 0 mismatches across all 8.68 million overlap cells. Byte-identical on every seam. Only then did the surface ship.

What I'm taking from it

Paranoid gates pay. The byte-identical rule looked like overkill right up until it was the only thing standing between us and shipping wrong frost dates for the coasts. The same class of gate had already caught two unrelated bugs during the pilot runs. It stays in the pipeline forever.

Agreement is not correctness. Two independent-looking computations agreeing is only evidence when their inputs are actually independent. Ours shared a poisoned well, and they agreed enthusiastically about the wrong answer.

Small discrepancies are messengers. 8,831 out of 8.68 million reads like rounding error. It was the visible 0.1% of a 3.3-million-cell problem. The size of the leak tells you nothing about the size of what's leaking.

If you garden and skimmed the middle: this is the level of checking that sits between NASA's raw data and the frost date the tool hands you. When we publish accuracy numbers, this is what the measuring looks like — the full method, misses included, is on the methodology page.

The frost lookup is free, no signup.

The app we're building on top of all this — it remembers your garden and tells you what to do and when — is in beta now. Join the beta if you want in.