The $9,223,372,036 trade that never happened

I was building an order flow aggregation pipeline over CME futures data — ES and NQ, GLBX.MDP3, pulled through Databento's Batch API. Concentration, spoofing and rotation indicators, aggregated into bars, fed into a backtest.

The numbers came out wrong. Not obviously wrong — that would have been a gift. Wrong in the way that costs you a week.

SymptomEither I don't understand my own indicator, or the data is wrong

The indicator output was wrong, and the chart showed swings that made no sense to me. Not wrong enough to be obviously broken — wrong in the way that leaves you with two options: either I was misunderstanding my own indicator, or something upstream was feeding it bad data.

I spent a while on the first assumption. It was the second.

Root causeint64 has no null

DBN — Databento Binary Encoding — stores prices as fixed-point int64 with a scale of 1e-9. A price of $5,432.25 is stored as 5432250000000.

When a price field has no meaningful value, DBN can't store null. There is no null in int64. So it stores a sentinel: INT64_MAX. Divide that by 1e9, as you must to get back to dollars, and you get nine point two billion dollars per contract.

That value was flowing into my aggregations. Not as an error, not as NaN, not as an exception. As a perfectly ordinary float64 that every downstream function was happy to accept.

ImpactWhy this is worse than a crash

A crash is a good bug. It stops, it points at a line number, you fix it. This one has three properties that make it expensive.

It's a valid number

NaN propagates loudly, and pandas has a whole vocabulary for it — isna, dropna, fillna. A sentinel is just a big float. df['price'].mean() returns a number. df['price'].max() returns a number. Nothing anywhere says this is not a price.

It survives type and schema validation

The field is an int64 and it contains an int64. Every assertion about types passes. A range check with a sane upper bound is the only thing that would have caught it — and most people don't write one for price.

Its visibility depends on the statistic

This is the part that cost me the week.

Feed one sentinel into a mean over 500 trades and the result is so absurd you spot it in a second. Feed it into a median and it vanishes entirely — one observation at the top of the sort, and the median doesn't care. Feed it into a volume-weighted calculation where the undefined record also carries a tiny or undefined size, and it barely moves the number, so you get an answer that's wrong by a plausible-looking amount.

The bug's visibility is inversely proportional to how robust your statistic is. The more careful you were about outliers, the better you hid it from yourself.

My concentration indicator used a ratio of extremes, which is why it saturated. My VWAP, computed over exactly the same contaminated data, drifted slightly and said nothing at all.

It had been in there for weeks, silently, across several scripts. Every aggregation script I'd written had inherited the same unvalidated load path.

ExposureWhere undefined prices show up

These aren't rare or exotic. They're normal in:

• MBP-10 book levels that aren't populated. If an instrument is quoting three levels deep and your schema gives you ten, levels four through ten come back with sentinel prices. Iterate all ten to compute book imbalance and you've just added seven nine-billion-dollar quotes to one side.
• Records where a price field doesn't apply to that event type but exists in the struct anyway.
• Size fields. UINT32_MAX is the equivalent sentinel for undefined sizes. Same failure mode, different magnitude.

The book-levels case is the one to watch. It isn't an edge case at all — it's the normal state of the back of the book in a thin instrument, and it's exactly where imbalance and concentration indicators do their work.

FixFilter at load time, once

Before anything touches the data:

import numpy as np
import pandas as pd

UNDEF_PRICE = np.iinfo(np.int64).max     # 9223372036854775807
UNDEF_SIZE  = np.iinfo(np.uint32).max    # 4294967295
PRICE_SCALE = 1e-9

def clean_dbn(df, price_cols, size_cols=(), max_plausible=1e6):
    """
    Convert DBN fixed-point prices to floats, mapping sentinels to NaN.

    Returns (df, report). Always read the report — a nonzero sentinel
    count in a column you did not expect is information.
    """
    report = {}
    out = df.copy()

    for c in price_cols:
        raw = out[c]
        n_sentinel = int((raw == UNDEF_PRICE).sum())
        px = raw.astype("float64") * PRICE_SCALE
        px = px.mask(raw == UNDEF_PRICE)

        # Belt and braces: anything absurd is not a price, whatever
        # produced it. Catches sentinels you have not met yet.
        n_absurd = int((px.abs() > max_plausible).sum())
        px = px.mask(px.abs() > max_plausible)

        out[c] = px
        report[c] = {"sentinel": n_sentinel, "absurd": n_absurd}

    for c in size_cols:
        n_sentinel = int((out[c] == UNDEF_SIZE).sum())
        out[c] = out[c].astype("float64").mask(out[c] == UNDEF_SIZE)
        report[c] = {"sentinel": n_sentinel}

    return out, report

Two things about this matter more than the code.

Return a report, and read it. Silent cleaning is how you get a second version of the same bug. If a column you assumed was always populated turns out to be 4% sentinels, you want to know, because it probably means you misread the schema.

Keep the max_plausible check. It's redundant against the sentinel you know about. It is not redundant against the next one. Sentinel conventions differ across vendors, formats and schema versions, and the line costs nothing.

Then fail loudly downstream:

if not np.isfinite(px).all():
    raise ValueError(f"non-finite prices in {name}: "
                     f"{(~np.isfinite(px)).sum()} of {len(px)}")

NaN isn't a solution either. It's a loud problem, which is the entire point — you decide what to do about it, in the open, instead of finding out six weeks later that your backtest was trading against a nine-billion-dollar quote.

GeneralThe question to ask any feed

Market data encodes "missing" in whatever the wire format allows. int64 gives you no null, so you get INT64_MAX. Elsewhere you'll find -1, 0, 999999, 1e30, or the string "N/A" sitting in an otherwise numeric column.

Every one of these is a valid value of its type. None will raise. All will flow into your statistics and give you an answer.

What does this format use to say "nothing here", and what does that turn into after my unit conversion?

For DBN prices, the answer is nine point two billion dollars.