Latency — v0.8.0

The latency-characterization tier stands beside correctness and never becomes a scorecard column. This release retains 3,157 cold timings. The 879 unsupported outcomes have no invented timing.

Recorded quiet-host windows

The warm and invocation-overhead stages ran only after repeated low-load observations recorded in the v0.8.0 ledger. OS caches remained uncontrolled, so the figures characterize those runs and do not establish causal speedups.

Joern Java batches 1, 2, 4, 8, 16 and Semgrep Java batches 1, 2, 4, 8, 12 were each repeated twice. Nine overhead groups were measured three times each. The full retained ranges are published; repeats are not averaged or selected.

What these numbers are, and what they are not

This is a descriptive characterization of per-case analyzer wall-clock, published under the contract preregistered in docs/latency-tier.md, which was merged before a single timestamp was captured. It is not a score, not a ranking, and not a benchmark result in the sense the rest of this site uses that phrase.

• Latency is never pooled with correctness. There is no combined number anywhere on this site, no efficiency-adjusted rate, and no leaderboard that blends the two. "Correct but slow" and "fast but wrong" stay independently visible. No correctness outcome in this freeze was derived from, conditioned on, or tie-broken by a timing value.
• A latency number is a property of a run, not of a case. No case carries a timing assertion, threshold, or budget.
• Characterization, not micro-benchmark. There are no repeated trials, no warm-up iterations, and no steady-state harness in the cold figures below. The case population is the sample and its spread is the statistics. The distribution columns below are that spread — not error bars on a measurement that was repeated, because it was not. (The warm-marginal section is the one place a series is measured more than once. It says so, retains every repeat, and publishes the range they span rather than averaging them into a statistic.)
• These are the conditions the numbers were produced under: a single developer machine, under light concurrent load, running the benchmark's standing sequential-run discipline — one analyzer at a time, never two at once. That discipline is what makes the numbers usable at all; the light concurrent load is what stops them being a controlled measurement. Read them as characterization of what this benchmark actually costs to run, at the order-of-magnitude and shape level, and not as a precise figure for any engine.

Environment stamp

Recorded once per run beside the tool identity the run witnessed, and shown here verbatim. Latency numbers are comparable within one environment and are not comparable across machines. Every published run to date executes on one maintainer machine, and that caveat travels with every number on this page.

Hardware model	OS	OS release	Logical CPUs	CPU architecture	Timed runs stamped

The granularity is unequal, and that is the point

A phase is timed if and only if the adapter already invokes it as a separate subprocess. The benchmark never instruments analyzer internals, never patches an upstream tool, and never infers a boundary a subprocess boundary does not expose. The adapters do not expose the same boundaries, so the rows below are not equally decomposed — and the two ways of papering over that would both misdescribe the evidence: summing a decomposed adapter's phases and presenting every row as one number of the same kind, or guessing phase splits for the adapters that expose only one.

So the table states each adapter's declared granularity as data. Phases compare within an adapter; analyzer-invocation totals compare across adapters. A CodeQL database-create number and a Joern total are not the same kind of quantity, and nothing on this page sets them side by side as though they were.

Adapter	Version	Observed subprocesses per case	Phases timed	Timed invocations
Bifrost	0.11.4	0 or 1	total	1031
FlowDroid	2.15.1	0 or 1 or 3	total compile dex analyze	154
Semgrep CE	1.177.0	0 or 1	total	196
Infer	v1.3.0	0 or 2	capture analyze	204
Pysa	0.10.0	1	total	102
OpenTaint	v0.4.6	0 or 1	total	152
Joern	4.0.628	0 or 1	total	460
CodeQL	2.27.0	2	database-create database-analyze	858

3 of 8 adapters expose more than one subprocess and are decomposed here (FlowDroid and Infer and CodeQL); the other 5 expose a single invocation and take one honestly labelled number, with interpreter or JVM start-up inside it. That is a fact about invocation shapes, not about engine architecture: an adapter with one number is not an adapter that does one thing.

The phase names above are the ones the retained evidence carries, so any name here can be found in the artifact it came from. For CodeQL those are database-create and database-analyze, after the subcommands that produce them, where the preregistration's table spells the same two boundaries extract and analyze. Two spellings, one pair of subprocess boundaries: nothing moved and nothing is attributed differently. The contract is immutable, so the difference is recorded in its Amendment A12 rather than edited out of its table.

Analyzer-invocation wall-clock, per adapter

The only cross-adapter reading this tier supports: each adapter's cold per-case analyzer-invocation wall-clock, over every case in this snapshot's explicitly bound latency population that it actually invoked. Adapter-observable setup phases remain visible below, but only phases the contract classifies as analyzer work enter this total. Per-case timing at this granularity includes per-invocation fixed costs — JVM start-up, extractor initialization, interpreter start — that a long-lived deployment of the same engine would amortize. This tier does not correct for that; it characterizes what the benchmark actually runs.

Fastest to slowest, and the same ranking per kernel

Ordered by median, on a logarithmic axis, with the spread drawn beside every median rather than left to the table. The toggle scopes the same ranking to one language kernel at a time: a whole-corpus median mixes the languages an adapter runs on, and holding the language fixed is the difference between "this engine is slow" and "this front end is expensive". Only the analyzers that actually invoked on a kernel appear in its view. The chart draws the median, the quartiles and the p10–p90 spread; the disclosure beneath it opens the same distributions as a table, with the minima and maxima the whiskers deliberately leave out.

All timed invocationsccppcsharpgojavajavascriptkotlinphppythonrubyrustscalatypescript

Every timed analyzer invocation the freeze binds — 3157 of them, across every score tier and both model profiles. This is the widest denominator on the site and the only one here that is not a single population: an adapter's median mixes the languages it runs on, whose fixtures differ in size and whose front ends differ in cost. The per-kernel views hold the language fixed.

The c kernel's 56 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 56, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The cpp kernel's 68 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 68, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The csharp kernel's 70 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 2 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 70, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The go kernel's 70 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 70, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The java kernel's 70 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 7 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 70, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The javascript kernel's 70 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 70, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The kotlin kernel's 70 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 5 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 70, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The php kernel's 70 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 70, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The python kernel's 70 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 5 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 70, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The ruby kernel's 70 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 70, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The rust kernel's 62 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 62, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The scala kernel's 70 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 1 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 70, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The typescript kernel's 70 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 70, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

• median, printed beside every row
• interquartile range (Q1–Q3)
• p10–p90; the minimum and maximum are in the tables, not the whiskers
• indented rows: an adapter's own declared phases
• measured warm marginal per case — the range its retained repeats span
• estimated per-invocation overhead (trivial fixture, upper bound), drawn across the range its repeats spanned, where that range starts at or above 25% of the row's median

How to read this figure

Every bar above is cold per-invocation wall-clock, and the warm marginal is a different quantity measured separately. Cold is what this benchmark actually runs — one process per case, start-up inside the number, because start-up is not observable from inside a single invocation. Read across runtimes, though, those bars overstate the steady-state gap: a JVM engine's row carries a JVM start a long-lived deployment pays once. So the other quantity is measured directly rather than estimated and subtracted — k cases through one tool process, for increasing k, reporting the slope of batch wall-clock against k. The carets mark it on the rows that have one, and they mark a range: the whole series is measured more than once and the figure published is the span its repeats cover, so the mark's width is its own precision rather than a number stated more exactly than it was measured. It is never subtracted from a median, never substituted for one, and never used to order the rows. Only adapters whose released CLI exposes a real multi-case batch have a figure at all; the rest are not observable with the released CLI, and the warm-marginal section records every verdict, measured and declined, with the evidence behind it.

The dashed spans are estimates, not measurements of the same kind as the bars. Each is the wall-clock of one complete adapter invocation — same pipeline, same policy, rule or query, same subprocess shape — over a trivial no-flow fixture: both benchmark endpoints declared, nothing connecting them, nothing to find. That is fixed per-invocation overhead plus the trivial fixture's own near-zero analysis, so it is an upper bound on what an adapter pays before it starts work, and it is a cold single-shot execution — the same posture the bars are measured in, and not a steady-state one. It is never subtracted from a median, never substituted for one, and never used to order the rows. The width of a span is the figure's precision, not a decoration: each measurement is repeated a fixed number of times and what is published is the range those repeats spanned — never a mean, never one chosen repeat, and never withheld for repeats that disagree, because a disagreement widens the range and that is the honest consequence of it. A span is drawn only where the range starts at or above 25% of that row's own cold median — a threshold fixed in the amendment before any estimate was measured, read at the low end so that no mark can appear on the strength of one slow repeat. The CodeQL estimate is below it and carries no mark — small, not unmeasured. Each estimate is measured in one named language, stated on the row it annotates, because boot cost is not language-free. The estimates table carries every value — every repeat behind every range, every unmarked row, and every adapter for which no estimate could be taken at all.

The axis is logarithmic. Each labelled tick is three times the one before it, so equal distances are equal ratios, not equal durations — the gap from 100 ms to 300 ms is drawn the same width as the gap from 10 s to 30 s. That is the only way this bound corpus's medians fit in one picture: they span 109 ms to 58.2 s, and on a linear axis every analyzer except the slowest would be a sliver against the origin. Because a log axis is easy to misread, every median is also printed as a number at the right of its own row.

The indented rows are phases, and they are not comparable across adapters. Only the 3 adapters whose preregistered row declares more than one subprocess have them. A phase mark sits on the same axis as the totals because it is the same kind of measurement — wall-clock of a subprocess — but reading one adapter's phase against another adapter's total is precisely the comparison the granularity rule forbids. Read a phase against the adapter it is indented under, and nothing else.

Ordering is not scoring, and this is never pooled with correctness. Rows are sorted by median because an unsorted ranking is unreadable, not because latency is a result. No correctness figure appears in this chart and no number here is blended with one: there is no combined score anywhere on this site, and a fast analyzer that answers wrongly is neither rewarded nor penalised by anything drawn above. Cases an analyzer declined before invocation are absent, not entered as zero — entering them as zero would make the analyzers that decline the most look the fastest, which is exactly backwards.

Show the data table — analyzer-invocation wall-clock per adapter, with minima and maxima

Adapter	Timed invocations	Median	Q1	Q3	Min	Max
Bifrost	1031	109 ms	94 ms	329 ms	79 ms	17.6 s
FlowDroid	154	683 ms	664 ms	856 ms	468 ms	926 ms
Semgrep CE	196	803 ms	787 ms	880 ms	763 ms	2.88 s
Infer	204	1.05 s	243 ms	3.62 s	237 ms	4.03 s
Pysa	102	2.60 s	2.56 s	2.74 s	2.47 s	5.21 s
OpenTaint	152	4.01 s	3.91 s	4.11 s	3.45 s	6.35 s
Joern	460	5.33 s	3.88 s	7.37 s	3.57 s	14.7 s
CodeQL	858	58.2 s	36.0 s	65.5 s	6.10 s	173.6 s

No mean is headlined anywhere on this page. The distributions are skewed by per-invocation fixed costs and by a small number of very long tails — the slowest single invocation in this bound latency corpus is 173.6 s, against a page-wide fastest median of 109 ms — and a mean over that shape would describe neither the typical case nor the tail. The adapters span roughly 534× between the fastest and slowest medians, which is the order-of-magnitude claim this tier is entitled to make, and the strongest one it makes.

Phase decomposition, within each adapter that has one

Only the 3 adapters whose declared row has phases appear here, and each is read down its own column. These numbers exist to decompose one adapter's own cost. Racing one adapter's phase against another's total is exactly the reading the granularity rule forbids. Each adapter's numbers open from its own disclosure, one per adapter, which is also the boundary the comparison rule draws.

FlowDroid2.15.1

Show the data table — FlowDroid's 4 observable phases against its analyzer total

Phase	Median	Q1	Q3	Min	Max	Share of the median analyzer total
total	767 ms	671 ms	859 ms	622 ms	926 ms	112%
compile — materialization	454 ms	446 ms	462 ms	425 ms	494 ms	excluded by contract
dex — materialization	297 ms	292 ms	305 ms	285 ms	325 ms	excluded by contract
analyze	517 ms	510 ms	524 ms	468 ms	543 ms	76%
analyzer total	683 ms	664 ms	856 ms	468 ms	926 ms	—

Inferv1.3.0

Show the data table — Infer's 2 observable phases against its analyzer total

Phase	Median	Q1	Q3	Min	Max	Share of the median analyzer total
capture	872 ms	61 ms	3.43 s	60 ms	3.85 s	83%
analyze	183 ms	181 ms	185 ms	175 ms	242 ms	17%
analyzer total	1.05 s	243 ms	3.62 s	237 ms	4.03 s	—

CodeQL2.27.0

Show the data table — CodeQL's 2 observable phases against its analyzer total

Phase	Median	Q1	Q3	Min	Max	Share of the median analyzer total
database-create	3.08 s	1.80 s	6.22 s	1.09 s	24.8 s	5%
database-analyze	51.7 s	30.7 s	62.4 s	4.35 s	172.3 s	89%
analyzer total	58.2 s	36.0 s	65.5 s	6.10 s	173.6 s	—

For analyzer phases, the share column is the phase median against the analyzer-total median, and the shares need not sum to 100%: a median is not additive, and the phase whose median is largest is not necessarily the phase that dominated any particular case. Materialization phases are labelled and excluded rather than made to look comparable. The analyzer-total row is the distribution of per-case contract-selected phase sums, computed per case and then summarized — not the sum of the phase medians.

Warm marginal cost, measured separately

Everything above is cold per-invocation wall-clock, and it stays the headline of this tier. It is also what this benchmark actually runs: one analyzer process per case, start-up inside the number, because start-up is not observable from inside a single invocation. Nothing in this section adjusts, corrects, or replaces one number above it.

But the ranked chart sets a native binary, a Python CLI, and five JVM or JVM-fronted engines on one axis, and read across runtimes those bars overstate the steady-state gap: a JVM row carries a JVM start that a long-lived deployment of the same engine pays once. The honest response is not to estimate that start-up and subtract it — that would be exactly the after-the-fact adjustment this tier's contract refuses. It is to measure the other quantity directly, which is what this section reports, under Amendment A15, preregistered before the first warm number existed.

The method, in one sentence: run k cases through one tool process for increasing k, and report the slope of batch wall-clock against k — the cost of one more case in a process that has already paid its start-up. A slope, never an average: an average per case at k still carries a 1/k share of the fixed cost, which is the very quantity the figure exists to remove. The batch reuses the cold runner's own case selection, endpoint resolution, workspace materialization, and query logic, so what is timed is the same work; only the number of cases sharing one process differs.

Adapter	Kernel	Cold median, same cases	Warm marginal (least squares)	Warm marginal (endpoint)	Batches	Repeats	Fitted fixed cost
Joern	java	7.59 s	4.80 s – 5.10 s	4.80 s – 5.12 s	k = 1, 2, 4, 8, 16	2	2.97 s – 3.61 s
Semgrep CE	java	816 ms	61 ms – 73 ms	53 ms – 74 ms	k = 1, 2, 4, 8, 12	2	741 ms – 858 ms

The cold column is the median over exactly the cases the warm batch analyzed, not over the whole kernel: two figures compared across different populations would mix a population effect into a start-up effect. The two columns are never subtracted. The "fitted fixed cost" is the same fit's intercept, published as the descriptive estimate it is — it is not measured, it is not a decomposition of any cold number, and nothing on this site subtracts it from one.

What makes these figures checkable. Each batch retains the per-case evidence it produced, beside the series above. Every document the largest Joern batch produced matches that case's retained cold evidence field for field — same analyzed state, same method count, same source and sink node counts, same flow count — differing only in the scratch path each run recorded for its own workspace. A batch that had quietly analyzed less would be visible there rather than merely fast. Every figure above is a range, and that is deliberate. The whole batch series is measured more than once, every repeat is retained, and what is published is the span the repeats cover. A single slope over a handful of batches on a developer machine has a precision, and there were only two other ways to give it one — publish one run and hide the spread, which understates it, or gate publication on an agreement tolerance, which means choosing that tolerance after the numbers exist, precisely the after-the-fact decision this tier's contract refuses. The range needs neither: its width is the precision, the reader sees it directly, and nothing between the measurement and this table is discretionary. The repeats are never averaged — that would turn repeated trials into the statistic the tier's non-goals rule out — and never picked between.

These are fresh v0.8.0 series. Historical A15/A21 measurements remain on their original snapshot pages. Semgrep's original batch-16 attempt failed before analysis; its prospectively corrected largest batch is 12. No batch-16 comparison, idle-machine explanation, or causal cross-release speedup is inferred from these desktop observations.

Show the batch series — Joern on the java kernel, all 2 retained repeats

Cases in the batch (k)	Run 1 — wall-clock of the one process	Run 2 — wall-clock of the one process	Average per case at run 2 (not the reported figure)
1	8.39 s	7.96 s	7.96 s
2	13.0 s	13.4 s	6.69 s
4	23.1 s	23.6 s	5.89 s
8	42.0 s	43.2 s	5.40 s
16	80.4 s	84.7 s	5.29 s
slope (least squares)	4.80 s	5.10 s	—

The average-per-case column is shown precisely so it can be discarded: it falls with k because it is still paying a shrinking share of the fixed cost, and it would keep falling with a larger k without ever being the marginal cost. The reported figure is the slope of a wall-clock column against the first, and the published figure is the range those slopes span. Measured on Mac16,1, macos27.0.0, 10 logical CPUs — the same stamp rule as every other number here, and not comparable across machines. Machine conditions: the one-minute load average sampled immediately before each batch ranged 1.4 to 3.8 across every batch of every repeat, on a 10-core machine. The tier's hygiene rule is that no other analyzer competes for the machine while one is timed, and background load is reported rather than asserted away — so the reader can weigh the conditions instead of taking "quiet machine" on trust.

Show the batch series — Semgrep CE on the java kernel, all 2 retained repeats

Cases in the batch (k)	Run 1 — wall-clock of the one process	Run 2 — wall-clock of the one process	Average per case at run 2 (not the reported figure)
1	1.04 s	816 ms	816 ms
2	906 ms	886 ms	443 ms
4	1.03 s	1.04 s	260 ms
8	1.33 s	1.32 s	164 ms
12	1.62 s	1.63 s	136 ms
slope (least squares)	61 ms	73 ms	—

The average-per-case column is shown precisely so it can be discarded: it falls with k because it is still paying a shrinking share of the fixed cost, and it would keep falling with a larger k without ever being the marginal cost. The reported figure is the slope of a wall-clock column against the first, and the published figure is the range those slopes span. Population:one `semgrep scan` carries one --config, so a batch is restricted to cases resolving to identical rule text: 12 of the 14 invocable Java kernel assertions. Measured on Mac16,1, macos27.0.0, 10 logical CPUs — the same stamp rule as every other number here, and not comparable across machines. Machine conditions: the one-minute load average sampled immediately before each batch ranged 2.1 to 2.1 across every batch of every repeat, on a 10-core machine. The tier's hygiene rule is that no other analyzer competes for the machine while one is timed, and background load is reported rather than asserted away — so the reader can weigh the conditions instead of taking "quiet machine" on trust.

Which adapters this could be measured on, and why not the rest

Each verdict below was reached by interrogating the pinned distribution — its help output, and where that was ambiguous its own bytecode — never from a README or an assumption about the runtime. No adapter is patched, forked, or invoked outside its released interface to make a batch possible. An adapter that ships no way to analyze several cases in one process has no warm figure here, and none is estimated for it from its runtime or from another adapter's slope.

Adapter	Observable with the released CLI?	Evidence
Joern	Yes — measured	A released multi-case batch that does the same per-case work, timed above.
Semgrep CE	Yes — measured	A released multi-case batch that does the same per-case work, timed above. one `semgrep scan` carries one --config, so a batch is restricted to cases resolving to identical rule text: 12 of the 14 invocable Java kernel assertions.
FlowDroid	Yes in the CLI — not measured here	The released CLI does have a batch: `-a/--apkfile` accepts a directory, and the shipped `soot-infoflow-cmd` main class lists its APKs, builds the taint wrapper once outside the loop, and iterates them in one JVM — its own help documents `-si/--skipapkfile` as "APK file to skip when processing a directory of input files", and it refuses a non-directory output with "The output file must be a directory when analyzing multiple APKs". It is not measured here because one invocation carries one `-s` sources-and-sinks definition, so a k-APK batch runs a union of k per-case endpoint configurations rather than each case's own. Whether that changes any case's result is an empirical question that has to be answered across the whole population before a marginal derived from it may be published. Named follow-up work, not a decline.
OpenTaint	Not observable with the released CLI	The pinned analyzer takes one `--project` and one `--output-dir` and exits after that project. A `project.yaml` may list several `javaProjects`, but analyzing their union is one whole-program analysis over a merged call graph and a merged entry-point set — different work, not k independent case analyses — and `--semgrep-rule-set` is one rule set for the whole invocation while the benchmark resolves a rule per case. No released mode processes separate case projects in one process.
Pysa	Not observable with the released CLI	`pyre analyze` is one-shot. The client does expose daemon commands (`start`, `incremental`, `query`), but they serve the type checker, not the taint analysis, and `analyze` never attaches to a running server. `--source-directory` is repeatable but merges directories into one project — again one whole-program analysis rather than k — and each case carries its own `.pyre_configuration`, `pyrefly.toml`, and resolved models. A daemon-shaped measurement would be stateful and not reproducibly preregisterable against this pin, so it is declined rather than attempted.
CodeQL	Not observable with the released CLI	`codeql database analyze` takes exactly one mandatory database, and `database create` produces exactly one database per invocation. Neither subcommand has a multi-database form in the pinned CLI.
Infer	Not observable with the released CLI	`--results-dir` names one capture database for one project and the analyzer exits after it. Worth stating explicitly: Infer's analyzer is a native binary, and the JVM cost inside its Java row is the traced `javac` in `capture` — per-project compilation work, not process start-up a batch could amortize.
Bifrost	Not observable with the released CLI	The policy CLI takes one `--root` per invocation; the repeatable `--workspace NAME=PATH` is documented as requiring `--mcp` and does not reach the policy path. What can be said without measuring anything is a bound: Bifrost's cold median already includes its own process start, so its warm marginal lies between zero and that cold number. Warm figures can therefore only move the other rows down toward Bifrost's, never Bifrost's row down further — the asymmetry this amendment corrects is one the publishing vendor's engine loses by.

The decline that matters most is Bifrost's, and it is stated in the direction that costs this benchmark's publisher something. Bifrost has no warm figure here, and its cold median already includes its own process start — so whatever its warm marginal is, it lies between zero and a number already published. Adding warm figures can therefore only move the other rows down toward Bifrost's, never Bifrost's row down further. The conflation this section corrects is one the vendor's own engine benefits from, which is the reason to correct it in public rather than leave the cold chart to be read as a steady-state ranking.

These artifacts are auxiliary evidence, outside the freeze. They are retained under reports/raw/warm-latency/ — the batch series, the environment stamp, and the per-case evidence each batch produced, so that a reader can check the batch did the real work rather than less of it — but freeze/v1 does not digest them, and this amendment does not extend it to. A warm number carries the pinned amendment commit 80d4f01bb189…'s immutability for its bytes and no stronger guarantee, exactly as the cold timing sidecars do. No warm run wrote a normalized report, produced an outcome, or touched a scored population; validate-reports and the freeze manifest never read this directory.

Estimated per-invocation overhead

The warm marginal above answers what one more case costs a process already running, and only one adapter's released CLI let it be measured. This section answers a different question that every adapter can be asked: what does one invocation cost before it has anything to find? It is published under Amendment A24, preregistered — estimator, bias, tolerance, fixtures and presentation — before the first estimate existed.

The method, in one sentence: run one complete adapter invocation — same pipeline, same committed policy, rule or query, same flags, same subprocess shape, both subprocesses where the adapter has two — over a trivial no-flow fixture that declares the benchmark's own source and sink endpoints and never connects them, and take the runner's wall-clock around it. The fixture is generated into a scratch workspace before the process is spawned, exactly as every other fixture is, and is retained beside the measurement. Nothing is added to cases/: no population, denominator or freeze sees these files, and they are not cases.

What the number is biased by, in both directions. A trivial file is still parsed, still extracted, still queried, so the measurement is fixed overhead plus that near-zero analysis: it is an upper bound on start-up and warm-up, and the true fixed cost is at or below it. And it is a cold, single-shot execution — no warm JIT, no primed cache — which is exactly the posture the cold rows above are measured in, and exactly not the posture of a resident deployment, where both this figure and those rows would fall. It is an estimate, labelled one everywhere it appears, and it is never subtracted from a cold number, never substituted for one, and never used to order a row.

Adapter	Fixture language	Estimated overhead (range over every repeat)	Width	Every repeat	Cold median, same kernel	Low end as a share of cold	Load observed	Marked on the chart?
Bifrost	python	83 ms – 206 ms	123 ms	206 ms, 83 ms, 85 ms	86 ms	97%	1.6	yes
CodeQL	ruby	3.65 s – 53.8 s	50.2 s	53.8 s, 3.73 s, 3.65 s	107.9 s	3%	1.9 to 2.9	no — below 25%
FlowDroid	java	584 ms – 615 ms	31 ms	615 ms, 587 ms, 584 ms	671 ms	87%	2.4 to 2.5	yes
Infer	c	261 ms – 1.02 s	760 ms	1.02 s, 261 ms, 262 ms	245 ms	107%	1.9	yes
Joern	java	7.52 s – 8.39 s	876 ms	8.39 s, 7.52 s, 7.62 s	7.89 s	95%	1.8 to 2.1	yes
Joern	php	3.54 s – 4.95 s	1.41 s	4.95 s, 3.59 s, 3.54 s	3.84 s	92%	2.6 to 2.6	yes
OpenTaint	kotlin	3.92 s – 5.26 s	1.34 s	3.92 s, 4.80 s, 5.26 s	4.01 s	98%	1.6 to 11.6	yes
Pysa	python	2.45 s – 2.49 s	36 ms	2.49 s, 2.45 s, 2.46 s	2.59 s	95%	2.8 to 3.2	yes
Semgrep CE	kotlin	772 ms – 968 ms	196 ms	968 ms, 772 ms, 773 ms	795 ms	97%	1.1	yes

The published figure is a range, and its width is the measurement's precision. Each estimate is repeated a fixed number of times — the count is a constant in the runner's source, not a per-run choice — every repeat is retained and printed above, and what is published is the range those repeats span. Never a mean, never a chosen repeat, and never withheld because the repeats disagreed: a disagreement widens the range, which is the honest consequence of it, and there is no agreement threshold anywhere in this machinery to be justified or tuned — a unit test asserts that no such constant exists. This is the same convention the warm-marginal figures publish under, shared rather than re-derived. The build re-derives each range from the retained repeats and fails if it disagrees with what the runner wrote.

The last column is what decides a chart mark, and the threshold is preregistered. A row carries a dashed span only where its range starts at or above 25% of that adapter's cold median on the same kernel. The cut is relative because the chart's axis is logarithmic and its rows span two orders of magnitude, so a share of each row's own median is the same visual claim everywhere, where a fixed millisecond cut would mark every slow adapter and no fast one whatever its overhead actually was; and it reads the range's low end so that a mark can never appear on the strength of one slow repeat. An unmarked row is not an unmeasured one — every value is in the table above.

The load column is the machine's own condition, published rather than summarized as the word "quiet". Each repeat samples the one-minute load average immediately before its subprocess is spawned, and the column is the range those samples span, so a figure taken on a busy machine can be discounted instead of taken on trust. Two disciplines stand behind those numbers: no other analyzer under measurement competes for the machine, and — because nine heavy analyzers run back to back drive the load up by themselves — each adapter's measurement waits for the load to settle before it begins, so what a row records is the machine's state and not its own position in the sequence.

Joern is the one adapter where three figures can be set side by side, and they are three different things. Its estimated per-invocation overhead on the Java kernel is 7.52 s – 8.39 s — a measured upper bound, published across the range its repeats spanned. The fitted fixed cost, the intercept of the warm batch series on the same kernel, is 2.97 s – 3.61 s — not measured at all, but inferred from a line through five batches. And its measured warm marginal, the cost of one more case in a process already running, is 4.80 s – 5.10 s. The first two are two routes to the same quantity and the third is a different quantity entirely.

Those first two routes disagree, by roughly a factor of three, and the disagreement is the informative part. Neither corrects the other and their difference is not published as a measurement, but the direction is exactly what the two constructions predict. The estimate is an upper bound that contains work the intercept excludes by construction: a whole trivial invocation — a fresh JVM, the Java front end loaded, a CPG built for the fixture, and the kernel script run over it — where the fitted intercept is what is left when a line through five batch sizes is extended back to zero cases, and every per-case cost the batch pays has been taken out of it. So the true once-per-process cost is bounded above by the estimate and approached from below by the intercept, and the gap between them is the part of "fixed cost" that neither construction can see on its own: not observable from inside one invocation, and not separable from per-case work inside a batch. That is the honest reading, and it is why this page publishes both figures rather than reconciling them into one.

Which adapters an estimate could be taken for

Each adapter is estimated in the language of its cheapest kernel arm — the core kernel whose cold median is that adapter's lowest — because that is the arm where fixed cost is the largest share of the number, and the arm whose trivial-fixture invocation is least dominated by analysis. Joern is additionally estimated on Java, so that A21's Java warm figures have a same-language estimate to be compared against; that second figure is labelled by its own language and is not the one the cheapest-arm rule selects.

Adapter	Fixture language	Verdict	Evidence
Bifrost	python its cheapest kernel arm	Measured	3 repeats — 206 ms, 83 ms, 85 ms — published as the 83 ms to 206 ms range they span.
CodeQL	ruby its cheapest kernel arm	Measured	3 repeats — 53.8 s, 3.73 s, 3.65 s — published as the 3.65 s to 53.8 s range they span.
FlowDroid	java its cheapest kernel arm	Measured	3 repeats — 615 ms, 587 ms, 584 ms — published as the 584 ms to 615 ms range they span.
Infer	c its cheapest kernel arm	Measured	3 repeats — 1.02 s, 261 ms, 262 ms — published as the 261 ms to 1.02 s range they span.
Joern	php its cheapest kernel arm	Measured	3 repeats — 4.95 s, 3.59 s, 3.54 s — published as the 3.54 s to 4.95 s range they span.
OpenTaint	kotlin its cheapest kernel arm	Measured	3 repeats — 3.92 s, 4.80 s, 5.26 s — published as the 3.92 s to 5.26 s range they span.
Pysa	python its only kernel arm	Measured	3 repeats — 2.49 s, 2.45 s, 2.46 s — published as the 2.45 s to 2.49 s range they span.
Semgrep CE	kotlin its cheapest kernel arm	Measured	3 repeats — 968 ms, 772 ms, 773 ms — published as the 772 ms to 968 ms range they span.
Joern	java not its cheapest arm — measured so that A21's Java warm figures have a same-language estimate to be compared against	Measured	3 repeats — 8.39 s, 7.52 s, 7.62 s — published as the 7.52 s to 8.39 s range they span.

There is exactly one kind of missing figure here, and it is a fact about a machine.Environment means the pinned distribution is not installed where the estimator ran, so the invocation was never attempted — explicitly not a statement about the adapter's released CLI, unlike A15's capability declines above, and resolved by running the same committed command where the distribution is installed. There is no second kind: because the figure is a range over every repeat, noisy repeats produce a wide range rather than a withheld number, so nothing is ever missing for having been measured badly. Nothing is filled in by inference either — no estimate here is derived from an adapter's runtime, its architecture, another adapter's estimate, or the same adapter's estimate in another language.

These artifacts are auxiliary evidence, outside the freeze, retained under reports/raw/invocation-overhead/: every repeat with its phase split and the one-minute load average it was taken under, the trivial fixture itself with its digest, the resolved configuration where the adapter's is per-case, the published range, and the environment stamp. freeze/v1 does not digest them, exactly as it does not digest the warm artifacts or the cold timing sidecars, so an estimate carries the pinned amendment commit 80d4f01bb189…'s immutability for its bytes and no stronger guarantee. No estimator run wrote a normalized report, produced an outcome, or touched a scored population.

Per-slice medians

The tier's declared unit of aggregation: one row per slice — adapter × language × population — because a median over a whole adapter mixes languages whose fixtures differ in size and whose front ends differ in cost. Each row is one bound report. Rows are not ranked and columns are not compared across adapters with different granularity. One disclosure per adapter, for the same reason: the rows inside one of them are the rows that share a granularity.

Bifrost — 20 slices

Show the data table — 20 slices for Bifrost

Slice	Language	Profile	Timed / bound	Median	Q1	Q3
bifrost-smoke	13 languages	benchmark-controlled	117 / 118	98 ms	89 ms	5.21 s
bifrost-c-kernel	c	benchmark-controlled	58 / 58	159 ms	136 ms	187 ms
bifrost-cpp-kernel	cpp	benchmark-controlled	68 / 68	104 ms	98 ms	113 ms
bifrost-csharp-kernel	csharp	benchmark-controlled	70 / 70	98 ms	93 ms	105 ms
bifrost-go-kernel	go	benchmark-controlled	70 / 70	137 ms	133 ms	146 ms
bifrost-java-kernel	java	benchmark-controlled	70 / 70	5.74 s	5.52 s	6.31 s
bifrost-java-modeling	java	benchmark-controlled	8 / 24	5.44 s	5.38 s	5.56 s
bifrost-java-native	java	tool-native	0 / 12	n/a	n/a	n/a
bifrost-javascript-kernel	javascript	benchmark-controlled	70 / 70	147 ms	105 ms	177 ms
bifrost-javascript-modeling	javascript	benchmark-controlled	8 / 24	88 ms	84 ms	94 ms
bifrost-javascript-native	javascript	tool-native	0 / 12	n/a	n/a	n/a
bifrost-kotlin-kernel	kotlin	benchmark-controlled	70 / 70	5.35 s	5.23 s	5.51 s
bifrost-php-kernel	php	benchmark-controlled	70 / 70	91 ms	87 ms	103 ms
bifrost-python-kernel	python	benchmark-controlled	70 / 70	86 ms	83 ms	98 ms
bifrost-python-modeling	python	benchmark-controlled	8 / 24	195 ms	165 ms	267 ms
bifrost-python-native	python	tool-native	0 / 12	n/a	n/a	n/a
bifrost-ruby-kernel	ruby	benchmark-controlled	70 / 70	96 ms	91 ms	103 ms
bifrost-rust-kernel	rust	benchmark-controlled	64 / 64	97 ms	91 ms	105 ms
bifrost-scala-kernel	scala	benchmark-controlled	70 / 70	5.29 s	5.25 s	5.35 s
bifrost-typescript-kernel	typescript	benchmark-controlled	70 / 70	92 ms	88 ms	102 ms

FlowDroid — 4 slices

Show the data table — 4 slices for FlowDroid

Slice	Language	Profile	Timed / bound	Median	Q1	Q3	total median	compile median	dex median	analyze median
flowdroid-java-kernel	java	benchmark-controlled	70 / 70	671 ms	655 ms	680 ms	n/a	n/a	n/a	n/a
flowdroid-java-modeling	java	benchmark-controlled	14 / 24	517 ms	510 ms	524 ms	n/a	454 ms	297 ms	517 ms
flowdroid-java-native	java	tool-native	0 / 12	n/a	n/a	n/a	n/a	n/a	n/a	n/a
flowdroid-kotlin-kernel	kotlin	benchmark-controlled	70 / 70	860 ms	843 ms	876 ms	n/a	n/a	n/a	n/a

Semgrep CE — 17 slices

Show the data table — 17 slices for Semgrep CE

Slice	Language	Profile	Timed / bound	Median	Q1	Q3
semgrep-c-kernel	c	benchmark-controlled	14 / 56	791 ms	788 ms	798 ms
semgrep-cpp-kernel	cpp	benchmark-controlled	14 / 68	802 ms	795 ms	811 ms
semgrep-go-kernel	go	benchmark-controlled	14 / 70	908 ms	896 ms	916 ms
semgrep-java-kernel	java	benchmark-controlled	14 / 70	814 ms	808 ms	818 ms
semgrep-java-modeling	java	benchmark-controlled	10 / 24	795 ms	794 ms	803 ms
semgrep-java-native	java	tool-native	0 / 12	n/a	n/a	n/a
semgrep-javascript-kernel	javascript	benchmark-controlled	14 / 70	882 ms	844 ms	945 ms
semgrep-javascript-modeling	javascript	benchmark-controlled	10 / 24	794 ms	792 ms	807 ms
semgrep-javascript-native	javascript	tool-native	0 / 12	n/a	n/a	n/a
semgrep-kotlin-kernel	kotlin	benchmark-controlled	14 / 70	795 ms	790 ms	799 ms
semgrep-php-kernel	php	benchmark-controlled	14 / 70	774 ms	771 ms	779 ms
semgrep-python-kernel	python	benchmark-controlled	14 / 70	813 ms	806 ms	823 ms
semgrep-python-modeling	python	benchmark-controlled	10 / 24	844 ms	828 ms	921 ms
semgrep-python-native	python	tool-native	12 / 12	2.59 s	2.51 s	2.61 s
semgrep-ruby-kernel	ruby	benchmark-controlled	14 / 70	770 ms	768 ms	776 ms
semgrep-rust-kernel	rust	benchmark-controlled	14 / 62	777 ms	773 ms	786 ms
semgrep-typescript-kernel	typescript	benchmark-controlled	14 / 70	774 ms	771 ms	784 ms

Infer — 5 slices

Show the data table — 5 slices for Infer

Slice	Language	Profile	Timed / bound	Median	Q1	Q3	capture median	analyze median
infer-c-kernel	c	benchmark-controlled	56 / 56	245 ms	242 ms	255 ms	61 ms	183 ms
infer-cpp-kernel	cpp	benchmark-controlled	68 / 68	244 ms	241 ms	1.08 s	62 ms	181 ms
infer-java-kernel	java	benchmark-controlled	70 / 70	3.66 s	3.61 s	3.69 s	3.47 s	184 ms
infer-java-modeling	java	benchmark-controlled	10 / 24	3.54 s	3.52 s	3.58 s	3.36 s	184 ms
infer-java-native	java	tool-native	0 / 12	n/a	n/a	n/a	n/a	n/a

Pysa — 3 slices

Show the data table — 3 slices for Pysa

Slice	Language	Profile	Timed / bound	Median	Q1	Q3
pysa-python-kernel	python	benchmark-controlled	70 / 70	2.59 s	2.57 s	2.64 s
pysa-python-modeling	python	benchmark-controlled	20 / 24	4.51 s	4.09 s	4.74 s
pysa-python-native	python	tool-native	12 / 12	2.50 s	2.49 s	2.54 s

OpenTaint — 4 slices

Show the data table — 4 slices for OpenTaint

Slice	Language	Profile	Timed / bound	Median	Q1	Q3
opentaint-java-kernel	java	benchmark-controlled	70 / 70	4.02 s	3.97 s	4.11 s
opentaint-java-modeling	java	benchmark-controlled	12 / 24	3.69 s	3.62 s	3.76 s
opentaint-java-native	java	tool-native	0 / 12	n/a	n/a	n/a
opentaint-kotlin-kernel	kotlin	benchmark-controlled	70 / 70	4.01 s	3.90 s	4.14 s

Joern — 12 slices

Show the data table — 12 slices for Joern

Slice	Language	Profile	Timed / bound	Median	Q1	Q3
joern-java-kernel	java	benchmark-controlled	70 / 70	7.89 s	7.78 s	7.95 s
joern-java-modeling	java	benchmark-controlled	16 / 24	7.66 s	7.62 s	7.70 s
joern-java-native	java	tool-native	0 / 12	n/a	n/a	n/a
joern-javascript-kernel	javascript	benchmark-controlled	70 / 70	3.88 s	3.85 s	3.90 s
joern-javascript-modeling	javascript	benchmark-controlled	16 / 24	3.96 s	3.90 s	3.99 s
joern-javascript-native	javascript	tool-native	0 / 12	n/a	n/a	n/a
joern-php-kernel	php	benchmark-controlled	70 / 70	3.84 s	3.80 s	3.87 s
joern-python-kernel	python	benchmark-controlled	70 / 70	4.69 s	4.26 s	5.67 s
joern-python-modeling	python	benchmark-controlled	16 / 24	3.72 s	3.67 s	3.80 s
joern-python-native	python	tool-native	0 / 12	n/a	n/a	n/a
joern-ruby-kernel	ruby	benchmark-controlled	70 / 70	6.73 s	6.48 s	7.37 s
joern-rust-kernel	rust	benchmark-controlled	62 / 62	5.57 s	5.42 s	6.55 s

CodeQL — 17 slices

Show the data table — 17 slices for CodeQL

Slice	Language	Profile	Timed / bound	Median	Q1	Q3	database-create median	database-analyze median
codeql-c-kernel	c	benchmark-controlled	58 / 58	54.5 s	51.7 s	58.2 s	1.30 s	52.8 s
codeql-cpp-kernel	cpp	benchmark-controlled	68 / 68	54.1 s	52.3 s	57.1 s	1.38 s	52.4 s
codeql-csharp-kernel	csharp	benchmark-controlled	70 / 70	57.1 s	56.1 s	58.7 s	12.1 s	45.1 s
codeql-go-kernel	go	benchmark-controlled	70 / 70	27.6 s	27.4 s	28.0 s	3.01 s	24.6 s
codeql-java-kernel	java	benchmark-controlled	70 / 70	57.8 s	56.9 s	58.5 s	5.28 s	52.4 s
codeql-java-modeling	java	benchmark-controlled	24 / 24	35.9 s	35.7 s	36.3 s	5.29 s	30.6 s
codeql-java-native	java	tool-native	12 / 12	14.6 s	14.5 s	14.9 s	5.23 s	9.37 s
codeql-javascript-kernel	javascript	benchmark-controlled	70 / 70	65.5 s	65.2 s	66.1 s	2.99 s	62.4 s
codeql-javascript-modeling	javascript	benchmark-controlled	24 / 24	50.6 s	50.1 s	92.1 s	3.23 s	47.5 s
codeql-javascript-native	javascript	tool-native	12 / 12	11.4 s	11.4 s	11.5 s	2.95 s	8.48 s
codeql-kotlin-kernel	kotlin	benchmark-controlled	70 / 70	61.0 s	60.8 s	61.5 s	9.55 s	51.4 s
codeql-python-kernel	python	benchmark-controlled	70 / 70	29.7 s	29.5 s	29.8 s	1.80 s	27.8 s
codeql-python-modeling	python	benchmark-controlled	24 / 24	22.0 s	22.0 s	22.2 s	1.82 s	20.2 s
codeql-python-native	python	tool-native	12 / 12	6.19 s	6.15 s	6.22 s	1.79 s	4.41 s
codeql-ruby-kernel	ruby	benchmark-controlled	70 / 70	107.9 s	101.9 s	124.7 s	1.23 s	106.4 s
codeql-rust-kernel	rust	benchmark-controlled	64 / 64	93.9 s	93.2 s	94.6 s	18.7 s	75.0 s
codeql-typescript-kernel	typescript	benchmark-controlled	70 / 70	65.4 s	65.1 s	65.6 s	3.14 s	62.2 s

Cases with no latency number

Of the case results in this bound latency corpus, 3157 invoked an analyzer and carry a timing sidecar. 879 do not, and every one of them is accounted for:

Outcome of the untimed case	Cases
unsupported	879

An unsupported case is decided from case metadata before the analyzer is invoked, so there is no subprocess and there is nothing to time. Those cases are absent from every distribution on this page, not entered as zero — entering them as zero would make the adapters that decline the most look the fastest, which is exactly backwards. Every case that did invoke an analyzer is timed; the timed and untimed columns account for the whole bound population with no remainder.

Provenance and freeze guarantees

Every cold number above was derived at build time from v0.8.0's archived latency-evidence bundle. The bundle was generated from that release commit's retained timing sidecars, and its population comes from the results model generated by freeze manifest 582b25896487…. Snapshot pages select this bundle explicitly: advancing the current correctness freeze cannot add a case, replace a timing, or relabel an analyzer on a historical page. The warm and invocation-overhead amendments are likewise read from a checked-in auxiliary bundle pinned to commit 80d4f01bb189d530849f9ceb5d775680fcedfb52; they never read the mutable working-tree artifact directories.

One limit, stated rather than left to be assumed. The freeze/v1 manifest binds one raw-evidence digest per result; the timing sidecar and the environment stamp are additive files beside that evidence and are not themselves digested by the manifest. So a number on this page carries the freeze's guarantee that the run it belongs to is bound and byte-verified, and the release commit plus the checked-in archive's own Git history preserve the timing bytes selected for rendering. It does not carry a freeze-manifest digest over those timing bytes. Extending the manifest to bind them is a freeze/v2 question, and it is named here rather than smuggled in under a schema version that does not describe it.