> ## Documentation Index
> Fetch the complete documentation index at: https://docs.databunker.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Performance & benchmarks

> Measured Databunker Pro throughput, detokenisation latency, and storage at 10M, 50M, and 100M records on dedicated AWS RDS PostgreSQL — with a sizing guide for choosing database and application instances by record count.

This page gives architects the numbers needed to size a Databunker Pro deployment. Every figure below is **measured** — we loaded a 120-field user-profile vault to **10, 50, and 100 million records** against a **dedicated AWS RDS PostgreSQL** database and recorded throughput, latency, storage, and resource utilisation. The methodology is documented so you can reproduce it with your own payloads.

## At a glance

* **Storage is linear and predictable: \~5.7 KB per encrypted 120-field record.** Multiply by your record count to size the volume (10 M ≈ 59 GB, 100 M ≈ 585 GB).
* **Index size scales with how many fields you make searchable — plan for *your* field set.** This benchmark indexed **only `email`** (plus the always-present `token` and retention indexes) → \~139 bytes/record. **Each additional searchable field you index (`phone`, `login`, `custom`) adds another index: budget roughly +40–50 bytes/record and more write CPU per field.** Size the database RAM to cache *all* your indexes (with email only, 100 M ≈ 14 GB; cache-hit ≥ 97%).
* **Detokenisation is fast: \~14 ms** per single-record `UserGet`, regardless of vault size.
* **Write throughput depends on scale.** At small scale it is limited by the **application tier** and one instance does \~5,800 records/sec. At tens of millions of rows it is limited by **database CPU** and plateaus at **\~12,000 records/sec** — so you size the **database** for large vaults, not the app tier.
* **Rule of thumb:** pick your database and application instances from the [sizing guide](#how-to-size-your-deployment) by record count. For most workloads a single Databunker Pro instance is already more than enough.

## What we measured

Three runs, same workload, increasing scale. In every run Databunker Pro (stateless, in Docker) ran on EC2 with the load generator co-located, writing to a **dedicated RDS PostgreSQL 16** instance in the same VPC (`eu-central-1`). Multiple app instances share one database and one wrapping key.

| Run   | Records | Application tier  | Dedicated RDS database              | Write result                        |
| ----- | ------- | ----------------- | ----------------------------------- | ----------------------------------- |
| **A** | 10 M    | 1 × `m6i.2xlarge` | `db.m6i.2xlarge` — 8 vCPU / 32 GB   | \~5,800 rec/s — **app-bound**       |
| **B** | 50 M    | 4 × `m6i.2xlarge` | `db.m6i.4xlarge` — 16 vCPU / 64 GB  | \~11,800 rec/s — **database-bound** |
| **C** | 100 M   | 5 × `m6i.2xlarge` | `db.m6i.8xlarge` — 32 vCPU / 128 GB | \~11,900 rec/s — **database-bound** |

**Common workload**

* **Profile:** 120 fields per record (mixed PII + custom application fields).
* **Write path:** `UserCreateBulk` at **4,000 records/request, 8 concurrent workers per instance**.
* **Storage & encryption:** gp3 volumes; **AES-256 at rest, TLS in transit** (defaults).
* **Indexes per record:** every user always has a unique **token** index (the detokenisation lookup) and a **retention** (`finaltime`) index. Search fields add one salted-hash index each — this benchmark sets only `email`, giving **three indexes/record** (\~139 bytes total). Databunker Pro can also index `phone`, `login`, and `custom` (a hashed index is written only when the field is present); setting those adds per-insert database CPU and lowers sustained write throughput at scale. `name` and the other profile fields are encrypted but not indexed.
* **Placement:** loaders co-located with Databunker Pro (no client-side network noise); the only network hop measured is **App → RDS**.

<Note>
  Redis is used only for transient session state and is **not** exercised by bulk writes; a production deployment points all instances at a shared Redis cluster. A separate 5 M run with PostgreSQL **co-located** on the app host managed only \~1,700 rec/s — moving to a **dedicated** database roughly tripled write throughput, which is why every run here uses dedicated RDS.
</Note>

## Results

| Records   | DB size   | Per-record | Indexes | Sustained bulk write             | Detok p50 | Bottleneck               |
| --------- | --------- | ---------- | ------- | -------------------------------- | --------- | ------------------------ |
| **10 M**  | 58.5 GB   | \~5.7 KB   | 1.4 GB  | **\~5,800 rec/s** (1 instance)   | \~14 ms   | app tier (RDS \~20% CPU) |
| **50 M**  | 292 GB    | \~5.7 KB   | 6.9 GB  | **\~11,800 rec/s** (4 instances) | —         | database (RDS \~80% CPU) |
| **100 M** | 584 GB    | \~5.7 KB   | 13.9 GB | **\~11,900 rec/s** (5 instances) | —         | database (RDS \~82% CPU) |
| 150 M     | \~875 GB  | \~5.7 KB   | \~21 GB | database-bound *(projected)*     | —         | database                 |
| 200 M     | \~1.17 TB | \~5.7 KB   | \~28 GB | database-bound *(projected)*     | —         | database                 |

All runs completed with **zero failed batches**. Per-record storage is essentially constant across scale — storage planning is simple linear math.

## The key insight: the bottleneck moves with scale

This is the single most important thing to understand when sizing Databunker Pro.

**At small scale, the application tier is the limit.** Encrypting and hash-indexing each record is CPU work done by Databunker Pro. At 10 M records one instance used \~3.2 of its 8 cores while the database sat at just \~20% CPU. Here, **adding Databunker Pro instances raises throughput** — the stateless app tier scales cleanly (we verified near-linear scaling from 1 to 6 instances on a small vault), and the database has plenty of headroom.

**At large scale, the database CPU is the limit.** As the table grows, every insert costs the database more (deeper B-tree indexes, page splits, WAL, autovacuum). By tens of millions of rows this dominates:

|                          | 10 M (1 instance)         | 50 M (4 instances)         | 100 M (5 instances)        |
| ------------------------ | ------------------------- | -------------------------- | -------------------------- |
| RDS instance             | `db.m6i.2xlarge` (8 vCPU) | `db.m6i.4xlarge` (16 vCPU) | `db.m6i.8xlarge` (32 vCPU) |
| **RDS CPU (avg / peak)** | **20% / 33%**             | **80% / 95%**              | **82% / 95%**              |
| Aggregate write          | \~5,800 rec/s             | \~11,800 rec/s             | \~11,900 rec/s             |
| App CPU (per instance)   | \~3.2 cores               | saturating DB              | saturating DB              |

**Throughput plateaus at \~12,000 records/sec.** Note that 50 M and 100 M delivered almost the same throughput **even though the 100 M run used twice the database vCPU** (32 vs 16). Doubling the database didn't help because the per-insert cost had also doubled with the larger index. To push past this plateau, database vCPU has to grow *faster* than the table.

**What this means for you:**

* **Steady-state and real-time workloads** are nowhere near these limits — one Databunker Pro instance on a modest database handles them comfortably.
* **Bulk backfills of very large vaults are database-bound and take hours** (100 M ≈ \~2.3 h here). Size the **database** for the row count, and don't expect more app instances to speed it up once RDS is CPU-saturated.

## How to size your deployment

Pick the row you're targeting. Storage scales at \~5.7 KB/record. Index RAM depends on **how many fields you index** — the numbers below assume the benchmark's set (`token` + retention + **one** search index, `email`) ≈ \~139 bytes/record; **add \~40–50 bytes/record for each extra searchable field** you index (`phone`, `login`, `custom`). The database instance is the throughput limiter at scale, so it's sized for **vCPU (write speed) and RAM (index cache)**. Application nodes are stateless `m6i.2xlarge` (Databunker Pro uses \~3 vCPU at full load) — add them to speed up bulk loads only until the database reaches \~70–80% CPU.

| User records  | RDS database instance                     | RDS storage (gp3) | Index size, **email only** (vs instance RAM) | App instances       | Confidence       |
| ------------- | ----------------------------------------- | ----------------- | -------------------------------------------- | ------------------- | ---------------- |
| **≤ 10 M**    | `db.m6i.2xlarge` — 8 vCPU / 32 GB         | 150 GB            | \~2 GB (of 32)                               | 1 × `m6i.2xlarge`   | measured         |
| **10–50 M**   | `db.m6i.4xlarge` — 16 vCPU / 64 GB        | 500 GB            | \~7 GB (of 64)                               | 2–4 × `m6i.2xlarge` | measured @ 50 M  |
| **50–100 M**  | `db.m6i.8xlarge` — 32 vCPU / 128 GB       | 1 TB              | \~14 GB (of 128)                             | 4–5 × `m6i.2xlarge` | measured @ 100 M |
| **100–150 M** | `db.r6i.8xlarge` — 32 vCPU / 256 GB       | 1.1 TB            | \~21 GB (of 256)                             | 4–5 × `m6i.2xlarge` | projected        |
| **150–200 M** | `db.r6i.8xlarge` or **Aurora PostgreSQL** | 1.3–1.5 TB        | \~28 GB                                      | 4–6 × `m6i.2xlarge` | projected        |

The **index column is for the benchmark's index set only** (`token` + retention + the single `email` search index). Note the large headroom — 14 GB of indexes on a 128 GB instance — because these instances are chosen for **write vCPU**, not index RAM, so they hold several more indexes comfortably. **If you index more fields, recompute:** \~139 bytes/record (email only) **+ \~40–50 bytes/record per additional searchable field** (`phone`, `login`, `custom`), × your record count, and confirm it still fits in RAM. Storage and write vCPU are unaffected by this — only index RAM and per-insert write cost grow.

**Three sizing rules**

1. **Storage** = 5.7 KB × records, plus **30–40% headroom** for WAL, bloat, and backups.
2. **RAM** must exceed your **total index size + hot rows** so lookups stay cached. Index storage = `token` + retention + **one per searchable field**. With only `email` indexed this was \~139 bytes/record (100 M ≈ 14 GB; cache-hit ≥ 97%); **add \~40–50 bytes/record for every extra field you index** (`phone`, `login`, `custom`). Multiply by your record count.
3. **Database vCPU** is the sustained-write limiter at scale — grow it (or move to **Aurora** for storage auto-scaling and read replicas) before adding app instances once RDS passes \~70% CPU.

## Reads: detokenisation latency

Detokenisation (`UserGet`) is a hashed-index point lookup by token, measured against the fully-loaded 10 M vault:

| Metric                       | Value                                                       |
| ---------------------------- | ----------------------------------------------------------- |
| **Latency p50 / p95 / p99**  | **\~14 ms / \~15 ms / \~16 ms** per single-record `UserGet` |
| Read throughput (1 instance) | \~660 reads/sec synchronous (16 concurrent clients)         |

Latency is dominated by the round trip (HTTP, auth, index lookup, decryption), **not** index depth — the index cache-hit ratio stayed ≥ 97% at every scale, so lookups are served from memory. Read **throughput** scales horizontally: add Databunker Pro instances (and, for very large vaults, database read replicas) to serve more concurrent lookups.

## Storage footprint

Encrypted profile blobs dominate the footprint; indexes are comparatively small (which is why a modest amount of RAM keeps them cached).

| Component                      | 10 M    | 100 M    | Per record   |
| ------------------------------ | ------- | -------- | ------------ |
| Total database size            | 58.5 GB | 584 GB   | **\~5.7 KB** |
| Encrypted profile data (TOAST) | \~55 GB | \~550 GB | \~5.5 KB     |
| Heap (`users` table)           | 1.8 GB  | \~9 GB   | \~90 B       |
| **Indexes — email only**       | 1.4 GB  | 13.9 GB  | **\~139 B**  |

The two figures that scale with your design:

* **Encrypted profile data** grows with **payload size** — a bigger/smaller profile than our 120-field one moves the \~5.5 KB/record directly.
* **Indexes** grow with **how many fields you make searchable.** The \~139 B/record above is the benchmark's set only — `token` (always) + retention + the single `email` search index. **Add \~40–50 B/record for each additional indexed field** (`phone`, `login`, `custom`). Storage total, heap, and per-record profile size are unaffected by indexing choices — only the index line (and per-insert write CPU) grows.

## Write latency: bulk vs single record

The \~0.17 ms per record at 10 M is **amortised inside a 4,000-record bulk request**. Individual `UserCreateBulk` calls take longer end-to-end (p50 \~4.5 s for a 4,000-record batch at 10 M, growing with table size), and **single-record synchronous writes** are typically 3–10 ms over a real network because each call carries its own connection, auth, and transaction overhead.

* **Backfills / batch flows:** use `UserCreateBulk`.
* **Real-time API flows:** assume single-digit-millisecond single-record writes; production throughput = `concurrency ÷ per-call latency`.

## Reproduce this benchmark

Start from the open-source Python load scripts in the [`databunkerpro-python`](https://github.com/securitybunker/databunkerpro-python) SDK repo:

* **[`bulk_user_creator.py`](https://github.com/securitybunker/databunkerpro-python/blob/main/bulk_user_creator.py)** — drives `UserCreateBulk`.
* **[`bulk_user_fetcher.py`](https://github.com/securitybunker/databunkerpro-python/blob/main/bulk_user_fetcher.py)** — drives `UserGet` for read latency.

<Steps>
  <Step title="Provision">Dedicated managed PostgreSQL (RDS / Aurora / Cloud SQL) plus a separate application host in the same VPC.</Step>
  <Step title="Match your payload">Edit the profile shape in `bulk_user_creator.py` to the JSON your application actually sends.</Step>
  <Step title="Load">Run the loader co-located with Databunker Pro, with multiple concurrent workers (and multiple instances for large loads).</Step>
  <Step title="Measure">Record sustained records/sec, p50/p95/p99 bulk latency, **database CPU and IOPS**, and **application-tier CPU** — whichever is saturated is your bottleneck.</Step>
  <Step title="Read test">Run `bulk_user_fetcher.py` against the same vault to capture `UserGet` latency and throughput.</Step>
</Steps>

Your numbers will vary with payload size, number of search indexes, access-control policy depth, and database instance/IOPS. Contact [office@databunkertech.com](mailto:office@databunkertech.com) if you'd like the professional-services team to run a benchmark against your representative payloads.

## Scope & roadmap

These runs cover the **PII vault write path** (`UserCreateBulk`) and the **detokenisation read path** (`UserGet`). They do **not** cover the secure session-storage API, format-preserving tokenisation, or agreement/legal-basis flows — those have different characteristics and are measured separately.

Coming next: 150 M–200 M measured runs (to replace the projected sizing rows), secure session-storage API, format-preserving tokenisation throughput, and a MySQL backend comparison.