Engineers reviewing a system diagram on a whiteboard

Architecture decision review · fictional case

Streaming event ingestion at Meridian

From nightly batch to a six-hour freshness target

Decision requested: approve phase 1 · 2026

Context

2.4B events/day, six-hour freshness, and a batch window that no longer fits

Volume

2.4B events/day, 3.1× year over year.

Freshness

Product needs six hours; batch gives 24.

Pressure

The nightly window now overlaps peak ingest.

Requirements

What any option must satisfy

Correctness

Exactly-once semantics at the sink; idempotent replay.

Replay

90-day replay for backfill and incident recovery.

Cost

≤ 1.4× current run rate at 3× volume.

Migration

No customer-visible cutover; rollback in one hour.

Options

Three credible paths

Option	Shape	Strength	Weakness
A · Batch + micro-batch	Keep batch, add 15-min micro-batches	Lowest migration risk	Cannot reach 6h freshness at peak
B · Kafka + Flink	Streaming ingest, managed Kafka, Flink jobs	Meets freshness and replay	Operational load; new on-call surface
C · Managed ingestion	Vendor-managed pipeline	Lowest ops	Cost and lock-in at 3× volume

Trade-offs

B wins on the requirements; the cost is operational

Criterion	A	B	C
Six-hour freshness	No	Yes	Yes
90-day replay	Partial	Yes	Vendor-dependent
Cost at 3×	Low	Medium	High
Operational load	Low	High	Low
Migration risk	Low	Medium	Medium

Recommendation

Adopt B, phased, with batch retained for backfill.

Managed Kafka for ingest, Flink for exactly-once aggregation, and a one-hour rollback to the batch path.

Migration

Four phases, one rollback point

P1Dual-write to Kafka and batch; compare counts.

P2Shadow-read from streaming; reconcile 30 days.

P3Cut over product reads; batch becomes backfill.

P4Decommission batch ingest after two clean weeks.

Risks

What could break, and the mitigation

Ordering

Partition key change can reorder events; freeze the key at P2.

Schema evolution

Registry compatibility mode; reject breaking changes in CI.

Cost spike

Per-tenant quotas; weekly cost review during P1–P3.

On-call load

Two engineers trained before cutover; runbook and game day.

Decision

Approve phase 1

Success criteria: 6-hour freshness at 3× volume, 99.95% sink availability, one-hour rollback proven in a game day.

Next review: end of phase 2 · Q&A follows

Appendix

Backup

Current batch topology and failure modes
Kafka/Flink sizing and cost model
Exactly-once proof sketch and replay test
Migration runbook and game-day plan