10 · Data Platform
Milestones of the roadmap items in this theme
As of 2026-10-22 · today
Gantt
Milestones
| # | Milestone | Start | Target | Status | Waits for | Team | Epic | Derived |
|---|---|---|---|---|---|---|---|---|
| 27 · Data Platform | ||||||||
| 27.3 | Phase 1 — on-premises data platform ready | 2026-09-01 | 2026-11-11 | in progress | — | Platform Engineering | — | — |
| 27.4 | Phase 2 — US | 2026-11-12 | 2028-02-25 | planned | 27.3 | Platform Engineering | — | waiting 27.3 |
| 27.8 | Phase 3 — asia-east1; three-site federation complete | 2027-06-19 | 2028-12-10 | planned | 27.4 | Platform Engineering | — | waiting 27.4 |
| 27.9 | ETL scheduler live, with a scheduled-query entry like BigQuery's | 2026-11-12 | 2027-04-12 | planned | 27.3 | Platform Engineering | — | waiting 27.3 |
| 27.10 | StarRocks migrated to the data platform | 2026-11-12 | 2028-12-10 | planned | 27.3 | Platform Engineering | — | waiting 27.3 |
| 27.11 | Existing jobs moved in as scheduled queries | 2027-01-12 | 2028-02-25 | planned | 27.9 | Platform Engineering | — | waiting 27.9 |
| 27.12 | DAGs rebuilt for the moved jobs | 2027-02-01 | 2028-12-10 | planned | 27.11 | Platform Engineering | — | waiting 27.11 |
| 28 · Event Bus (Kafka) | ||||||||
| 28.1 | Event Bus proven end to end — from an HTTP publish to a queryable table | — | 2026-08-27 | done | — | Platform Engineering | — | — |
| 28.2 | Control plane in git — schema, topic and access rules as code, with CI | — | 2027-09-17 | planned | — | Platform Engineering | — | — |
| 28.3 | Backend services publish over authenticated HTTP, validated before the write | — | 2028-03-17 | planned | 28.2 | Platform Engineering | — | waiting 28.2 |
| 28.4 | Kafka at the US site stood up as a read-only replica | 2027-01-04 | 2028-07-19 | planned | 28.2 | Platform Engineering | — | waiting 28.2 |
| 28.5 | IDC convergence point live — events and change data landing in one place | — | 2028-09-17 | planned | 28.3 | Platform Engineering | — | waiting 28.3 |
| 28.6 | Mirroring established across the three sites, each landing its own copy | — | 2028-11-17 | planned | 28.4, 28.5 | Platform Engineering | — | waiting 28.4, 28.5 |
| 28.7 | Replication lag and health observable and alerted, before the first edge carries traffic | 2027-01-04 | 2028-09-17 | planned | 28.2 | Platform Engineering | — | waiting 28.2 |
| 28.8 | A site cut and its catch-up rehearsed | — | 2029-02-22 | planned | 28.6, 28.7 | Platform Engineering | — | waiting 28.6, 28.7 |
Details
27.3Phase 1 — on-premises data platform ready2026-11-11in progress · 2/8
- Fault tolerance of the existing storage cluster measured and written downowner · TBDThe existing cluster runs with two parity blocks: losing one node, or a rolling upgrade, leaves no tolerance at all.
- Storage decision settled — cost, lead time, rack, power and network confirmedowner · TBDarchitecture →
- Queryable in SIT across the existing warehouse and Icebergowner · TBD
- Authentication, group mapping and authorization engine proven end to endowner · TBDIdentity signed by the external IdP into a JWT, verified per request, authorization decided by policy-as-code beside it.
- Authorization RFC reviewed with the infrastructure and data warehouse teams; receiving team settledowner · TBD
- Column masking demonstrable end to endowner · TBDUntil this is done the RFC is explicit: do not claim sensitive columns are masked.
- Scheduler scope: data pipelines on the existing scheduler listed by owner, split from application-level jobsowner · TBD
- Production readiness signed: capacity baseline, backup and recovery rehearsed, handover with the data warehouse teamowner · TBD
27.4Phase 2 — US2028-02-25planned · 0/7
- GKE at the US site under IaC; query cluster registered to the site gatewayowner · TBDOne gateway per site: each knows only its own clusters.
- Cross-continent traffic audited and alertedowner · TBD
- Export scope settled from the query history; selected tables readable as Iceberg on GCS-USowner · TBDThe export is selective, not a full migration. Minimum-privilege service account: read and read-session only.
- Elastic scaling on preemptible capacity under a hard budget ceilingowner · TBD
- Exported cloud data under the same authorization and audit; one policy sourceowner · TBD
- Cloud pipelines run on the 27.9 scheduler against the US siteowner · TBD
- asia-east1 role and the data-residency answer asked at the start of this phaseowner · TBDA "must stay in Asia" answer raises Phase 3 ahead of the rest of this phase.
27.8Phase 3 — asia-east1; three-site federation complete2028-12-10planned · 0/3
- Query cluster on the existing GKE capacity, behind its own site gatewayowner · TBD
- Scheduler at asia-east1; one scheduler per site across all threeowner · TBD
27.9ETL scheduler live, with a scheduled-query entry like BigQuery's2027-04-12planned · 0/4
- Scheduler and dbt deployedowner · TBD
- Scheduled-query entry: a user gives SQL, a schedule and a target table; the platform generates the jobowner · TBDSo existing jobs can move as they are, without a rewrite first.
- Jobs run under the identity of the team that owns themowner · TBD
- One team has moved a job without helpowner · TBD
27.10StarRocks migrated to the data platform2028-12-10planned · 0/4
- StarRocks tables and their readers listed by ownerowner · TBD
- Tables served from Iceberg through the data platformowner · TBD
- Reports and SQL repointed, with the data warehouse teamowner · TBD
- StarRocks retired once its last reader has movedowner · TBD
27.11Existing jobs moved in as scheduled queries2028-02-25planned · 0/3
- Data jobs listed by owner — BigQuery scheduled queries and the existing scheduler; application-level jobs excludedowner · TBD
- Moved without a rewriteowner · TBD
- The old schedule switched off for each job that movedowner · TBD
27.12DAGs rebuilt for the moved jobs2028-12-10planned · 0/3
- Order by how much depends on the job, most firstowner · TBD
- Dependencies declared, so lineage comes from the DAGowner · TBDA DAG is a dependency graph: this is where "what breaks if this table breaks" is answered.
- Each rebuilt job replaces its scheduled-query formowner · TBD
28.1Event Bus proven end to end — from an HTTP publish to a queryable table2026-08-27done · 3/3
- Publish over HTTP, schema-validated, written under a verified identityowner · TBD
- Landed into Iceberg without duplicating or dropping eventsowner · TBD
- Queryable from Trino; 147 tests greenowner · TBDPolicy unit tests 52/52, plus fail-closed, delete protection and decision logging verified separately. Last verified 2026-08-27.
28.2Control plane in git — schema, topic and access rules as code, with CI2027-09-17planned · 0/3
- One file per topic as the single source of truthowner · TBDSchemas, authorization data and the landing tables are generated from it; none of the three is edited by hand.
- CI blocks a change that breaks the schema rulesowner · TBD
- Registry topology settled — one writable site, the others read-onlyowner · TBD
28.3Backend services publish over authenticated HTTP, validated before the write2028-03-17planned · 0/3
- HTTP endpoint with authentication and policy-based authorizationowner · TBD
- Brokers verify tokens directly; access rules generated from the topic filesowner · TBD
- Caller identity carried through unchanged to the brokerowner · TBDIdentity does not change hands and authorization is decided at exactly one point.
28.4Kafka at the US site stood up as a read-only replica2028-07-19planned · 0/3
- Brokers and controllers provisioned, sized from the existing site as referenceowner · TBDThe plan sizes this at four brokers plus three controllers and 28 TB of data disk, and calls it the largest hardware line of its phase.
- Site registered in the topic control plane as read-onlyowner · TBD
- Retention and capacity sized from measured event volumeowner · TBD
28.5IDC convergence point live — events and change data landing in one place2028-09-17planned · 0/3
- Kafka at the IDC established as the common landing pointowner · TBD
- Monitoring that does not rely on the broker healthcheckowner · TBDThe healthcheck runs on the internal listener as a super user and short-circuits before authorization is consulted: measured with authorization fully broken and every request denied, the broker still reported healthy.
- Free-form payloads defended by more than a control-plane lint ruleowner · TBDLoosely defined payloads lose content at landing time with no error, no dead-letter entry, and reconciliation totals that still add up. The only defence is one lint rule that can be waived, and a waiver is not a property of the cluster.
28.6Mirroring established across the three sites, each landing its own copy2028-11-17planned · 0/3
- Directed replication edges configured between the three sitesowner · TBDA bidirectional edge needs the default naming policy, or a topic mirrored back to its origin collides with itself.
- Schema registry replicated one way, from the writable site outwardowner · TBD
28.7Replication lag and health observable and alerted, before the first edge carries traffic2028-09-17planned · 0/3
- Lag and health metrics collected from every replication edgeowner · TBDReplication is asynchronous by design, so the goal is that lag is observable and alerted, never that it is zero.
- Alert thresholds agreed, wired, and ownedowner · TBD
28.8A site cut and its catch-up rehearsed2029-02-22planned · 0/3
- A site cut rehearsed end to endowner · TBD
- Recovery time and backlog drain measured, not estimatedowner · TBD
- Runbook written from what the rehearsal showedowner · TBD
Back to the Platform Roadmap overview · the direction behind these milestones is the Platform Roadmap.