Data Platform Architecture — Overview
internal engineering docs
The shape in one picture. The detailed design — capacity, phases, cost, risks and the open decisions — lives in the Platform Engineering build plan, linked at the bottom; this page does not repeat it, so there is one source and not two.
What the picture says
Every site is a full warehouse, not a shard of one. Each of the three sites runs its own Trino against its own object storage and answers from the data it holds, so a WAN cut or a single-site outage degrades that site alone (item 27).
Data converges by mirroring, not by cross-site query. Topics are mirrored between the sites and each site lands its own copy; replication is asynchronous, so sites stay independent at query time (item 28, the same platform that carries business events).
One scheduler, because lineage has a ceiling. A DAG is a dependency graph, so the scheduler is where "what breaks if this table breaks" comes from. Split the pipelines across two schedulers and the lineage graph is broken by construction — which is why the scheduler is unified rather than left per-team (item 27). Application-level scheduled jobs are out of this scope and stay where they are.
Access is decided per person, not per service account. Users sign in to the front door through SSO and that identity is carried through to Trino, so a query runs as the person who asked it and policy is evaluated against that identity (item 27).
Where the work is
| Item | Milestones |
|---|---|
| 27 · Data Platform | data platform milestones |
| 28 · Event Bus (Kafka) | event bus architecture |
The full plan
- 資料平台建置計畫 v5.14 — target architecture, three build phases, capacity, cost, risks, open decisions
- 簡報版 — the same material as a deck