Designed in 2021, asked back in 2025: a Data & AI platform that lasted

The challenge

The organisation’s competitive intelligence platform was a set of monolithic applications, built and run by teams funded from three different business areas, each with its own priorities. It needed to be modernised without stopping the work that depended on it, and to become something new products could be built on quickly and safely.

2021–2022: the design

I joined as principal programme and platform architect, and technical lead, one of a three-person leadership team with the Director and the Programme Manager.

• Led five distributed teams of 20–40 people, balancing the priorities of the three funding business areas.
• Moved the monolithic applications onto a shared Kubernetes environment, and worked with the platform teams to fix the gaps I found in it.
• Migrated user security to OAuth2 / OIDC on Azure AD, and replaced custom legacy code with OpenSearch and DynamoDB.
• Designed the target AWS platform blueprint, built on fully managed services to cut the support burden.

After I left, the client moved the platform onto that design.

2025–2026: the platform

The client brought me back as technical lead to turn it into a multi-tenant Data & AI product platform.

• Shared services: foundation services (security, networking, runtime and integration), delivery services (development standards, release and quality management) and data & AI services, all managed as infrastructure as code with Terraform and Python. It runs on ECS, API Gateway and Elastic Load Balancing, with multi-tenant DocumentDB, OpenSearch, Aurora PostgreSQL and S3.
• A standard product blueprint: I introduced a product operating model in which every product team starts with its own GitHub repository, Terraform, GitHub Actions pipelines with SonarQube quality gates, and Datadog observability. About 25 products now run on the platform, and a new one is bootstrapped in under a working day instead of weeks.
• The RAG data path: I built the full path that generative AI applications depend on, from ETL ingestion, parsing, chunking and embeddings to hybrid search on OpenSearch, with filtering, and the retrieval APIs the LLM applications call.
• Technical leadership: architecture and technical direction for the product teams building on the platform.

The outcome

A design that lasted. The client built on the 2021 blueprint without me, then asked me back to take it further. Today it’s a governed, multi-tenant platform where product teams start from a working, secure, observable skeleton, and where the data and retrieval layer for generative AI is already in place.