Data, Search & Applied AI

Generative AI is only as good as the data and retrieval underneath it, and that’s the part I build. It means ingestion pipelines that parse, chunk and embed documents; search indexes that combine keyword and vector search; filtering so people only see what they’re allowed to; and the retrieval APIs that AI applications call. It’s data and search engineering, held to production standards.

It also covers data platforms more broadly: operational and analytical stores, event and change-stream pipelines, and data lakes.

Useful, not fashionable

I add AI where it improves a real system, not to have a demo. That means working out security, cost, evaluation and what happens when the model is wrong before the first prompt is written, not after.

Where I’ve done it

• A global pharmaceutical company: built the full RAG data path for a multi-tenant Data & AI platform, from ETL ingestion, chunking and embeddings to hybrid search on OpenSearch and the retrieval APIs the LLM applications call.
• A global media and publishing company: made the case to the board for an AI and data science spin-off, as its designated CTO, and architected the data lake and analytics platform behind it on AWS (S3, Glue, Redshift, EMR, Databricks, Dataiku).
• Trainline: mentored the e-commerce team on NoSQL data modelling with DynamoDB and OpenSearch.
• Envizage: as CTO, designed the data model and the Monte Carlo analytics engine at the heart of a wealth-management product.