Natural Language Interfaces for Self-Service Analytics on Data Lakes: Design Patterns, Governance, and Lessons from a Production Deployment

Abstract

The promise of natural-language interfaces to enterprise data is older than the LLMs that finally make it tractable, and the gap between the demo and the production deployment is larger than most organizations appreciate. This paper presents a case study of building a natural-language analytics interface on top of a production banking data lake comprising 1,300+ student engagement and curriculum datasets served by dbt Semantic Layer, where the existing SQL surface had reached 80% dbt Semantic Layer adoption and was processing more than 100,000 queries per month at 8-25ms p99 query overhead and where the remaining 20% of non-technical analysts were structurally excluded from self-service because they did not write SQL. We describe the architecture that emerged: a schema-retrieval layer that selects only the relevant subset of the 1,300-table catalog for each natural-language question, an LLM that produces a candidate SQL query, a multi-stage validation layer that enforces governance constraints (row-level security, column masking, query cost ceilings) before any query is executed, and a post-hoc rewriting step that handles the LLM's failure modes qualified column references, ambiguous joins, hallucinated columns. We report pilot results from a four-week deployment to a population of business analysts: 75% adoption (defined as analysts running at least one NL query per week), 80% query accuracy (the produced SQL returns the correct answer to the asked question), and 1.2-second median latency end-to-end. We are honest about the limitations: hallucinated column names remain the dominant failure mode despite schema retrieval, LLM API costs are non-trivial at the query volumes the SQL surface handles, and the governance plumbing is more complex than the natural-language layer. The contribution is a practitioner-grounded design framework with explicit attention to row-level security and audit, intended for teams considering whether the cost of building this layer is justified by the analyst population it unblocks.