A confident answer needs a foundation.
Designing enterprise knowledge assistance around permissions, provenance and evaluation.
Amazon Bedrock · OpenAI · retrieval · evaluation
The question
How do you let people ask questions of an enterprise without turning a language model into an unofficial source of policy? A useful answer needs to reflect both the current source and what this particular person is allowed to see.
The hard decision
Start with one bounded knowledge domain and one accountable content owner. The difficult work is often retiring conflicting documents, defining access and agreeing which questions deserve an answer. Adding a larger model does not resolve those decisions.
Inside the system
An ingestion pipeline records source versions, access labels and freshness. Retrieval filters to the user’s permitted knowledge before relevant passages reach the model. Bedrock or OpenAI generates a response with source references. The application enforces tool permissions separately from model instructions.
A citation is not the whole evaluation
A response can cite a real document and still misrepresent it. Evaluation therefore checks retrieval relevance, whether claims are supported, access boundaries and the quality of abstention. Include stale policies, conflicting versions and questions that cannot be answered from the corpus.
The operating measures
Track supported-answer rate, permission violations, useful abstentions, user corrections, latency and cost per useful response. Separate retrieval defects from generation defects so the team knows which layer to change.
What the client owns
A governed corpus, documented retrieval and model choices, a reusable evaluation set and release criteria. Operational ownership includes source updates and incident handling, not just application uptime.
What would this look like
in your business?
A working conversation about the opportunity, the constraints and a useful first step.