A knowledge base that audits itself
Building internal tools on the Claude API, and the guardrails I put around them.
Sales and customer teams kept answering the same product and compliance questions in Slack, and useful feedback from customer calls kept getting lost. I built a few internal tools to fix that, and wrote a governance proposal so tools like them have clear limits.
The KB Bot
When a teammate reacts to a good answer in Slack, the bot captures it and sends it for human review. Approved answers are used to reply to repeat questions in chosen channels, and they're written back into the source documents. Nothing reaches a document without a person approving it.
Every write-back is recorded along with where it went, so retiring an entry removes its content from every file it touched. Knowing changes could be undone made people much more willing to approve them.
Making the audit predictable
Knowledge bases rot. My first version flagged entries that hadn't been checked in 30 days, which meant rarely used entries were never looked at. I replaced it with a rotation: each entry is hashed into one of four buckets, and each Monday covers one bucket. Every entry gets reviewed once a month, and each session summary links back to the original Slack messages.
// Each knowledge-base entry always lands in the same weekly bucket
function entryBucket_(entryId) {
const digest = Utilities.computeDigest(
Utilities.DigestAlgorithm.MD5, String(entryId));
return Math.abs(digest[0]) % 4;
}
// Week of the month decides which bucket is audited this Monday
function currentAuditBucket_(date) {
return Math.min(Math.floor((date.getDate() - 1) / 7), 3);
}Listening to customer calls
I also built skills that scan recent customer and prospect calls for bugs and feature requests, remove duplicates, and post each one to the right Slack channel with a count of how often it's been asked for. They run on a schedule without anyone starting them.
Guardrails
Tools like these touch real company data, so I wrote an AI governance proposal for leadership aligned to NIST AI RMF and ISO 42001. It covers orchestration, evaluation, and how to choose pilot agents, with a 44-criterion rubric across 17 categories. The core rule is the same one I use for evidence fetchers: each agent gets only the access its task needs.