How do we monitor AI?

As we work with our first wave of clients for AI Monitor, www.deepfutureanalytics.ai, I am surprised at some of the use cases that are arising. These speak much more deeply to the general problem of AI governance. Let me share one I didn’t anticipate.

Retrieval augmented generation (RAG) models are often promoted as being safer and more reliable than general purpose LLM’s. Without arguing that point, a governance question came up with one such installation.

With many AI installations, we are finding that the risk comes not from the answers being given, but the questions being asked. For a system that was installed to provide quick policy answers to support agents, what happens if the agent asks for a loophole? Of course they would never use that language, because they would know that they shouldn’t be trying to work around policy, but they have incentives and clients they want to make happy and there are many reasons to see if an alternative exists.

The governance team has discovered that they need monitoring not of the AI’s answers, but of the questions being posed by the humans. Keyword searching would be hopeless because of the intentional nuance being put into the questions. Finding errant questions requires the same kind of subtlety.

This is not a compliance risk or an ethical risk so much as a business rules violation risk, and in some cases, legal risks arise from this as well. Therefore, our installation will be sampling the questions asked of the AI to make sure that they don’t deviate from the policies that were shared during staff training. We all know that humans who are behind schedule or targets can be expected to cut corners at times. We need to find those corners. AI systems of any kind can equally be an ally or an accomplice.

Joseph Breeden
Posted on LinkedIn