hila Monitoring now standard for LLMs in Conversational Finance

June 14, 2024

Today, hila Monitoring is now a standard component for Conversational Finance and a core feature of the hila Platform. With hila Monitoring, administrators can set up key policies that monitor the cost, quality and speed of the LLM models and their outputs.  

A graph with purple linesDescription automatically generated
This shows the overall change for the various metrics we’re monitoring, including cost, and latency

For example, should the number of users in a system remain the same, but the cost increase, the reason could be an increased number of tokens sent to the model. Put differently, users could be asking far longer or more complex questions.  

There are other items that impact cost, of course, such as the type of LLM used (GPT3.5 costs approximately 10X less than GPT4, for example), number of queries, the number of users and the model’s efficiency (the length of the answer compared with the length of the question).  

In addition to these various aspects of an LLM, hila Monitoring tracks the time that it takes for an LLM to respond, the relevancy of an answer to the original question, and the quality of the SQL generated.

A screenshot of a computerDescription automatically generated
This is a breakdown of the metrics we are monitoring. The metrics include many dimensions of cost (such as input and interaction cost) and latency (both for the overall response and for the generated SQL)

Should there appear a major change in any of the metrics tracked, hila Monitoring provides compelling visualizations that provide context around the shift.  

A graph with purple squaresDescription automatically generated
A closer look at one of our monitored metrics, generated SQL latency, we can see the discrepancy between the target date (month of June) and the baseline date (month of May)

Administrators also receive alerts if costs exceed preset boundaries, the quality drops below a set threshold, or the latency rise above average. This builds on the value of our hila Conversational Finance application, which works on financial data and demonstrates a significant value for enterprises to rapidly retrieve visualizations and insights from their data.  

Should someone want to get started quickly, Conversational Finance is set up for business users to begin to derive value from generative AI.  

This newest feature adds to an overall suite of components in the hila Platform that enable generative AI to be useful for an enterprise. There are substantial technical aspects to hila Monitoring as well for data scientists, including root-cause analysis, segmentation, policy management, outlier analysis, and explainability. All of these contribute to the mission of Vianai to build responsible AI for enterprises.  

The other components in the hila Platform include txt2sql, unstructured querying, anti-hallucination, custom and fine-tuned LLMs, RAG, vectorizing documents, agentic village components for additional extractions, and more.