Building Bot Framework Analytics Dashboards using Application Insights

Objectives

If you are building chatbots, the one thing you always need to keep in mind is the build useful bots. In most cases you will start by implementing a more or less simple QnA bot utilizing QnA Maker in your bot to answer commonly asked questions. And even with such a simple bot, which is “only” answering questions stored in a knowledge base, you would likely need to have some sort of analytics service, which indicates if your bot is performing well or not.

If you don’t build bots that are actually used, you are not building bots

Therefore, this blog post will walk you through the steps needed in order to add analytics via Azure Application Insights to your bot to track the bot’s performance.

Prepare the Bot

First of all, we to provision a new Application Insights instance in our Azure environment. This can be simply done by searching for “Azure Application Insights” in the Azure portal and create a new service instance. Atfer this has been provisioned, we need to grab the “Instrumentation Key” from the App Insights pane which we’ll later on use within the bot to establish a communication between this service and our bot:

Now after this you would need to follow this link to add all neceassary components to your bot to enable the logging into App Insights.

After doing so you can go to your App Insights service in the Azure portal and execute this query from the “Logs” section:

customEvents
| where name == 'QnaMessage'
| extend answer = tostring(customDimensions.answer)
| summarize count() by answer

After executing this query, you should see the following result (with your answers):

But as you might guess, there is something missing from the above results. It would be far better if we would not only get the answer in there, but also the question, so we can build a list which holds each question and the corresponding answer. To achieve that, we need to adapt our bot a little as this link indicates.

The first thing we need to add a new class DemoQnAMaker.cs in our project, which should look like this:

public class DemoQnAMaker : QnAMaker
{
    public DemoQnAMaker(
        QnAMakerEndpoint endpoint,
        QnAMakerOptions options = null,
        HttpClient httpClient = null,
        IBotTelemetryClient telemetryClient = null,
        bool logPersonalInformation = false) : base(endpoint, options, httpClient, telemetryClient, logPersonalInformation)
    {
    }
    protected override async Task OnQnaResultsAsync(QueryResult[] queryResults, Microsoft.Bot.Builder.ITurnContext turnContext, Dictionary<string, string> telemetryProperties = null, Dictionary<string, double> telemetryMetrics = null, CancellationToken cancellationToken = default(CancellationToken))
    {
        var eventData = await FillQnAEventAsync(queryResults, turnContext, telemetryProperties, telemetryMetrics, cancellationToken).ConfigureAwait(false);
        // Get question
        var userquestion = turnContext.Activity.Text.ToString();
        // Add new property
        eventData.Properties.Add("UserQuestion", userquestion);
        // Log QnAMessage event
        TelemetryClient.TrackEvent(QnATelemetryConstants.QnaMsgEvent, eventData.Properties, eventData.Metrics);
    }
}

After that, we need to change the code in the DemoBot.cs (or however your bot logic file is called) to use this new class instead of the default QnAMaker class:

var qnaMaker = new DemoQnAMaker(new QnAMakerEndpoint
    {
        KnowledgeBaseId = _configuration["QnAKnowledgebaseId"],
        EndpointKey = _configuration["QnAEndpointKey"],
        Host = _configuration["QnAEndpointHostName"]
    },
    null,
    httpClient,
    _telemetryClient);

This implementation now makes sure, that all activities in which QnA Maker is involved, that all questions from the users will also be sent to App Insights for further analytics. Now after your bot is published to Azure, you can go ahead and test the bot in whatever channel you like, which will trigger App Insights to collect the telemetry data.

Create Queries and a Dashboard

Now the bot is ready and sends telemetry data to our App Insights service, we can create and execute queries in the Azure portal. To give you an idea which queries you can run, the following queries can be used to visualize different information:

Users in last 14 days

This query basically visualizes the number of active users in the last 14 days in a linechart:

Number of users:
let queryStartDate = ago(14d);
let queryEndDate = now();
let groupByInterval = 1d;
customEvents
| where timestamp > queryStartDate
| where timestamp < queryEndDate
| summarize uc=dcount(user_Id) by bin(timestamp, groupByInterval)
| render timechart

QnA Results for all users

If you want to see all questions and answers along with the count of occurences, you can use the following query:

customEvents
| where name == 'QnaMessage'
| extend answer = tostring(customDimensions.answer)
| extend UserQuestion = tostring(customDimensions.UserQuestion)
| where UserQuestion != ""
| summarize count() by answer, UserQuestion

QnA Results by user

The following query enhances the previous query by also adding the userId so you can filter questions and answers by the userId.

customEvents
| where name == 'QnaMessage'
| extend UserQuestion = tostring(customDimensions.UserQuestion)
| extend answer = tostring(customDimensions.answer)
| extend user = tostring(user_Id)
| where UserQuestion != ""
| summarize by UserQuestion, answer, user

NOTE: This query requires you to add the following line to your DemoQnAMaker.cs to also send the userId to App Insights:

eventData.Properties.Add("UserName", username);

These queries can now be added to a dashboard just from the query explorer:

After you have added the query results to your dashboard, you can customize your dashboard to make it look more appealing, like the following for example:

Conclusion

Now that you have enabled your bot to log information into Application Insights, you can go ahead and build beautiful dashboards, which could be shared with users, like the bot admins or the content creators, so they know how the bot is performing and where to tune something to make sure that the end users are happy and adopt your bot (as this is the most important thing at the end of the day). Also please make sure to stay GDPR compliant and do not store any user data against the user’s will. Always tell the users that you are storing conversation and user data for analytics purposes. Happy analyzing 🐱‍🏍