Is there any way to capture and save end-to-end conversation data into blob storage or cosmos DB in bot framework using SDK V4 Nodedjs - azure

Question

I want to store the conversation data to the storage account or cosmos DB. By trying this https://learn.microsoft.com/en-us/azure/bot-service/bot-builder-howto-v4-storage?view=azure-bot-service-4.0&tabs=javascript#using-blob-storage
I am able to send the utteranceslog into blob storage. But I want to store end-to-end conversation data which includes data of both users as well as bot responses using javascript.
I tried using saving user state and conversation state but didn't achieve the desired output.

Answer 1

I created a custom logger (based on an old botduilder-samples sample that isn't there anymore) that accomplishes this using TranscriptLoggerMiddleware. I chose CosmosDB instead of Blob Storage because I felt it was easier to store (and retrieve) as a JSON document. But you could tweak this concept to use any DB. Here is what I did.
First, create your custom logger code. As mentioned, I used CosmosDB so you might have to change some things if you're using a different DB. The timing of the activities was creating concurrency issues, so instead of working around that, I'm storing the transcript object locally and overwriting the DB object on each turn. Maybe not the most elegant, but it works. Also, I've found my wait function to be required. Otherwise you only get one side of the conversation. I've been told this type of wait function is not a best practice, but awaiting a promise or other methods of creating a delay did not work for me. Here is the code:
customerLogger.js
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
const { CosmosDbStorage } = require('botbuilder-azure');
const path = require('path');
/**
* CustomLogger, takes in an activity and saves it for the duration of the conversation, writing to an emulator compatible transcript file in the transcriptsPath folder.
*/
class CustomLogger {
/**
* Log an activity to the log file.
* #param activity Activity being logged.
*/
// Set up Cosmos Storage
constructor(appInsightsClient) {
this.transcriptStorage = new CosmosDbStorage({
serviceEndpoint: process.env.COSMOS_SERVICE_ENDPOINT,
authKey: process.env.COSMOS_AUTH_KEY,
databaseId: process.env.DATABASE,
collectionId: 'bot-transcripts'
});
this.conversationLogger = {};
this.appInsightsClient = appInsightsClient;
this.msDelay = 250;
}
async logActivity(activity) {
if (!activity) {
throw new Error('Activity is required.');
}
// Log only if this is type message
if (activity.type === 'message') {
if (activity.attachments) {
var logTextDb = `${activity.from.name}: ${activity.attachments[0].content.text}`;
} else {
var logTextDb = `${activity.from.name}: ${activity.text}`;
}
if (activity.conversation) {
var id = activity.conversation.id;
if (id.indexOf('|') !== -1) {
id = activity.conversation.id.replace(/\|.*/, '');
}
// Get today's date for datestamp
var currentDate = new Date();
var day = currentDate.getDate();
var month = currentDate.getMonth()+1;
var year = currentDate.getFullYear();
var datestamp = year + '-' + month + '-' + day;
var fileName = `${datestamp}_${id}`;
var timestamp = Math.floor(Date.now()/1);
// CosmosDB logging (JK)
if (!(fileName in this.conversationLogger)) {
this.conversationLogger[fileName] = {};
this.conversationLogger[fileName]['botName'] = process.env.BOTNAME;
}
this.conversationLogger[fileName][timestamp] = logTextDb;
let updateObj = {
[fileName]:{
...this.conversationLogger[fileName]
}
}
// Add delay to ensure messages logged sequentially
await this.wait(this.msDelay);
try {
let result = await this.transcriptStorage.write(updateObj);
} catch(err) {
this.appInsightsClient.trackTrace({message: `Logger ${err.name} - ${path.basename(__filename)}`,severity: 3,properties: {'botName': process.env.BOTNAME, 'error':err.message,'callStack':err.stack}});
}
}
}
}
async wait(milliseconds) {
var start = new Date().getTime();
for (var i = 0; i < 1e7; i++) {
if ((new Date().getTime() - start) > milliseconds) {
break;
}
}
}
}
exports.CustomLogger = CustomLogger;
Now you need to attach this to the botframework adapter in your index.js file. The relevant pieces of code are:
index.js
const { TranscriptLoggerMiddleware } = require('botbuilder');
const { CustomLogger } = require('./helpers/CustomLogger');
//
//Your code to create your adapter, etc.
//
const transcriptLogger = new TranscriptLoggerMiddleware(new CustomLogger(appInsightsClient));
adapter.use(transcriptLogger);
I'm assuming here you already have your index.js file figured out, but if you need any assistance getting that set up and getting the transcript logger to work with it, just let me know.
EDIT: By request, here is what the object looks like in CosmosDB. Normally I would have the "from name" displayed, but because of the way I was testing the bot it came through "undefined".
{
"id": "2020-3-21_IfHK46rZV42KH5g3dIUgKu-j",
"realId": "2020-3-21_IfHK46rZV42KH5g3dIUgKu-j",
"document": {
"botName": "itInnovationBot",
"1584797671549": "Innovation Bot: Hi! I'm the IT Innovation Bot. I can answer questions about the innovation team and capture your innovation ideas. Let me know how I can help!",
"1584797692355": "undefined: Hello",
"1584797692623": "Innovation Bot: Hello.",
"1584797725223": "undefined: Tell me about my team",
"1584797725490": "Innovation Bot: The innovation team is responsible for investigating, incubating, and launching new technologies and applications. The innovation focus areas are:\n\n* Chatbots\n\n* Augmented Reality/Virtual Reality\n\n* Blockchain\n\n* Robotic Process Automation\n\n* AI & Machine Learning\n\nLet me know if you want to learn more about any of these technologies!",
"1584797746279": "undefined: Thanks",
"1584797746531": "Innovation Bot: You're welcome."
},
"_rid": "OsYpALLrTn2TAwAAAAAAAA==",
"_self": "dbs/OsYpAA==/colls/OsYpALLrTn0=/docs/OsYpALLrTn2TAwAAAAAAAA==/",
"_etag": "\"a4008d12-0000-0300-0000-5e7618330000\"",
"_attachments": "attachments/",
"_ts": 1584797747
}
To read the conversation back (even if still in the middle of the conversation), you just create a connector in your bot, recreate the key, and read the file as below (in this case id is passed into my function and is the conversation id):
const transcriptStorage = new CosmosDbStorage({
serviceEndpoint: process.env.COSMOS_SERVICE_ENDPOINT,
authKey: process.env.COSMOS_AUTH_KEY,
databaseId: process.env.DATABASE,
collectionId: 'bot-transcripts',
partitionKey: process.env.BOTNAME
});
// Get today's date for datestamp
var currentDate = new Date();
var day = currentDate.getDate();
var month = currentDate.getMonth()+1;
var year = currentDate.getFullYear();
var datestamp = year + '-' + month + '-' + day;
var filename = `${datestamp}_${id}`;
var transcript = await transcriptStorage.read([filename]);