Monday, 21 August, 2023 UTC
Summary
With Natural Language Processing (NLP), you can chat with your own documents, such as a text file, a PDF, or a website–I previously wrote about how to do that via SMS in Python. You can also, however, apply LLMs to spoken audio! Read on to learn how to answer questions from a Twilio Programmable Voice Recording Transcription with LangChain.js.
Thank you to my sister-teammate Craig Dennis for pair programming with me in the office!
What is LangChain? LangChain is an open-source tool that wraps around many large language models (LLMs) and tools. It is the easiest way (if not one of the easiest ways) to interact with LLMs and build applications around LLMs.
Prerequisites - A Twilio account - sign up for a free Twilio account here
- A Twilio phone number with Voice capabilities - learn how to buy a Twilio Phone Number here
- Node.js (version 18 or above) installed - download Node.js here
- OpenAI account and API key – make an OpenAI account here and get an OpenAI API Key here
- ngrok, a handy utility to connect the development version of our Python application running on your machine to a public URL that Twilio can access.
ngrok is needed for the development version of the application because your computer is likely behind a router or firewall, so it isn’t directly reachable on the Internet. You can also choose to automate ngrok as shown in this article.
Setup your Node.js project
Make a directory for this project, run
npm init, and accept all the defaults.mkdir langchain-audio-rec cd langchain-audio-rec npm init Add a .env file containing your OpenAI API key:
OPENAI_API_KEY= {REPLACE-WITH-YOUR-OPENAI-API-KEY} Record a phone call with Twilio Programmable Voice
Install some required packages that you'll use:
npm install -S express npm install -S twilio In a file called record.js, add the following code to record an inbound phone call to a Twilio phone number and transcribe the call:
"use strict"; const express = require('express'); const res = require('express/lib/response'); const VoiceResponse = require('twilio').twiml.VoiceResponse; const bodyParser = require('body-parser') const app = express(); app.use(bodyParser.urlencoded({ extended: false })); let recentTranscription; // Returns TwiML which prompts the caller to record a message app.post('/record', (request, response) => { // Use the Twilio Node.js SDK to build an XML response const twiml = new VoiceResponse(); twiml.say('Hello. Please leave a message after the beep.'); // Use <Record> to record the caller's message, transcribe and pass to /handle_transcription twiml.record({ transcribe: true, transcribeCallback: '/handle_transcription'}); // End the call with <Hangup> twiml.hangup(); // Render the response as XML in reply to the webhook request response.type('text/xml'); response.send(twiml.toString()); }); app.post('/handle_transcription', (request, response) => { recentTranscription = request.body.TranscriptionText; //could store this in a database }) app.get('/recent_transcription', (request, response) => { return response.json({recentTranscription}); }) // Create an HTTP server and listen for requests on port 3000 app.listen(3000); Run the file with
node record.js and in another terminal tab, run ngrok http 3000. Grab that forwarding URL so you can configure your purchased Twilio phone number to send a request to your web application.This callback mechanism is called a webhook and can be done in the number's configuration page as shown below.
Click Save configuration. Now call your purchased Twilio number and the call will be recorded! I recorded myself saying Herb Brooks' speech in the movie Miracle.
Afterwards, you can view and hear the recording in your Incoming Calls Log.
In your ngrok tab, you can see the /record and /handle_transcription endpoints were hit.
Now go to your local web server in the browser and append /recent_transcription (http://localhost:3000/recent_transcription). You can see the audio recording transcription there in the browser!
For more information about Twilio Programmable Voice Transcriptions, you can check out the documentation here.
Use LangChain.js to Answer Questions about the Call Recording Add the following line to your
package.json above scripts:."type": "module", In a new file called handle_transcription.js, add the following code importing OpenAI so we can use their models, LangChain's loadQAStuffChain to make a chain with the LLM, and Document so we can create a Document the model can read from the audio recording transcription:
import 'dotenv/config'; //"type": "module", in package.json import { OpenAI } from "langchain/llms/openai"; import { loadQAStuffChain } from 'langchain/chains'; import { Document } from "langchain/document"; (async () => { const llm = new OpenAI({temperature:0.7}); //temperature optional parameter const chain = loadQAStuffChain(llm); const url = "http://localhost:3000/recent_transcription"; //your local server with the transcription const res = await fetch(url); const json = await res.json(); const doc = new Document({ pageContent: json.recentTranscription }); const response = await chain.call({ input_documents: [doc], question: "Great moments are born from what?", //insert question about audio recording transcription }); console.log(response.text); })(); Now, running the file (containing the speech from the movie Miracle) with
node handle_transcription.js should yield the following output:Alternatively, you could upload the transcription as a .txt file to Twilio Assets to host it with a publicly-accessible URL.
The complete code can be found here on GitHub.
What's Next for Twilio Voice Transcriptions, Recordings, and LangChain.js LangChain.js makes it easy to build apps around LLMs–you could easily replace the OpenAI LLM with another from, say, Hugging Face or Cohere. There's also a lot of fun to be had with Twilio Programmable Voice data. You could store phone call recordings in databases, analyze them, take note of patterns, et cetera. This Twilio blog post answers questions about Twilio Voice Recordings with LangChain.js and performs transcription with AssemblyAI.
Developers could perform Retrieval Augmented Generation (RAG) question-answering (QA) on podcasts, lectures, interviews, and so many other recordings. I can't wait to see what you build–let me know online what you're working on with AI!
- Twitter: @lizziepika
- GitHub: elizabethsiegle
- Email: [email protected]