In this tutorial, you'll build a speech translation bot that analyzes your voice on a call and responds with a translated version in the language of your choice. Most speech translation services require an internet connection. However, the bot you’re going to build would just require your cellular network and as a result, could be used on even the simplest feature phones.

To build this, you'll use Twilio Programmable Voice and IBM Watson’s Speech Translation API.

In order to complete this tutorial you will need the following:

To get started, you will need a Twilio account. If you don’t have one, you can register for a free Twilio account and begin with a free trial number and $10 credit.

To receive calls with a Twilio phone number, you must configure a webhook. Webhooks are HTTP callbacks that are triggered by an event. The event, in this case, would be an incoming phone call to your Twilio phone number.

To create a webhook, you’ll have to create a REST service able to handle incoming HTTP requests and perform some function. This service would need to be accessible from a URL. This is where Twilio Functions comes in.

Twilio Functions is a serverless environment maintained and deployed by Twilio’s cloud infrastructure. With it, you can create webhooks to respond to emitted Twilio events. In this case, you will use Twilio Functions to return TwiML in response to the webhooks.

TwiML, short for Twilio Markup Language, is a way to instruct Twilio on what to do when an event is triggered. For this tutorial, you'll need to respond to users dialing your Twilio number.

For example, this TwiML says hello:

<Respond> <Say>Hello</Say> </Respond> 

This TwiML collects phone number pad inputs during a call:

<Respond> <Gather input="dtmf"/> </Respond> 

Commands after a <Gather> verb don’t run if the caller presses the keypad, so the following command would say "You didn't pick a number" if the user doesn't press the keypad:

<Respond> <Gather input="dtmf"/> <Say>You didn't pick a number</Say> </Respond> 

The <Gather> verb can have nested <Say> commands. This has the effect of speaking while waiting for user input:

<Respond> <Gather input="dtmf"> <Say>Press 1 for option 1</Say> <Say>Press 2 for option 2</Say> </Gather> <Say>You didn't pick a number</Say> </Respond> 

You’ll learn later how to use Twilio Functions to store the TwiML for our bot.

To develop Twilio Functions on your local computer, you’ll need to install the Twilio Serverless Toolkit plugin. First the Twilio CLI will need to be installed, followed by the plugin.

Here’s how to do this:

npm install -g twilio-cli twilio plugins:install @twilio-labs/plugin-serverless 

After the packages have been successfully installed, you’ll need to login to your Twilio account via the CLI:

twilio login 

Upon execution, you'll be asked for your Twilio Account SID and Auth Token. These can be obtained from your Console dashboard.

With that done, you'll create a new Twilio project:

twilio serverless:init twilio-speech-translation-watson cd twilio-speech-translation-watson 

This command will create a new directory with a file structure similar to this:

. ├── assets │ ├── index.html │ ├── message.private.js │ └── style.css ├── functions │ ├── hello-world.js │ ├── private-message.js │ └── sms │ └── reply.protected.js └── package.json 

Each Twilio function is a *.js file in the aptly named functions folder. These files can be nested arbitrarily without any issue.

Next, install the Twilio npm library:

npm install twilio 

Open up the functions/hello-world.js file in a text editor. It should have the following content:

exports.handler = function(context, event, callback) { const twiml = new Twilio.twiml.VoiceResponse(); twiml.say('Hello World!'); callback(null, twiml); }; 

Every Twilio function exports a handler function. This handler function accepts three arguments: context, event, and callback. For now, the important one to focus on is the callback argument. It is a callback function to respond and complete a Twilio Functions execution. Its first parameter is an error message, while the second is a response.

The response in this case is a TwiML <Say> verb, which tells Twilio to speak to the user. Apart from the <Say> verb, in this tutorial you'll also use the <Gather> verb to collect user input and the <Redirect> verb to transfer control of the call to another Twilio Function.

You’re done setting up Twilio. Now you’ll need to get access to IBM Watson’s Translation API. Watson has a generous free tier which is perfect for this tutorial. Register or login if you already have an account.

Next, head over to the IBM Cloud Language Translator. Select the free Lite plan and create a new service.

You’ll be redirected to the Language Translator service page. Select Service credentials from the left-hand menu.

Copy the apiKey from the Auto-generated service credentials dropdown.

Now head back to your project folder and open your .env file. On a new line, add an additional environment variable called WATSON_KEY with a value of the API key you just obtained.

WATSON_KEY=<YourApiKey> 

Now copy the url field from the Auto-generated service credentials dropdown as well. Save this URL somewhere handy, you’ll use it later on.

With that completed, you’re done setting up the Watson Translate service.

The translation service you’re building has three functions:

The following sections will show you how to create these functions.

Create a greeting.js file in the functions folder. This file will contain the handler triggered when a call has started:

touch ./functions/greeting.js 

Windows:

type nul > ./functions/greeting.js 

Copy and paste the following code into this new file:

const VoiceResponse = require('twilio').twiml.VoiceResponse; const voiceConfig = { voice: "Polly.Amy-Neural" } const supportedLanguages = ['german', 'french', 'japanese']; exports.handler = function(context, event, callback) { const twiml = new VoiceResponse(); const gather = twiml.gather({ finishOnKey: '', action: '/handle-language' }); gather.say(voiceConfig, "What language do you want to translate to ?"); supportedLanguages.forEach((language, index) => { gather.say(voiceConfig, `Select ${index + 1} to translate to ${language}`); }); callback(null, twiml); }; 

The gather() function receives an object as an argument with details that it uses to construct a <Gather> TwiML verb.

These arguments are:

The say() function has two arguments. The first is a voiceConfig object which allows you to choose the voice of the narrator. The second is the message to be narrated.

Notice how you loop through each language in the supportedLanguages array, constructing a new TwiML <Say> verb for each language.

When the caller enters a number with the keypad, Twilio calls the /handle-language endpoint with a Digits request value containing the caller’s numeric input.

Now that you’ve collected the user’s speech input from the previous /greeting endpoint, you'll instruct Twilio to <Say> the caller’s language input.

Create a new file called handle-language.js:

touch ./functions/handle-language.js 

Windows:

type nul > ./functions/handle-language.js 

Copy and paste the following code into this file:

const VoiceResponse = require('twilio').twiml.VoiceResponse; const languageOptions = { "1": { language: "german", voiceCode: "de-DE" }, "2": { language: "french", voiceCode: "fr-FR" }, "3": { language: "japanese", voiceCode: "ja-JP" } } exports.handler = async function(context, event, callback) { const twiml = new VoiceResponse(); const digit = event.Digits; if (!languageOptions[digit]) { twiml.say("You didn't say anything."); } else { const { language, voiceCode } = languageOptions[digit]; twiml.say(`Great! Your words will be translated to ${language}`) } callback(null, twiml); }; 

The caller's reply from the /greetings endpoint is accessible from event.Digits.

Let’s check what you have so far. Deploy the functions with the following command:

twilio serverless:deploy 

Head to the Phone Number section of the Twilio Console:

Purchase the desired phone number (or select the trial one provided) and on the phone number’s configuration page, scroll down to the Voice & Fax section:

Select Function from the A call comes in dropdown:

Update the remaining dropdowns to satisfy the image below:

Remember to save.

Now try calling the phone number. If you’ve done everything right, you should get a prompt asking you to enter the translation target language. After this, you should also hear a confirmation message reiterating the language you picked.

If this doesn’t work, check the following:

Next, check if the user’s selected target language is supported. You'll use the ibm-watson library to query Watson’s language API.

Install the ibm-watson npm library:

npm install ibm-watson 

Update the handle-language.js file to import ibm-watson by adding the following two lines to the very top of the file:

const LanguageTranslatorV3 = require('ibm-watson/language-translator/v3'); const { IamAuthenticator } = require('ibm-watson/auth'); 

Now create functions to consume the library you just imported. Add these functions at the bottom of the handle-language.js file:

async function getLanguageCode(language, apikey) { const supportedLanguages = await getSupportedLanguages(apikey); const desiredLanguage = supportedLanguages.find(lan => lan.language_name.toLowerCase() === language); return desiredLanguage.language; } async function getSupportedLanguages(apikey) { try { const languageTranslator = new LanguageTranslatorV3({ version: '2018-05-01', authenticator: new IamAuthenticator({ apikey, }), serviceUrl: 'YOUR_SERVICE_URL', }); const languages = (await languageTranslator.listLanguages()).result.languages; return languages; } catch (error) { console.log(error); throw 'Issue getting languages'; } } 

Remember the URL you copied from the Watson service page? Replace the value of the serviceURL above with your URL. Truncate the URL to not include the string after “ibm.com”.

For example, if your URL is:

https://api.eu-gb.language-translator.watson.cloud.ibm.com/instances/af3dasf833

You should shorten it to:

https://api.eu-gb.language-translator.watson.cloud.ibm.com

Replace the exports.handler function in this file with the following code in order to make use of your new functions:

exports.handler = async function(context, event, callback) { const twiml = new VoiceResponse(); const digit = event.Digits; if (!languageOptions[digit]) { twiml.say("You didn't say anything."); } else { try { const { language, voiceCode } = languageOptions[digit]; const languageCode = await getLanguageCode(language, context.WATSON_KEY); if (languageCode) { twiml.gather({ input: ['speech'], action: `/translate-message?languageCode=${languageCode}&voiceCode=${voiceCode}` }) .say(`Great! Your words will be translated to ${language}, What message do you want to translate ?`); } else { twiml.say("Sorry, language not supported"); twiml.redirect('/greeting'); } } catch (error) { console.log(error); twiml.say("There was an issue getting supported languages"); } } callback(null, twiml); }; 

Notice you pass context.WATSON_KEY to the getLanguageCode() function. As you may have guessed, the context argument is an object with the environment variables you defined in the .env file.

When a language is supported, you reaffirm the user’s language input with the <Say> verb. you then ask the user to speak the message to be translated using the <Gather> verb. Lastly, you redirect the conversation flow to be handled by the /translate-message endpoint.

Let’s look at the gather() function:

twiml.gather({ input: ['speech'], action: `/translate-message?languageCode=${languageCode}&voiceCode=${voiceCode}` }); 

The action endpoint you specify has two query parameters: languageCode and voiceCode. You need languageCode to specify the Watson Translate API’s target language. Then you need voiceCode to specify the Twilio narrator’s voice.

When query parameters are specified in a Twilio endpoint, it is exposed under the event function argument for the function translate-message.js.

Create a translate-message.js file with the following command:

touch ./functions/translate-message.js 

Windows:

type nul > ./functions/translate-message.js 

Add the following content:

const VoiceResponse = require('twilio').twiml.VoiceResponse exports.handler = function (context, event, callback) { const twiml = new VoiceResponse(); const speechResult = event.SpeechResult; twiml.say("You said, " + speechResult); callback(null, twiml); }; 

This simply repeats what the user says in the previous step. Twilio exposes the transcript of the caller’s voice input from the previous endpoint in event.SpeechResult.

You’re now ready to deploy your code and test it out to make sure everything is working so far.

twilio serverless:deploy 

Now it’s time to make the function translate the voice response and say it to the caller.

Replace all the code in translate-message.js with the following:

const VoiceResponse = require('twilio').twiml.VoiceResponse; const LanguageTranslatorV3 = require('ibm-watson/language-translator/v3'); const { IamAuthenticator } = require('ibm-watson/auth'); exports.handler = async function (context, event, callback) { const twiml = new VoiceResponse(); const speechResult = event.SpeechResult; const languageCode = event.languageCode; const voiceCode = event.voiceCode; const translatedSpeech = await translateSpeechResult(speechResult, languageCode, context.WATSON_KEY); if (!translatedSpeech) { twiml.say("Could not translate"); callback(null, twiml); } twiml.say(`${speechResult} means`); twiml.say({ language: voiceCode }, `${translatedSpeech}`); callback(null, twiml); }; async function translateSpeechResult(speechResult, targetLanguageCode, apikey) { try { const languageTranslator = new LanguageTranslatorV3({ version: '2018-05-01', authenticator: new IamAuthenticator({ apikey, }), serviceUrl:'<watson-url>', }); const result = (await languageTranslator.translate({ text: speechResult, target: targetLanguageCode })).result; return result.translations[0].translation; } catch (err) { console.log(err); return ''; } } 

Remember to replace the <watson-url> placeholder with the URL you noted in the IBM Watson dashboard.

Here’s a break down what’s happening:

First, you obtain the request parameters voiceCode and languageCode from the events object. Remember, the previous function passed these parameters.

You pass two attributes to the translate function’s object argument:

With that done, deploy your code one last time:

twilio serverless:deploy 

Try calling the phone number. Everything should work. If it doesn't, try retracing your steps or check out the source code.

In this tutorial you’ve learned how to build a simple speech translating bot. You used TwiML to instruct Twilio how to handle incoming calls and callback requests. You also learned how a Twilio function works and the contents of its various arguments. Lastly, you learned how to call the Watson Translate API with the watson-api npm library.

If you get stuck, here’s the source code: https://github.com/scroobius-pip/twilio-speech-translation-watson-tutorial

For more details of Twilio functions, check:

https://www.twilio.com/docs/runtime/functions/invocation

For more details on the Translate API, check:

https://cloud.ibm.com/apidocs/language-translator?code=node#translate