Thursday, 6 December, 2018 UTC
Summary
If you’re like me, you drool just a little bit over serverless architectures. When Rackspace and AWS EC2 made cloud-based computing a mainstream reality, that was awesome enough. But you still had to spin up and maintain your own virtual servers.
With the introduction of things like Twilio Functions or Lambda for truly serverless function execution, DynamoDB for cached state, and API Gateway for click-and-deploy routing—just to name a few—it’s become deliciously easy to build and deploy powerful (and fun) services in minutes. Seriously, the IoT possibilities are endless!
With that in mind, let’s build something fun with Python – a Serverless SMS Raffle. What if users could text a word to a phone number and were entered in to a raffle? Then when we were ready to choose a winner, we could execute a Lambda to choose some number of winners at random and close the raffle?
To do this, we’re going to need accounts on two platforms I’ve already mentioned: Twilio and AWS. Don’t worry, they’re both free, and when all is said and done, running this raffle will cost us just pennies.
So, let’s get started. First things first, we need to setup an endpoint in AWS for Twilio to use when a text is received. We’ll setup this endpoint using API Gateway, which will in turn execute a Lambda function that process entries into the raffle. Easy peasy.
Configure an AWS Role AWS Roles give us a set of permissions to work with. Before we can do anything, we need to create a Role that allows our raffle Lambdas to create logs in CloudWatch and manage tables in DynamoDB.
In the AWS Console, under "My Security Credentials", create a new Role. Choose the "Lambda" service, then create a new Policy.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "dynamodb:CreateTable", "dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:UpdateItem", "dynamodb:DescribeTable", "dynamodb:GetShardIterator", "dynamodb:GetRecords", "dynamodb:ListStreams", "dynamodb:Query", "dynamodb:Scan" ], "Resource": "*" } ] } Name this new Policy "AWSLambdaDynamo", then attach it to your new Role and name it the same.
Great, now let's make some Lambdas using this Role!
Create the AWS Lambda Functions Actually, before we create our Lambdas, let's give them a place to store the raffle data. Create a DynamoDB table with a partition key of
PartitionKey.Alright, now let's make two Lambdas. The first one we'll attach to an API Gateway for receiving an inbound text message—this one will let people enter the raffle and manage all the housekeeping there. The second one will be a Lambda we manually execute—it will close the raffle and choose the winners.
Inbound Message Lambda
Go ahead and create a new Lambda function from scratch, naming it "Raffle_POST" and choosing "Python 3.6" for the runtime. When inbound text messages are sent to our API Gateway (which we'll setup next), they will be processed by this Lambda, and it'll store the sender's phone number in our DyanmoDB table.
Before we plop a bunch of code in there, let's define some environment variables for the function.
SUPER_SECRET_PASSPHRASE(some phrase people must text in order to be entered in to the raffle)BANNED_NUMBERS(JSON list of phone numbers formatted["+15555555555","+15555555556"])DYNAMODB_REGION(likeus-east-1)DYNAMODB_ENDPOINT(likehttps://dynamodb.us-east-1.amazonaws.com)DYNAMODB_TABLE
Now that we have our environment variables defined, let's write some code in the Lambda.
First though: some housekeeping. Let's declare the imports we'll need and bring in those environment variables for easy access.
import os import json import logging import boto3 from urllib.parse import parse_qs SUPER_SECRET_PASSPHRASE = os.environ.get("SUPER_SECRET_PASSPHRASE", "Ahoy") BANNED_NUMBERS = json.loads(os.environ.get("BANNED_NUMBERS", "[]")) DYNAMODB_REGION = os.environ.get("DYNAMODB_REGION") DYNAMODB_ENDPOINT = os.environ.get("DYNAMODB_ENDPOINT") DYNAMODB_TABLE = os.environ.get("DYNAMODB_TABLE") logger = logging.getLogger() logger.setLevel(logging.INFO) dynamodb = boto3.resource("dynamodb", region_name=DYNAMODB_REGION, endpoint_url=DYNAMODB_ENDPOINT) table = dynamodb.Table(DYNAMODB_TABLE) def lambda_handler(event, context): # TODO implement return { 'statusCode': 200, 'body': json.dumps('Hello from Lambda!') } Awesome. Now let's write some helper functions to determine if a raffle is closed (we know it's closed if any rows in the table have been marked a winner), and determine if the person texting us is Karen — Karen is, of course, banned from entering the raffle.
def _is_raffle_closed(table): winner_response = table.scan( FilterExpression=boto3.dynamodb.conditions.Attr("Winner").exists() ) return winner_response["Count"] > 0 def _is_karen(phone_number): return phone_number in BANNED_NUMBERS Let's also write a helper function that conveniently builds a proper TwiML response for us.
Why does this matter? Because whatever response our Lambda gives will be passed back to Twilio. If it's valid TwiML XML, we can task Twilio with an action in our response, in this case, sending a text message back to the sender.
def _get_response(msg): xml_response = "<?xml version='1.0' encoding='UTF-8'?><Response><Message>{}</Message></Response>".format(msg) logger.info("XML response: {}".format(xml_response)) return {"body": xml_response} Great. Now we're ready to write the
lambda_handler.We need to check the incoming text message for the super secret word before entering the sender in to the raffle. Then we'll respond with TwiML to let the sender know they were successfully entered (or shame them accordingly, if they try entering multiple times. Or if they're Karen.).
def lambda_handler(event, context): logger.info("Event: {}".format(event)) data = parse_qs(event["body-json"]) phone_number = data["From"][0] body = data["Body"][0] logger.info("Received '{}' from {}".format(body, phone_number)) if body.lower().strip() != SUPER_SECRET_PASSPHRASE: return _get_response("Hmm. That's not the right entry word for the raffle.") # If the raffle has already been closed (i.e. the Lambda to choose the winners has already been run), # no longer accept new entries if _is_raffle_closed(table): return _get_response("Sorry, this raffle has closed.") # Shame the people who know they aren't allowed to enter the raffle but try to anyway if _is_karen(phone_number): return _get_response("Nice try, Karen. You know you're not allowed to enter the raffle.") db_read_response = table.get_item( Key={ "PartitionKey": "PhoneNumber:{}".format(phone_number) } ) logger.info("DyanmoDB read response: {}".format(db_read_response)) if "Item" in db_read_response: logger.info("Number has already entered raffle") response_msg = "Cheater. You can only enter the raffle once. This incident has been reported to the proper authorities." else: db_write_response = table.put_item( Item={ "PartitionKey": "PhoneNumber:{}".format(phone_number) } ) logger.info("DyanmoDB write response: {}".format(db_write_response)) response_msg = "Boomsauce, your number has been entered in to the raffle. Good luck!" return _get_response(response_msg) The complete code for this Lambda can be found on GitHub. You can also find an example event of a Twilio SMS event for the Lambda on GitHub.
Create a "Choose the Winners" Lambda
We're also going to create a Lambda that we can use to close the raffle and randomly choose winners, texting the winners and updating their records in the DyanmoDB table. So create another Lambda and name it "RaffleChooseWinners". Again, let's start with the environment variables.
DYNAMODB_REGION(likeus-east-1)DYNAMODB_ENDPOINT(likehttps://dynamodb.us-east-1.amazonaws.com)DYNAMODB_TABLETWILIO_ACCOUNT_SIDTWILIO_AUTH_TOKENTWILIO_SMS_FROM(Twilio phone number formatted+15555555555)NUM_WINNERS(defaults to "10" if not set)
Again, let's first declare the imports we'll need and bring in those environment variables for easy access.
import os import logging import boto3 import random import base64 from urllib import request, parse DYNAMODB_REGION = os.environ.get("DYNAMODB_REGION") DYNAMODB_ENDPOINT = os.environ.get("DYNAMODB_ENDPOINT") DYNAMODB_TABLE = os.environ.get("DYNAMODB_TABLE") TWILIO_ACCOUNT_SID = os.environ.get("TWILIO_ACCOUNT_SID") TWILIO_AUTH_TOKEN = os.environ.get("TWILIO_AUTH_TOKEN") TWILIO_SMS_URL = "https://api.twilio.com/2010-04-01/Accounts/{}/Messages.json" # This should be the same Twilio number that users are texting to be entered in to the raffle TWILIO_SMS_FROM = os.environ.get("TWILIO_SMS_FROM") # If the number of entries ends up being less than this number, this Lambda will not run NUM_WINNERS = os.environ.get("NUM_WINNERS", 10) logger = logging.getLogger() logger.setLevel(logging.INFO) dynamodb = boto3.resource("dynamodb", region_name=DYNAMODB_REGION, endpoint_url=DYNAMODB_ENDPOINT) table = dynamodb.Table(DYNAMODB_TABLE) def lambda_handler(event, context): # TODO implement return { 'statusCode': 200, 'body': json.dumps('Hello from Lambda!') } Let's also write a helper function to send a Twilio-powered SMS to each of the winners. Normally we might do this with less code using Twilio's SDK or even the
requests library.However, Lambda does not have native
pip support (you can very easily bundle a zip to upload, but that's a separate tutorial), so let's just do this using native Python.def _send_sms(to_number, msg): populated_url = TWILIO_SMS_URL.format(TWILIO_ACCOUNT_SID) post_params = {"To": to_number, "From": TWILIO_SMS_FROM, "Body": msg} data = parse.urlencode(post_params).encode() req = request.Request(populated_url) authentication = "{}:{}".format(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN) base64string = base64.b64encode(authentication.encode("utf-8")) req.add_header("Authorization", "Basic %s" % base64string.decode("ascii")) with request.urlopen(req, data) as f: logger.info("Twilio returned {}".format(str(f.read().decode("utf-8")))) Add some helper functions to choose the winners, send them a text, and update their record in DynamoDB.
def _choose_winners(eligible_response, fe): # Process as many entries as are given to us in the first response (it may be all of them) entries = set(i["PartitionKey"] for i in eligible_response["Items"]) # If Dynamo gave us a pageable response, it means there are more items that matched our scan, # so rinse and repeat before choosing the winners while "LastEvaluatedKey" in eligible_response: no_winner_response = table.scan( FilterExpression=fe ) entries.union(set(i["PartitionKey"] for i in eligible_response["Items"])) # Select a random sample of size NUM_WINNERS from the list of entries return random.sample(list(entries), NUM_WINNERS) def _process_winners(winners): winner_msg = "You won the raffle! WHAT ARE THE ODDS?! Well, 1 in {}, to be exact, you lucky duck!".format(len(entries)) # Update the record for each winner, and send a text message informing them of their good fortune for winner in winners: winner_phone_number = winner[len("PhoneNumber") + 1:] updated_response = table.update_item( Key={ "PartitionKey": winner }, UpdateExpression="set Winner = :w", ExpressionAttributeValues={ ":w": "true" }, ReturnValues="UPDATED_NEW" ) logger.info("DynamoDB updated response: {}".format(updated_response)) _send_sms(winner_phone_number, winner_msg) Now that we have things initialized and our helper functions implemented, let's write the
lambda_function, which is simply going to validate and iterate over the entries in the DyanmoDB table and choose NUM_WINNERS at random.def lambda_handler(event, context): logger.info("Event: {}".format(event)) winner_nex_fe = boto3.dynamodb.conditions.Attr("Winner").not_exists() winner_ex_fe = boto3.dynamodb.conditions.Attr("Winner").exists() # These queries are ugly and inefficient, but all we're really trying to do here is check if winners # have already been chosen, because if they have, the raffle is closed and the Lambda shouldn't run winner_response = table.scan( FilterExpression=winner_ex_fe ) no_winner_response = table.scan( FilterExpression=winner_nex_fe ) if winner_response["Count"] == 0 and no_winner_response["Count"] == 0: logger.info("Nothing to do, the DyanmoDB table {} does not yet have any entries.".format(DYNAMODB_TABLE)) return { "statusCode": 204 } elif winner_response["Count"] > 0: logger.info("Nothing to do, as the DynamoDB table {} has already had raffle winners processed.".format(DYNAMODB_TABLE)) return { "statusCode": 204 } elif no_winner_response["Count"] < NUM_WINNERS: logger.info("Uh-oh, it doesn't look like we have enough raffle entries to choose {} randomly.".format(NUM_WINNERS)) return { "statusCode": 400 } winners = _choose_winners(no_winner_response["Items"], winner_nex_fe) logger.info("Chose {} winner(s), updating table and messaging them now.".format(len(winners))) _process_winners(winners) return { "statusCode": 200 } The complete code for this Lambda can be found on GitHub.
Awesome, the heavy lifting is done! Now it's time to plug it in.
Set Up a Twilio-Compatible API Gateway This step is actually incredibly useful anytime we need to setup an AWS API Gateway that is compatible with a Twilio webhook. More specifically, we're able to accept something other than
application/json (which is API Gateway's default) for the request/response. As such, you'll have to forgive me for ensuring this paragraph contains the maximum number of XML-based keywords so others will stumble upon this solution—figuring out how to use API Gateway for non-JSON requests ended up being more of a chore than I thought it would be.Setup a new API Gateway. Create a "POST" method at its root with an "Integration type" of "Lambda Function" and point it to the "Raffle_POST" Lambda.
Now we have a new API route, but it's all setup for
application/json content. When Twilio POSTs to this endpoint, it'll do so with a content-type of application/x-www-form-urlencoded, and it'll expect a content-type of application/xml in the response. But Lambda expects that the events it consumes will be JSON. To make this work, we're going to take the URL encoded query parameters Twilio gives us and stuff them in to a "body-json" parameter in the event we'll send to Lambda.Edit the method's "Integration Request". Under "Mapping Templates", add a "Content-Type" of
application/x-www-form-urlencoded using the "General template" of "Method Request Passthrough".Edit the "POST" method's "Method Response". Edit the 200 response so it has a "Content type" of
application/xml.Last, under the "POST" method's "Integration Response", edit the 200 response. Under "Mapping Templates" for "Content-Type" of
application/xml with the following template:#set($inputRoot = $input.path('$')) $inputRoot.body Deploy the new API Gateway and boom! We're now ready for raffle entries. Before moving on to the Twilio configuration, note the "Invoke URL" AWS gives you after deploying the stage.
Configure Our Twilio Phone Number We're now in the home stretch.
In Twilio, create a phone number and set it up. Under "Messaging", select "Webhook". For when "A Message Comes In", select "POST" and enter the deployed API Gateway URL.
Run the Serverless Raffle! Users can now text our super secret passphrase to our phone number to be entered in to the Serverless Python SMS Raffle. When we are ready to close the raffle and have the winners randomly chosen, all we have to do is execute the "RaffleChooseWinners" Lambda (click "Test" and use the "Hello World" event template), which will close the raffle and text the winners.
A list of entries, with winners marked accordingly, can also be found by viewing the items in the DynamoDB table for the raffle.
So, now that the raffle is won and over and you've collected this easily accessible list of phone numbers, I expect you to delete this table immediately – obey the GDPR.
The complete code for this tutorial can be found on GitHub. Comments and questions can be directed to @alexdlaird. Thanks for reading!
Alex Laird is an Engineer at Twilio based in San Francisco, California. He enjoys long walks on the beach and getting caught in the rain – probably the result of being a Bay Area transplant from the Midwest. He also makes a mean cheesecake upon request. (But not for Karen.)