Switch to a laptop (or maximize your browser window) for an optimal viewing experience.

UWB Hacks the Cloud Documentation

All the essential information needed to create an amazing cloud-based application at the UW Bothell 2020 hackathon.

Go to Documentation Home Go to Hackathon Website View on GitHub

CrowFacts, an Example Project for AWS

One of the biggest problems faced by humanity is that we have no way to easily get facts about crows. Until now.

Goals

User Experience

  1. The user goes to our website and has the option to get crow facts or enter new crow facts.

  2. The user can input their own crowfacts (assuming it doesn’t contain any no-no words). A list of all submitted words will show on the right side.

via GIPHY

Architecture

To create a functional user website which achieved the above goals, we used AWS services to host the website. The website’s design is described in detail below.

Visit the website here.

View the Lambda code for the PUT call here.

View the Lambda code for the GET call here.

The diagram of the system flow is below: Visual representation of the archtectural design of crow facts

Implementation Notes

S3 & Website Construction

Website Creation

The website was written on a local machine and deployed to S3.

The website for this project was built with a straightforward HTML, CSS, and vanilla JavaScript stack. jQuery was used to perform HTTP requests to the database call pipeline. The complete code for the website is available in the project repository.

To streamline site deployment, we used a GitHub Action which will copy specified files to the S3 bucket when changes are pushed to master. This was great, because we didn’t have to manually update the contents of the S3 bucket during website development.

Domain Configuration

To set up the custom domain for the site, we did the following:

  1. Chose a desired URL for the site: https://crowfacts.uwbhacks.com
  2. Created an S3 bucket named as the full URL and set it up for static website serving, by following these instructions from the AWS docs. Because we already owned the domain, we skipped all steps related to AWS Route 53 in that tutorial.
  3. In Cloudflare, our DNS management service, we created the crowfacts subdomain for uwbhacks.com as a CNAME record.
  4. We then updated the bucket accesss policies with this template provided by Cloudflare.

We used the following tutorials and resources as references:

API Gateway

Basic Configuration

The basic API Gateway configuration we did was:

  1. Create a new API in the AWS Console. We named ours crowfacts and chose the REST API protocol.
  2. Create two resources: getCrowSpecies and UserFacts.
  3. Ensure that the Lambda functions we wanted to link HTTP methods to were in the AWS account, because the functions have to be available at the time of method creation.
  4. Under getCrowSpecies, create a GET method and link it to the corresponding Lambda function we wrote for this purpose.
  5. Under UserFacts, create a GET method and link it to the corresponding Lambda function we wrote for this purpose.
  6. Under UserFacts, create a POST method and link it to the corresponding Lambda function we wrote for this purpose.
  7. Under the “Actions” menu dropdown, click “Enable CORS”. Accept all default settings and click the “Enable” button.
  8. Under the “Actions” menu dropdown, click “Deploy API”. Create a new deployment stage and click Deploy.
  9. After deployment, click on “Stages” in the left-hand menu for the API. Expand the resource tree and click on each method to get the URL endpoint for that method & resource.

Ta-da! After completing these steps, we were able to successfully cURL the endpoints for our API and validate that it worked as expected.

Custom HTTP Error Handling

This application processes user input, and as such, we wanted to validate that input and return errors to the client when the input didn’t meet our specifications for database entries. We did this by setting up custom error handling in the API Gateway POST method under the /UserFacts resource.

To enable the client to receive custom HTTP errors based on the Lambda function’s responses, we set up Integration and Method responses. We used this AWS blog post as our primary reference.

Lambda Function Configuration

Instead of returning a JSON blob from Lambda, we had to raise an exception in the Lambda function to get API Gateway to use custom error responses. An example of the correct syntax is:

import json
import boto3
import os

def lambda_handler(event, context):
    raise Exception('This is an example exception from Lambda.')

The actual implementation of where we raised exceptions can be found here.

Method Responses from API Gateway to the Client

In order to set up error handling from Lambda, we had to first define what status codes we wanted to return to the client. Each status code indicates a different type of error and is accompanied by a different error message which the client site can parse.

We used the following error codes:

For each of these method responses, we had to add the Access-Control-Allow-Origin header to the status code so that the custom error would fulfill the browser’s CORS requirement. Clicking “Enable CORS” in the console action dropdown only enables CORS for the standard 200 status code response.

After completing this step, our method response section looked like this:

Method Response Configuration for Custom Errors to Client

Integration Responses from Lambda to API Gateway

After we designated the HTTP error status codes we wanted to use in Method Response, we had to define what Lambda errors mapped to what status codes.

To do this, we created integration responses for each custom method. For each integration response, we provide a regular expression which searches the error message returned from Lambda. If a match for the regular expression is found, API Gateway will return the HTTP error code associated with that regular expression.

We have the option to define post-processing for the Lambda error message, but we elected to return Lambda’s raw string; this is called a passthrough response.

Our definitions look something like this:

Configuration for integration responses in API Gateway

After we created the Integration responses for our custom errors, we had to enable CORS for the errors by hand. We did this by creating a new header mapping for each Lambda Error Regex entry. We set the header to Access-Control-Allow-Origin and the mapped value to '*'. The syntax for this step is very important; it must respect the CORS protocol.

After setting this up, our integration responses looked like this:

Integration Response Configuration for Custom Errors from Lambda

Handling Error Responses in the Client

The jQuery function used by the client to makes the request requires a definition of how to handle HTTP errors. We chose to simply display the error message from API Gateway. The definition for how we handled the custom errors can be seen here.

Lambda

AWS Lambda handles all of CrowFacts’ computation. Within AWS Lambda, there are several important things that must be configured in order for CrowFacts’ code to operate correctly.

Give Lambda access to DynamoDB

By default, creating a Lambda function from scratch, adding DynamoDB code, and pressing “Test” won’t work. Why? Simple, because the Lambda function does not have access to DynamoDB. Cloud systems operate on a least-required permissions model, meaning that in order for your Lambda function to access DynamoDB, you must explicitly grant Lambda access to DynamoDB (and any other AWS services you might want Lambda to access). Refer to AWS documentation on IAM permissions for DynamoDB, and check out the TL;DR explanation of IAM on the documentation site for this hackathon.

Set up environment variables for Lambda functions

DynamoDB

The information presented on the website was stored in a DynamoDB database table. This is a NoSQL database, and there are important factors to consider when implementing this in a project.

Consider your usage

NoSQL databases do not require a defined relational schema as SQL databases do. They are flexible, allowing attributes to be introduced as needed. If the relationships between data points are important, then consider AWS’s relational database options such as Amazon RDS.

Create a table

Your table is the place where you store your information. Each table requires a primary key, which can be a single item, or a combination of a partition and sort key. Consider your choice carefully, as there is no way to change your primary key once the table is created. Reference the list of reserved words for DynamoDB, to make sure you are not using one of those words, which will cause problems down the line.

Set up Permissions

Interaction with other AWS services requires the set up of permissions, so that those services may interact with your database. AWS utilizes IAM permissions, which can be made as broad or as narrow as your use-case requires. Setting your permissions up is a key factor for being able to use Lambda functions on your database or make API Gateway calls.

Import/Export Datasets

There are multiple approaches for handling the import and export of your DynamoDB information. See the following links for some options on how to implement it.

AWS Resources, SDK Links, Tutorials, and Helpful Tidbits: Importing Large Datasets</br> This document goes over some options for importing and exporting data in DynamoDB. It includes functional code for Lambda calls to import/export a .JSON file.

What is AWS Data Pipeline?</br> This is the AWS documentation on Data Pipeline, a service which can be used to import/export database information. Can become costly quickly, so use with caution.

Mockaroo</br> A good website to use to get some mock data to use when testing your import/export methods.

Gotchas & Lessons Learned