Tuesday, 4 September, 2018 UTC

Fetch at the Edge

Fetch at the Edge

Summary

We all remember the cringeworthy XMLHttpRequest we used back in the day to make requests, interact with servers, and retrieve data from URL’s, right? As I recall, it involved some pretty messy code, it didn't give us promises and let's be honest, it just wasn't easy to work with.
Since the dreadful days of XHR, JavaScript introduced the Fetch API, a super clean way to do all the wonderful things that you could do with XHR, plus some. The Fetch API created an entirely new way of making server requests, filled with promises and a more powerful, flexible set of features.
fetch() allows us to make network requests similar to XMLHttpRequest. Only, instead of using callbacks, the Fetch API uses Promises, which enables simpler and cleaner usability without having to remember the complexity of the XMLHttpRequest API.
Fly fetching
Fly Edge Apps are written in JavaScript and deployed all over the world. And since they're JavaScript, you can make them do almost anything. They're especially good at pulling content from just about anywhere and then customizing that content on-the-fly ... enter “fetching”.
With Edge Apps, you can use the exact same fetch method available in standard JavaScript, plus some extra built-in, fetch-like functions we’ve created using fetch that help you do more with your apps to speed 'em up around the world. These methods include fetch, mount, pipeline and proxy. Let’s take an in-depth look at each one of these methods and see how you can use them in an Edge App.
fetch()
Fly Edge Applications have access to the standard fetch API and use the modern async/await syntax. So, you can fetch data from an API, URL, website, database, local file, server, etc... just like you normally would with JavaScript.
If you want to fetch data from a local file, first create a .fly.yml file in the root of your app and define the files here:
# .fly.yml  
files:  
    - image.png
Then, retrieve the file from anywhere else within your app:
async function imageFetch() { 
    const image = await fetch("file://image.png")  
    image.headers.set("content-type", "image/png")  
    return image 
}
Or, fetch data from an API and store it in Fly’s global cache like this:
async function apiFetch() { 
    const url = 'https://randomuser.me/api/?results=20' 
    const data = await fetch(url) 

    fly.cache.set("api-data-2018", data, 86400) // cache for 24 hours 
    return data 
}
Overall, the fetch() method starts the process of fetching a network request and takes two parameters:
  1. A direct URL string or a Request object for the resource you wish to fetch
  2. Options for the request in the form of an init object
It then issues an HTTP/HTTPS request and returns a Promise that resolves to a Response object.
The awesome thing about fetching with Fly is that we've designed it to be lightning fast and, when combined with fly.cache, server load is reduced tremendously. In a nutshell, fetch/Response/Request are incredibly well designed APIs ... and they can be used to build an API for almost any kind of HTTP service. You can read more about this method in our docs, here.
proxy()
The proxy library generates a fetch-like function that makes requests to any given origin, very similarly to a reverse proxy. It can be particularly useful for load balancing (distributing the load to several web servers), caching static content (offloading web servers by caching content like pictures) and for compressing/optimizing content to speed up load time.
When this function makes origin requests, it adds standard proxy headers like X-Forwarded-Host and X-Forwarded-For. It can handle hostname forwarding, and even rewrites URLs before sending them off to the backend. It also passes headers from the original request to the origin.
The proxy() method can be used like this:
import proxy from '@fly/proxy' 

const example = proxy("https://www.example.com")
For example, with Fly, you can use the 'proxy()' function to make a request to your site, grab all your site's images, convert them to WebP and store them in Fly's global cache ... ultimately dramatically speeding up your site because your users will receive your content form whichever Fly datacenter is closest to them. See this example in action, here.
mount()
The “mount” function is useful for mounting different handlers onto different URL paths within your app. For example, you can “mount” certain assets onto any path within your app ... such as mounting “www.example.com/assets/images/2018/08” onto “www.mynewsite.com/images".
Here’s how:
import { mount } from '@fly/fetch/mount' 
import proxy from '@fly/proxy'  

const origin = proxy("https://www.example.com/") 
const assets = proxy("https://www.example.com/images/", { stripPath: "/images/" }) 

function mount(paths) { 
  return function mount(req, init) { 
    const url = new URL(req.url) 
    for (const p of Object.getOwnPropertyNames(paths)) { 
      if (url.pathname.startsWith(p)) { 
        return paths[p](req, init) 
      } 
    } 
    return new Response("no mount found", { status: 404 }) 
  } 
} 

fly.http.respondWith( 
    mount({ 
      "/images/": assets, 
      "/": origin 
    }) 
)
OR...
import { mount } from "@fly/fetch/mount" 
const mounts = mount({ 
    "/hello/": (req, init) => new Response("hello"), 
    "/": (req, init) => new Response("index") 
}) 

fly.http.respondWith(mounts)
pipeline()
The “pipeline” method allows us to combine multiple fetches into a single function by threading requests through each step in the pipeline ... providing us with middleware type functionality found in most modern web frameworks.
For example:
fly.http.respondWith(  
    pipeline(  
        responsiveImages,  
        processImages,  
        rewriteLinks,  
        mount({  
            "/images/": assets,  
            "/": origin  
        })  
    )  
)
Different transformations to this app are applied in the pipeline call. One resizes images, one converts image format, one rewrites image links and one “mounts” different handlers onto different URL paths within your app. This exact example can be found in action, here.
More information about the pipeline library can be found in our docs, here.
Fetch, customize, cache and more...
Most Edge Applications rely heavily on fetch and other fetch-like functions. Edge Apps accept requests and can to talk to a bunch of different backend services to make a bunch of different things happen within your app. Do you need to access an app server, object store, and a REST API to build the right response, no problem.
Fly Edge Apps are basically everything you need to build your own CDN ... and CDNs are really just Edge applications that run someone else's logic. We've been using fetch function generators to create higher level tools for Edge Apps. Most of our users love building Edge Apps that fetch static files from an origin, cache them all over the world, and serves them to nearby users. Try it out for yourself and let us know what you think!

... more @ fly.io

fly.io