The holidays are just around the corner, and Shopify app developers need to be ready. The busiest shopping days of the year are no time to experience a failure with your app — you need to be prepared to handle the influx of orders and updates to customers, products, and more.
"The busiest shopping days of the year are no time to experience a failure with your app — you need to be prepared to handle the influx of orders and updates to customers, products, and more."
Many Shopify developers make use of webhooks — messages that Shopify sends when something happens inside of Shopify. Webhooks are an efficient way to track changes in the ecommerce platform, and there’s no need to poll Shopify for changes — applications are notified via a push message when something happens in Shopify.
Until you start receiving webhooks at scale, you can't anticipate how many of them will be sent, and when. When we first launched our backup app, Rewind, we were surprised to see webhooks were being sent every day at midnight for a large number of stores.
After talking to customers, we learned that these webhooks were sent when third-party systems synced Shopify data. For example, an inventory system that syncs daily can cause your app to be flooded by thousands or tens of thousands of webhooks, with absolutely no warning.
In this article, we’ll help you create a system that can handle webhooks offline, so your application can handle any load that Shopify sends your way. You can then process the webhooks as quickly as you want, on your terms. This system is built on top of a “serverless” architecture, so it scales to handle large loads automatically, and is inexpensive.
You might also like: How to Brand Your Shopify App to Earn More in the App Store.
Creating a system to handle webhooks
Shopify requires that your application respond to webhooks in a timely manner, otherwise they'll be removed. If you process webhooks inline, as they happen, that can cause a major problem, as your application needs a powerful server to handle the webhooks that might be sent. This can be extremely expensive — you’ll need to run powerful servers 24/7, since you’ll never know when the webhooks may be sent.
Even with lots of large servers, you still run the risk of not responding quickly enough, and missing the data in the webhook, or having Shopify cancel all future webhooks.
A better idea is to handle them in a separate system, and have something that responds to Shopify quickly. If you acknowledge Shopify’s webhook quickly, and save the data somewhere else, your webhooks will never be cancelled, and you’ll have the data from the webhook so that you can process it in the future. You’ll never miss webhook data, and can ensure that your application never misses any data that’s sent to it.
This blog post describes the system we built for Rewind on top of Amazon Web Services, using API Gateway, Lambda, and SQS Queues. When Shopify sends a webhook to Rewind, we process it using API Gateway and Lambda — two “serverless” AWS services that autoscale to meet demand. The messages are stored in an SQS Queue, which saves the data from the webhook. Servers then processes the webhook queues at a rate you control.
You might also like: How to be Successful on the New Shopify App Store.
1. Set up the SQS queue
The first step in this process is to set up the SQS queue. SQS has two types of queues — a first in / first out (FIFO) queue, or a regular queue. You may want to use a FIFO queue to ensure the order of the webhooks is maintained, or you can use a regular queue and check the updated_at time of the message. Either will work, and which one to use depends on your usecase.
2. Set up the Lambda function
AWS Lambda allows you to run code on AWS servers, without having to manage the servers yourself. People often refer to this as "serverless" computing (although technically there are servers in the cloud.) Lambda servers are run by Amazon, and autoscale to handle increased loads. You only pay for the milliseconds that it takes for your code to execute, so it’s very inexpensive to run.
To get started with Lambda, create a new function. This code will take the incoming information from API Gateway (which we'll set up in the next step), check the security of the webhook (to ensure it's coming from Shopify), and then add it to the SQS Queue that we created in step one.
Take a look at the code:
3. Set up API Gateway
Once Lambda is set up, the next step is to set up API Gateway. API Gateway provides a way to define your API, which will run in AWS without any servers. Rather than hit your servers, webhooks will hit API Gateway. This way you don't need to worry about running any servers, and scaling them as the webhooks come in.
To set up API gateway, first create a new model with the following schema:
For the integration request, we set up the following mapping template:
4. Set up Elastic Beanstalk workers
The last piece is to work off the SQS queue with an Elastic Beanstalk worker. Elastic Beanstalk workers automatically pull an item from the SQS queue, and POST it to localhost on the machine.
Our workers are written in Ruby, and are a simple Ruby application. You can find the code for our workers in our Github repository. We’veopen sourced our code for this serverless Shopify webhook system on Github, and welcome contributions from others to help improve it.
The results we’ve seen
We've used this system for our own Shopify backup app with great success. We're able to process hundreds of thousands of webhooks per day, and are able to control how quickly and when we process the webhooks. This allows us the freedom to perform database maintenance tasks easily, without worrying about how our application will be affected.
The Elastic Beanstalk workers are very efficient, and can process SQS messages quickly. Since you can add workers as the queue size increases, your limiting factor will likely be your database. It's also very inexpensive — at scale, this system costs approximately $0.01/10,000 webhooks.
Preparing to handle increased webhook volume is a must for BFCM
In 2016, there was 220% increase in online shopping sessions, during Black Friday/Cyber Monday. In order for your app to be effective for the merchants who use it, preparing for this seasonal spike is a must. If your application is struggling to handle webhooks from Shopify, this system can help you handle the load without costing you a lot of money.
Have you begun preparing your app for BFCM? Let us know in the comments below!