[Insert Job Title]

PHP, MySQL and Servers.

Monday, 27 April 2015

Laravel Queues with Supervisor on ElasticBeanstalk

Job and/or message queues is an important component of a modern web application. Simple calls like sending verification emails should always be pushed to a queue instead of done directly, as these calls are expensive and will cause the user to wait a while for the website to finish loading.

In this blog post I will write how to keep a stable queue-worker running on an ElasticBeanstalk environment with the help of the watchdog: Supervisor.

First checkout queues.io for a list of queue-daemons and of course Laravel's 5 own documentation page about queues so you know what's coming up.

You will then most probably come to the conclusion that you need to run the following command for your queue to be actually processed:

$ php artisan queue:listen

Now, I have already seen the weirdest setups, but the most prominent might be maybe something like this:

$ nohup php artisan queue:listen &

The ampersand at the end will cause the call to go into the background, and the preceding nohup will make sure that it will keep running even if you exit your shell.
Personally I would always do something like this in a screen for various reasons - especially for convenience.

Anyways, on your server you will want this to run stable, for as long as possible, and automatically restart on crashes or server reboots.
This is especially true on ElasticBeanstalk, Amazon's poorly but unfortunately popular implementation of a "Platform as a Service":
  • Nothing really has a state - instances can go down and up independently of the application
    • This is especially true when AutoScaling is configured
  • Deploying can crash the queue-listener
  • The server could reboot for various reasons
  • Your queue-listener could crash for various reasons (this happens the most)
    • Application error (PHP exception, for example while working off a malformed payload)
    • SQS is down (yup, it happens!)
To get a grip of this you definitely need to use some kind of watchdog. You can either go with monit or use Supervisor which I found was easier to configure.

Use the following .ebextension to achieve the following (abstract, but checkout the source ;) ):
  1. Install Supervisor
  2. Make sure it runs after a reboot
  3. stop the queue-worker shortly before a new application version goes live
  4. start the queue-worker shortly after a new application version went live



You will notice that you have to set a new param SUPERVISE and set it to "enable" for the script to run. This allows me to switch it on - depending on the environment - or off, if a script is causing problems.
Also be aware, this will only work with newer ElasticBeanstalk versions (1.3+).

I almost forgot to mention the following commands (do not run as root!) that will help you around.

Display last Worker Output
$ supervisorctl tail -1000 laravel_queue

Display last Worker Errors
$ supervisorctl tail -1000 laravel_queue stderr

Display Worker Status
$ supervisorctl status

Start Worker
$ supervisorctl start laravel_queue

Stop Worker
$ supervisorctl stop laravel_queue

7 comments:

  1. > "Simple calls like sending verification emails should always be pushed to a queue instead of done directly"

    I love queues and it is a great way to speed up certain actions in your app. But I wouldn't say "sending verification emails should always be pushed to a queue". Sending an email is essentially instant on many servers since mail servers do their own queueing, so the send mail function returns immediately. It may be a bit slower if you're using a third party mail sending service, but is still probably very fast.

    I would say monitor and test your app, and use queues as needed. But I wouldn't automatically assume sending an email is one of the situations where you necessarily need to use a Laravel queue.

    ReplyDelete
    Replies
    1. This was indeed doing some assumptions, though:

      1) In most cases, email is not being sent anymore from the server itself (via mail()) but external services like mandrill are being used. Having a reliable delivery and deliver-ability reports is not that trivial to replicate in your own setup/infrastructure

      2) Yes, using mail() is faster because you are avoiding expensive external IO/TCP, but depending on the configuration of your MTA it will still do some checks that will block the function call

      But yeah, I am pretty sure there are better examples when to use a queue :)

      Delete
  2. Hello thanks for this, but I can't get it working. I get:
    /bin/sh: .ebextensions/supervise.sh: Permission denied
    Do I have to run it as root? And how do I do that?

    ReplyDelete
    Replies
    1. Hi Adam,

      If you haven't figured it out yet (or also for other readers): always make sure that your bash scripts have the executable bit set.

      So do something like this inside the .ebextensions folder: $ chmod +x *.sh

      Delete
  3. Hi, Thanks for the awesome script. Unfortunately, I can't get the environment variables. Any idea?

    ReplyDelete
  4. Here is the output https://www.dropbox.com/s/xeaqj1isjs6hqjz/awseb-queue-error.png?dl=0

    it shows that the database config username is "forge", which is the default value.

    ReplyDelete
  5. Anybody getting file permission error use this
    https://gist.github.com/mubbashar/4161543511f53d7087e17a8a7b1e163d

    ReplyDelete