[Insert Job Title]

PHP, MySQL and Servers.

Monday, 27 April 2015

Laravel Queues with Supervisor on ElasticBeanstalk

Job and/or message queues is an important component of a modern web application. Simple calls like sending verification emails should always be pushed to a queue instead of done directly, as these calls are expensive and will cause the user to wait a while for the website to finish loading.

In this blog post I will write how to keep a stable queue-worker running on an ElasticBeanstalk environment with the help of the watchdog: Supervisor.

First checkout queues.io for a list of queue-daemons and of course Laravel's 5 own documentation page about queues so you know what's coming up.

You will then most probably come to the conclusion that you need to run the following command for your queue to be actually processed:

$ php artisan queue:listen

Now, I have already seen the weirdest setups, but the most prominent might be maybe something like this:

$ nohup php artisan queue:listen &

The ampersand at the end will cause the call to go into the background, and the preceding nohup will make sure that it will keep running even if you exit your shell.
Personally I would always do something like this in a screen for various reasons - especially for convenience.

Anyways, on your server you will want this to run stable, for as long as possible, and automatically restart on crashes or server reboots.
This is especially true on ElasticBeanstalk, Amazon's poorly but unfortunately popular implementation of a "Platform as a Service":
  • Nothing really has a state - instances can go down and up independently of the application
    • This is especially true when AutoScaling is configured
  • Deploying can crash the queue-listener
  • The server could reboot for various reasons
  • Your queue-listener could crash for various reasons (this happens the most)
    • Application error (PHP exception, for example while working off a malformed payload)
    • SQS is down (yup, it happens!)
To get a grip of this you definitely need to use some kind of watchdog. You can either go with monit or use Supervisor which I found was easier to configure.

Use the following .ebextension to achieve the following (abstract, but checkout the source ;) ):
  1. Install Supervisor
  2. Make sure it runs after a reboot
  3. stop the queue-worker shortly before a new application version goes live
  4. start the queue-worker shortly after a new application version went live



You will notice that you have to set a new param SUPERVISE and set it to "enable" for the script to run. This allows me to switch it on - depending on the environment - or off, if a script is causing problems.
Also be aware, this will only work with newer ElasticBeanstalk versions (1.3+).

I almost forgot to mention the following commands (do not run as root!) that will help you around.

Display last Worker Output
$ supervisorctl tail -1000 laravel_queue

Display last Worker Errors
$ supervisorctl tail -1000 laravel_queue stderr

Display Worker Status
$ supervisorctl status

Start Worker
$ supervisorctl start laravel_queue

Stop Worker
$ supervisorctl stop laravel_queue
Sunday, 22 March 2015

Logstash recipe: Akamai on ELK

One of the perks of working for the new company is the usage of cool tools and providers. One of the new providers is Akamai - state of the art CDN/EdgeCache provider - and also the first one to exist.
The cool new tool: Kibana4!

Just a quick introduction to Kibana: Kibana belongs to the ELK Stack (Elasticsearch, Logstash and Kibana) - and as you spotted correctly comes in last, as it forms the Web/Userinterface to the underlying Elasticsearch database. Logstash sits somewhere in between and is a powerful tool to parse many log formats and also to inject them into Elasticsearch. Elasticsearch itself holds the data and offers a search engine/index.

Why do you need ELK? In a multi-server environment you will want to have your logs somewhere centralized - that is so you do not need to log into each box. Also you want to maybe have some kind of webinterface so you do simple tasks without some commandline-fu - like filtering all failed cronjobs.
There are some great tools that can achieve this as well, like syslog-ng or Graylog.

Wanna see what you are going to get? Here you go:



BTW, yes this is a demo dashboard only, meaning a lot is most probably redundant to Google Analytics - nevertheless it is more exact as it will also capture bots and file requests where no JavaScript is being loaded. The possibilities are of course fare more extensive.

This recipe will take care of three major points:
  • Actual grok filter to match the logs
  • Fix @timestamp to be parsed directly from log-line (as the logs come in batches and often also not in chronological order)
  • Apply GeoIP (via maxmind) filter to ClientIP so we can create cool looking maps on our dashboard

1) First things first

Currently there are two options to receive Logs from Akamai, via FTP and via email. You will want to receive it via FTP so I would suggest to setup a ftp server on your ELK setup.
Akamai will either send the log file gzipped or GPGP encrypted, both formats that Logstash can not read in-house, so you will need to build a script to translate it into plain/text.
Be smarter than me and chose a ftp-daemon that supports uploadscripts, like pure-ftpd or proftpd. It is much easier than using vsftpd + some funky logfile-analyzer-upload-script.

2) Setup Akamai Log Delivery Service (LDS)


  • Log into your Luna Control Center
  • Configure > Log Delivery
  • Select your Object > "Begin Delivery"
  • Make sure you choose "Combined + Cookie + Host Header" as log format - this will give us the possibility to extinguish between different projects later on Kibana
My settings look approx. like this:



3) Use the following Logstash configuration


$ sudo nano /etc/logstash/conf.d/11-akamai-access.conf
input {
  file {
    path => "/home/logs/incoming/akamai/access_*"
    exclude => "*.gz"
    type => "akamai-access"
  }
}

filter {
  if [type] == "akamai-access" {
    grok {
      match => { "message" => "%{IP:clientip} - - \[%{HTTPDATE:timestamp}\] %{HOSTNAME:hostname} \"%{WORD:verb} /%{HOSTNAME:origin}%{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} %{NUMBER:bytes:int} \"(?:%{URI:referrer}|-)\" %{QS:agent} %{QS:cookie}" }
    }
    
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
  }
  
  if [clientip] {
    geoip {
      source => "clientip"
      target => "geoip"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
  }
}


4) Thats it!


  • Restart/reload logstash
  • Wait a few for the log files to come in (might take some hours)
  • Explore the data and create some nice visuals!

Some final notes: there are some major advantages (well and also disadvantages) when analyzing logs directly from the CDN/EdgeCache:

  1. You will get the actual Client-IP (you might be able to redirect it through your ELB until down to your EC2 - but that might be hell of a job)
  2. You will get more accurate Data, as in the best scenario your actual webserver will only get hit once a day ;)
One of the disadvantages: you do not (though there might be products for that) get the data in real time.
Friday, 30 January 2015

Compile OpenSSH 6.7 with LibreSSL on OSX (10.10 / Yosemite)

Lets say you want to use the newest version of OpenSSH on your MacBook / OSX for reasons like:
  • your current version is too old for newer ciphers, key exchanges, etc.
  • you trust LibreSSL more than some OSSLShim
  • you are just some hipster that wants to have cipherli.st running
No worries, in this short tutorial I will show you how to compile OpenSSH 6.7p1 from source without replacing your current installed ssh implementation shipped by OSX.

We will be using LibreSSL instead of OpenSSL which is easier to compile and might be more secure than OpenSSL itself.

Some of the gists I took from here: https://github.com/Homebrew/homebrew-dupes/blob/master/openssh.rb

Get sources


$ wget \
http://mirror.is.co.za/mirror/ftp.openbsd.org/OpenSSH/portable/openssh-6.7p1.tar.gz \
http://www.nlnetlabs.nl/downloads/ldns/ldns-1.6.17.tar.gz \
http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.1.3.tar.gz

Compile LibreSSL


$ tar xvfz libressl-2.1.3.tar.gz
$ ./configure --prefix=/opt/libressl --with-openssldir=/System/Library/OpenSSL --with-enginesdir=/opt/libressl
$ make
$ sudo make install

Compile ldns


$ tar xvfz ldns-1.6.17.tar.gz
$ cd ldns-1.6.17.tar.gz
$ ./configure --with-ssl=/opt/libressl
$ make
$ sudo make install

Compile OpenSSH


$ tar xvfz openssh-6.7p1.tar.gz
$ cd openssh-6.7p1

$ wget \
https://trac.macports.org/export/131258/trunk/dports/net/openssh/files/0002-Apple-keychain-integration-other-changes.patch \
https://gist.githubusercontent.com/sigkate/fca7ee9fe1cdbe77ba03/raw/6894261e7838d81c76ef4b329e77e80d5ad25afc/patch-openssl-darwin-sandbox.diff \
https://trac.macports.org/export/131258/trunk/dports/net/openssh/files/launchd.patch

$ patch -p1 < 0002-Apple-keychain-integration-other-changes.patch
$ patch -p1 < patch-openssl-darwin-sandbox.diff
$ patch -p1 < launchd.patch

$ autoreconf -i
$ export CPPFLAGS="-D__APPLE_LAUNCHD__ -D__APPLE_KEYCHAIN__ -D__APPLE_SANDBOX_NAMED_EXTERNAL__"
$ export LDFLAGS="-framework CoreFoundation -framework SecurityFoundation -framework Security"
$ ./configure \
--prefix=/opt/openssh \
--sysconfdir=/etc/ssh \
--with-zlib \
--with-ssl-dir=/opt/libressl \
--with-pam \
--with-privsep-path=/opt/openssh/var/empty \
--with-md5-passwords \
--with-pid-dir=/opt/openssh/var/run \
--with-libedit \
--with-ldns \
--with-kerberos5 \
--without-xauth \
--without-pie
$ make
$ sudo make install


Use newly installed ssh-agent



$ sudo nano /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist
/usr/bin/ssh-agent > /opt/openssh/bin/ssh-agent

$ sudo launchctl unload /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist
$ sudo launchctl load /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist

Set alias


$ echo "alias ssh=/opt/openssh/bin/ssh" >> ~/.bash_profile


Reboot!


(verify with "ssh -V")
Saturday, 22 November 2014

Hunspell spell checking under PHP with enchant

The spell checking that works perfectly on Google Chrome, OpenOffice and Mozilla Firefox is available to you and PHP as well - all thanks to open source software!

The above mentioned apps use the "Hunspell" library, which can be used directly under PHP without the usage of ugly (and unsecure) exec/system calls.

The following steps I did on my OSX MPB (10.10 / Yosemite) but they will be very similar on any Linux/Unix system (well, might even be easier on Ubuntu or Debian via their package system).
Just make sure you at least use libenchant 1.5


Compile and Install hunspell 


$ wget http://downloads.sourceforge.net/hunspell/hunspell-1.3.3.tar.gz 
$ tar xvfz hunspell-1.3.3.tar.gz
$ cd hunspell-1.3.3
$ ./configure
$ make
$ sudo make install

Compile and Install libenchant 


$ wget http://www.abisource.com/downloads/enchant/1.6.0/enchant-1.6.0.tar.gz 
$ tar xvfz enchant-1.6.0.tar.gz 
$ cd enchant-1.6.0
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

Compile and Install php-enchant (in this case as shared lib)


(there is currently a bug in the configure file that will not recognize your libenchant version and thus not giving you some of the newer features, patch is here)

$ cd php-5.5.14/ext/enchant/
$ phpize
$ ./configure
$ make
$ sudo make install

Something something extension=enchant.so in your php.ini... Dictionaries

$ cd Dicts
$ sudo wget https://chromium.googlesource.com/chromium/deps/hunspell_dictionaries/+archive/master.tar.gz 
$ sudo tar xvfz master.tar.gz

Sample usage