Continuous Deployment Practice #4: Smoke test the deployment

2012/12/18 — 1 Comment
Continuous Deployment Practice #4: Smoke test the deployment, 5.0 out of 5 based on 2 ratings
VN:F [1.9.22_1171]
Rating: 5.0/5 (2 votes cast)

The first problem than you can have after a deployment is that…. the configuration files for your software are somehow wrong: a wrong database password, a wrong connection url, a wrong log file name… more or less every configuration parameter in your application can be wrong.

Antipattern: prepare a configuration file for the production environment, deploy the application and look at the logs.

Well, you know what happen now: the application sometimes fails to start and you have to spend your time looking at your log files, while the application is down and your services not available to users.

We found a simple solution for that: introduce a smoke test stage into the deployment pipeline.

The recipe to smoke test your deployment

First of all, what is a Smoke Test? According to Wikipedia,

a smoke test proves that “the pipes will not leak, the keys seal properly, the circuit will not burn, or the software will not crash outright

The idea is simple: read the configuration parameters and validate them in the actual environment. For instance:

  • search all the database connection parameters (hostname, database name, username, password, etc…) and try to make a real connection to that database.
  • search all the remote URLs your application uses (APIs, webservices, etc..) and try to connect to them.
  • search all the folder names and file names and check the existence and the permissions.

You can just create a script in your preferred language, execute it before going live with the new release (remember the Practice #3: Deploy on the same way from dev to production environment) and abort the build in case of a test failure: your application still at the actual release and continues to be available to your users.

It’s very import to execute the smoke test script in the enviroment where your application run, because you want to be sure that the resources (database, external API, etc..) are available from the same host where the application is installed, so, if you have multiple servers, run the smoke tests from all the servers in  the array.

Here some real how-to, taken from our smoke test scripts.

How to smoke test a MongoDB connection

Here an example script:

$ smoke-test-mongo-connection.sh srv1:27017,srv2:27017,srv3:27017
#!/bin/bash
# smoke-test-mongo-connection.sh
#
IFS=',' read -ra mongoConnections <<< "$1"
mongoConnection=${mongoConnections[0]}

mongoOutput=$(mongo  $mongoConnection --eval 'status = db.serverStatus(); print(status["ok"]);' --quiet)
if [ "$mongoOutput" != "1" ]; then
    echo "MongoDB not reachable at: $mongoConnection"
    exit 1
fi

echo "Check configuration OK"

How to smoke test folder permissions

Just run a script like this:

$ smoke-test-log-permissions.sh path/to/my/logs theuser

 

#!/bin/bash
# smoke-test-log-permissions.sh
#
applicationLogPath=$1
applicationLogPathPermissions=$2

applicationLogPathOutput=$(stat --format=%U $applicationLogPath)
if [ "$applicationLogPathOutput" != "$applicationLogPathPermissions" ]; then
    echo "Permissions or owner of file $applicationLogPath are wrong: $applicationLogPathOutput"
    exit 1
fi

echo "Check configuration OK"

How to smoke test an URL

You can just execute a script like this:

$ smoke-test-url.sh http://api.exmaple.com

 

#!/bin/bash
# smokte-test-url.sh
#
statusOutput=$(curl --insecure --silent -o /dev/null  -w %{http_code} $url)
if [[ "$statusOutput" == "000" || "$statusOutput" == "40*" || "$statusOutput" == "50*" ]]; then
    echo "$url not reachable"
    exit 1
fi
echo "Check configuration OK"

 Conclusions

As you can understand, I can show here just a few rows of our deployment smoke test script, but I think you can get the idea:

  • Parse the configuration files
  • Try to instantiate database connections, connect to URLs and writing to files
  • fail the build if one resource is not available

I would like to see your comments and thoughts here

VN:F [1.9.22_1171]
Rating: 5.0/5 (2 votes cast)
Be Sociable, Share!

Simonluca Landi

Posts Twitter Facebook Google+

Mi chiamo Simonluca Landi e mi occupo di tecnologie informatiche. Cerco di sviluppare ed implementare soluzioni che tengano conto della “big picture“, sperimentando differenti tecnologie e linguaggi ed evitando le guerre di religione. Segui su Twitter @sll . Leggi tutto si di me qui

Trackbacks and Pingbacks:

  1. Continuous Deployment From The Trenches - Simonluca Landi - 2012/12/18

    […] Practice #4: Smoke test the deployments Mitigate: major configuration errors […]

Leave a Reply

Text formatting is available via select HTML. <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*