Continous Deployment Practice #3: Deploy on the same way from dev to production environment

2012/12/02 — 2 Comments
Continous Deployment Practice #3: Deploy on the same way from dev to production environment, 5.0 out of 5 based on 1 rating
VN:F [1.9.22_1171]
Rating: 5.0/5 (1 vote cast)

We found that this is the best way to mitigate the usual impedance mismatch between “devs” and “operations”: deployment to production WILL fail, even if you write the best documentation possible, with a full description of all the steps required, even if the “operations” guys write some script to execute the procedures.

The real solution that worked in our environment was to create a single deploy script that is used to deploy the application in all the environments in the deployment pipeline, from dev to production environment.

The recipe to deploy on the same way from dev to production environment

As our application stack is based on PHP running on Linux, was natural for us to implement the deploy script using phing, scp and ssh

The workflow is:

  • build an “artifact” (environment independent)
  • copy the artifact to the destination environment in an new folder, let’s say “application.233”
  • copy the configuration files required for the destination environment
  • setup all the required permissions on destination environment
  • run the “migration” scripts required to update the database, if required
  • move a symbolic link “application” from “application.232” to “application.233”
  • apache will now see the new files and serves the new application

We deploy in all the environment using this same workflow, so even to deploy on “localhost” machine, we run “scp” to copy files and “ssh” to setup permissions, run migrations, etc… and because we have multiple servers on the each environment, we use the “parallel-ssh” and “parallel-scp” utilities.

Let’s see some of the magic that we have in the phing build.xml file…

How to use parallel-scp to copy on multiple servers

Here is the snippet from build.xml

 

<target name="exec-pscp">
 <exec
 command="pscp ${scp.host.remote} -O LogLevel=ERROR -O UserKnownHostsFile=/dev/null -O StrictHostKeyChecking=no -O IdentityFile=${ssh.privkeyfile} -t 0 ${file} '${todir}'"
 outputProperty="pscp.output"
 />
 <echo msg="pscp.output: ${pscp.output}" />
 <condition property="pscp.failed">
 <contains substring="FAILURE" string="${pscp.output}" />
 </condition>
 <fail if="pscp.failed" message="Remote copy failed: ${pscp.output}" />
 </target>

 

The property ${scp.host.remote} contains the list of the servers to deploy into, for instance “-H root@server1 -H root@server2”

As you can see, we store the output of the pscp command in the property “pscp.output”, check the result of the operation and fail the build if required.

 

How to use parallel-ssh to remote execute commands on multiple servers

A similar trick is used to remote execute commands on all the servers in the environment. Here is the snippet:

 

<target name="exec-pssh">
 <exec
 command="pssh ${ssh.host.remote} -O LogLevel=ERROR -O UserKnownHostsFile=/dev/null -O StrictHostKeyChecking=no -O IdentityFile=${ssh.privkeyfile} -P -i -t 0 '${ssh.command}'"
 outputProperty="pssh.output"
 />
 <echo msg="pssh.output: ${pssh.output}" />
 <condition property="pssh.failed">
 <contains substring="FAILURE" string="${pssh.output}" />
 </condition>
 <fail if="pssh.failed" message="Remote command failed: ${pssh.output}" />
 </target>

 

Again, the property ${scp.host.remote} contains the list of the servers to deploy and we check the result of the operation looking at the “outputProperty” of command.

The deploy worlflow

here is an extract from the build.xml, showing the main workflow for the deploy.

 

<target name="deploy">
<echo msg="Copying ${artifact.tgz.path} archive to environment ${deploy.environment} hosts:${fs.host}" />
 <phingcall target="exec-pssh">
 <property name="ssh.host.remote" value="${fs.hosts.argument}" />
 <property name="ssh.command" value="rm -Rf ${fs.deploy.basedir.number}; mkdir -p ${fs.deploy.basedir.number}" />
 </phingcall>
 <phingcall target="exec-pscp">
 <property name="scp.host.remote" value="${fs.hosts.argument}" />
 <property name="todir" value="${fs.deploy.basedir.number}" />
 <property name="file"  value="${artifact.tgz.path}" />
 </phingcall>
 <echo msg="${artifact.tgz.name} copied to environment ${deploy.environment} host:${fs.host}" />
<!-- we unzip the application artifacts in a new fresh location,           -->
 <!-- set permissions and and move a symlink to this location.              -->
 <!-- in this way we have a local backup of previuos version, useful for    -->
 <!-- a fast rollback (just move the link to the previous version           -->
<echo msg="Estracting archive to ${deploy.basedir.number} on host:${fs.host}" />
 <phingcall target="exec-pssh">
 <property name="ssh.host.remote" value="${fs.hosts.argument}" />
 <property name="ssh.command"
 value="tar -xzf ${fs.deploy.basedir.number}/${artifact.tgz.name} -C ${fs.deploy.basedir.number};
 rm -f ${fs.deploy.basedir.number}/${artifact.tgz.name}" />
 </phingcall>
 <echo msg="copying configs from configs/${deploy.environment} to ${fs.deploy.basedir.number}/code/configs on host:${fs.host}" />
 <phingcall target="exec-pssh">
 <property name="ssh.host.remote" value="${fs.hosts.argument}" />
 <property name="ssh.command" value="mkdir -p ${fs.deploy.basedir.number}/code/configs" />
 </phingcall>
 <phingcall target="exec-pscp">
 <property name="scp.host.remote" value="${fs.hosts.argument}" />
 <property name="todir" value="${fs.deploy.basedir.number}/code/configs" />
 <property name="file"  value="configs/${deploy.environment}/*.ini" />
 </phingcall>
 <echo msg="done" />

…. more steps here…. and then

 

<echo msg="================================================================================" />
 <echo msg="Final step: Linking ${deploy.basedir.number} to ${deploy.basedir}" />
 <echo msg="================================================================================" />
 <phingcall target="exec-pssh">
 <property name="ssh.host.remote" value="${ws.hosts.argument}" />
 <property name="ssh.command"
 value="rm -fR ${deploy.basedir};
 ln -s ${deploy.basedir.number} ${deploy.basedir};
 " />
 </phingcall>
 <echo msg="================================================================================" />
 <echo msg="Checking link..." />
 <phingcall target="check-link" />
 <echo msg="Linking OK!" />
 <echo msg="================================================================================" />
 </deploy>

Conclusions

I can show here just a few snippets of our build.xml, but I think you can get the basics from what I show you:

  • Write a single deploy script that can be used to deploy in every environment in the same way
  • Start from solving problems related to your production environment early (EG: multiple servers, minimize downtime, easy rollback, etc..)
  • Alway check the results of the operations run and fail as required.

Feel free to share your comments and thoughts

VN:F [1.9.22_1171]
Rating: 5.0/5 (1 vote cast)
Be Sociable, Share!

Simonluca Landi

Posts Twitter Facebook Google+

Mi chiamo Simonluca Landi e mi occupo di tecnologie informatiche. Cerco di sviluppare ed implementare soluzioni che tengano conto della “big picture“, sperimentando differenti tecnologie e linguaggi ed evitando le guerre di religione. Segui su Twitter @sll . Leggi tutto si di me qui

Trackbacks and Pingbacks:

  1. Continuous Deployment From The Trenches - Simonluca Landi - 2012/12/02

    […] Practice #3: Deploy on the same way from dev to production environment Mitigate: deployment to production fails […]

  2. Continuous Deployment Practice #4: Smoke test the deployment - 2012/12/18

    […] script in your preferred language, execute it before going live with the new release (remember the Practice #3: Deploy on the same way from dev to production environment) and abort the build in case of a test failure: your application still at the actual release and […]

Leave a Reply

Text formatting is available via select HTML. <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*