Continuous Deployment Practice #5: Use Business Metrics (KPI) to validate deploy

2013/04/15 — Leave a comment
Continuous Deployment Practice #5: Use Business Metrics (KPI) to validate deploy, 4.5 out of 5 based on 2 ratings
VN:F [1.9.22_1171]
Rating: 4.5/5 (2 votes cast)

After you deploy your application on production environment, you should be sure that everything is up an running fine, and that your business still run as expected. You look at you logs and everything is fine: no error is present and the application flows regularly, then you run a manual test and it works as expected.

The day after you receive a phone call: you daily income from is half of the day before, and all the people in the business team is angry with you…

Antipattern: validate your deploy looking at your logs and running a manual test of your application.

The recipe: use business metrics to validate a deploy

At onebip.com we found that a better way to validate if a deploy can be considered successful or not is to measure some business kpi after a deploy for a few minutes (2 minutes is enough): if the kpi values still in the normal range, than the deploy is considered “good”, otherwise an automatic rollback to previous version is run.

Here an example from our phing build.xml file:

 <target name="validate-deploy-by-kpi" depends="prepare">
   <echo msg="Validation by kpi for deploy ${build.number} started..." />
   <exec command="php ${project.basedir}/code/libraries/Onebip/Monitoring/validateDeploy.php" outputProperty="validate.deploy.kpi.output" />
   <echo msg="validate.deploy.output: ${validate.deploy.kpi.output}" />
   <condition property="validate.kpi.failed">
     <contains substring="FAILED" string="${validate.deploy.kpi.output}" />
   </condition>
   <fail if="validate.kpi.failed" message="Validation by kpi on deploy ${build.number} failed. We should rollback" />
   <echo msg="Validation by kpi for deploy ${build.number} completed" />
 </target>

The KPIs used for deploy validation are the same that we use for h24x7application monitoring: we instrumented the software using some probes (for the curious people, some details are shared here on DZone) and we send application metrics to Datadog, as SaaS solution born to track metrics and events.

To validate the deploy, we just query the Datadog service (2 minutes after the deploy) using a custom php script able to manage the http authentication and retrieve the most important business metrics, then we check if the last values are inside a predetermined confidence band, and decide for the “go/no-go”.

Here an example: the 2 red vertical bars mark the start/end of the “validation by KPI” step

Conclusions

Instrument your software to track business metrics, and then use the same metrics to validate if a deploy can be considered successful or not.

 

VN:F [1.9.22_1171]
Rating: 4.5/5 (2 votes cast)

Simonluca Landi

Posts Twitter Facebook Google+

Mi chiamo Simonluca Landi e mi occupo di tecnologie informatiche. Cerco di sviluppare ed implementare soluzioni che tengano conto della “big picture“, sperimentando differenti tecnologie e linguaggi ed evitando le guerre di religione. Segui su Twitter @sll . Leggi tutto si di me qui

No Comments

Be the first to start the conversation.

Leave a Reply

Text formatting is available via select HTML. <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*