Monday, 10 October 2016

Eight knobs to adjust and improve your Travis CI builds

After having refactored several Travis CI configuration files over the last weeks, this post will provide eight adjustments or patterns immediately applicable for faster, changeable, and economic builds.

1. Reduce git clone depth

The first one is a simple configuration addition with a positive impact on the time and disk space consumption, which should be quite noticeable on larger code bases. Having this configured will enable shallow clones of the Git repository and reduce the clone depth from 50 to 2.

.travis.yml
git:
  depth: 2

2. Enable caching

The second one is also a simple configuration addition for caching the Composer dependencies of the system under build (SUB) or its result of static code analysis. Generally have a look if your used tools allow caching and if so cache away. This one deserves a shout out to @localheinz for teaching me about this one.

The next shown configuration excerpt assumes that you lint coding standard compliance with the PHP Coding Standards Fixer in version 2.0.0-alpha and have enable caching in its .php_cs configuration.

.travis.yml
cache:
  directories:
    - $HOME/.composer/cache
    - $HOME/.php-cs-fixer

3. Enforce contribution standards

This one might be a tad controversial, but after having had the joys of merging GitHub pull requests from a master branch I started to fail builds not coming from feature or topic branch with the next shown bash script. It's residing in an external bash script to avoid the risk of terminating the build process.

./bin/travis/fail-non-feature-topic-branch-pull-request
#!/bin/bash
set -e
if [[ $TRAVIS_PULL_REQUEST_BRANCH = master ]]; then
  echo "Please open pull request from a feature / topic branch.";
  exit 1;
fi

.travis.yml
script:
  - ./bin/travis/fail-non-feature-topic-branch-pull-request
The bash script could be extended to fail pull requests not following a branch naming scheme, e.g. feature- for feature additions or fix- for bug fixes, by evaluating the branch name. If this is a requirement for your builds you should also look into the blocklisting branches feature of Travis CI.

4. Configure PHP versions in an include

With configuring the PHP versions to build against in a matrix include it's much easier to inject enviroment variables and therewith configure the version specific build steps. You can even set multiple enviroment variables like done on the 7.0 version.

.travis.yml
env:
  global:
    - OPCODE_CACHE=apc

matrix:
  include:
    - php: hhvm
    - php: nightly
    - php: 7.1
    - php: 7.0
      env: DISABLE_XDEBUG=true LINT=true
    - php: 5.6
      env: 
      - DISABLE_XDEBUG=true

before_script:
  - if [[ $DISABLE_XDEBUG = true ]]; then
      phpenv config-rm xdebug.ini;
    fi
I don't know if enviroment variable injection is also possible with the minimalistic way to define the PHP versions list, so you should take that adjustment with a grain of salt.

It also seems like I stumbled upon a Travis CI bug where the global enviroment variable OPCODE_CACHE is lost, so add another grain of salt. To work around that possible bug the relevant configuration has to look like this, which sadly adds some duplication and might be unsuitable when dealing with a large amount of environment variables.

.travis.yml
matrix:
  include:
    - php: hhvm
      env: 
      - OPCODE_CACHE=apc
    - php: nightly
      env: 
      - OPCODE_CACHE=apc
    - php: 7.1
      env: 
      - OPCODE_CACHE=apc
    - php: 7.0
      env: OPCODE_CACHE=apc DISABLE_XDEBUG=true LINT=true
    - php: 5.6
      env: OPCODE_CACHE=apc DISABLE_XDEBUG=true

before_script:
  - if [[ $DISABLE_XDEBUG = true ]]; then
      phpenv config-rm xdebug.ini;
    fi

5. Only do static code analysis or code coverage measurement once

This one is for reducing the build duration and load by avoiding unnecessary build step repetition. It's achived by linting against coding standard violations or generating the code coverage for just a single PHP version per build, in most cases it will be the same for 5.6 or 7.0.

.travis.yml
matrix:
  include:
    - php: hhvm
    - php: nightly
    - php: 7.1
    - php: 7.0
      env: DISABLE_XDEBUG=true LINT=true
    - php: 5.6
      env: 
      - DISABLE_XDEBUG=true

script:
  - if [[ $LINT=true ]]; then
      composer cs-lint;
      composer test-test-with-coverage;
    fi

6. Only do release releated analysis and checks on tagged builds

This one is also for reducing the build duration and load by targeting the analysis and checks on tagged release builds. For example if you want to ensure that your CLI binary version, the one produced via the --version option, matches the Git repository version tag run this check only on tagged builds.

.travis.yml
script:
  - if [[ ! -z "$TRAVIS_TAG" ]]; then
      composer application-version-guard;
    fi

7. Run integration tests on very xth build

For catching breaking changes in interfaces or API's beyond your control it makes sense do run integration tests against them once in a while, but not on every single build, like shown in the next Travis CI configuration excerpt which runs the integration tests on every 50th build.

.travis.yml
script:
  - if [[ $(( $TRAVIS_BUILD_NUMBER % 50 )) = 0 ]]; then
      composer test-all;
    else
      composer test;
    fi

8. Utilise Composer scripts

The last one is all about improving the readability of the Travis CI configuration by extracting command configurations i.e. options into dedicated Composer scripts. This way the commands are also available during your development activitives and not hidden away in the .travis.yml file.

composer.json
{
    "__comment": "omitted other configuration",
    "scripts": {
        "test": "phpunit",
        "test-with-coverage": "phpunit --coverage-html coverage-reports",
        "cs-fix": "php-cs-fixer fix . -vv || true",
        "cs-lint": "php-cs-fixer fix --diff --verbose --dry-run"
    }
}
To ensure you don't end up with an invalid Travis CI configuration, which might be accidently committed, you can use composer-travis-lint a simple Composer script linting the .travis.yml with the help of the Travis CI API.

Happy refactoring.

Tuesday, 20 September 2016

Anatomy of a dope PHP package repository

While contributing to Construct, maintained by Jonathan Torres, I gathered some insights and learnings on the characteristics of a dope PHP package repository. This post summarises and illustrates these, so that PHP package develeopers have a complementary guideline to improve existing or imminent package repositories. Jonathan Reinink did a good job in putting the PHP package checklist out there which provides an incomplete, but solid quality checklist for open-source PHP packages.

I'll distill the characteristics of a dope PHP package repository by looking at the repository artifacts Construct can generate for you when starting the development of a new PHP project or micro-package. The following tree command output shows most of the elements this post will touch upon. The artifacts in parenthese are optional and configurable from Construct but can nonetheless have an import impact on the overall package quality.

├── <package-name>
│   ├── CHANGELOG.md
│   ├── (CONDUCT.md)
│   ├── composer.json
│   ├── composer.lock
│   ├── CONTRIBUTING.md
│   ├── (.editorconfig)
│   ├── (.env)
│   ├── (.env.example)
│   ├── (.git)
│   │   └── ...
│   ├── .gitattributes
│   ├── (.github)
│   │   ├── CONTRIBUTING.md
│   │   ├── ISSUE_TEMPLATE.md
│   │   └── PULL_REQUEST_TEMPLATE.md
│   ├── .gitmessage
│   ├── .gitignore
│   ├── (.lgtm)
│   ├── LICENSE.md
│   ├── (MAINTAINERS)
│   ├── (.php_cs)
│   ├── (phpunit.xml.dist)
│   ├── README.md
│   ├── (docs)
│   │   └── index.md
│   ├── src
│   │   └── Logger.php
│   ├── tests
│   │   └── LoggerTest.php
│   ├── .travis.yml
│   ├── (Vagrantfile)
│   └── vendor
│           └── ...

Definition of a dope PHP package repository

Before jumping into the details, let's define what could be considered as a dope package repository. Therefor, being lazy, I'm going to simply reword this classic quote from Michael Feathers
> Clean code is code that is written by someone who cares.
to
> A dope PHP package repository is one that is created and maintained by someone who cares.

Artifact categories

The next shown pyramid illustrates the three main categories the artifacts of a package repository will fall into.
First and most important there's the main sourcecode, it's tests or specs, and the documentation which could be dependent on it's size reside in a README.md section or inside a dedicated docs directory. Using a docs directory also allows publishing the documentation via GitHub pages. Other aspects of a package which should be documented are the chosen license, how to contribute to the package, possibly a code of conduct to comply with, and the changes made over the lifespan of the package.

Second there's the configuration for a myriad of tools like Git, GitHub, EditorConfig, Composer, the preferred testing framework, the preferred continuous inspection / integration platform such like Scrutinizer or Travis CI, and so forth.

The final category includes tools which ease the life of maintainers and potential contributors equally. These tools can be helpful for releasing new versions, enforcing coding standard compliance, or commit message quality and consistency.

Consistency

Sourcecode

All sourcecode and accompanying tests or specs should follow a coding standard (PSR-2) and have a consistent formatting style, there's nothing new here. The perfect place to communicate such requirements is the CONTRIBUTING.md file.

Tools like PHP Coding Standards Fixer or PHP_CodeSniffer in combination with a present configuration .php_cs|ruleset.xml.dist and a command wrapping Composer script are an ideal match to ease compliance. The Composer script cs-fix shown next will be available for maintainers and contributors alike.

composer.json
{
    "__comment": "omitted other configuration",
    "scripts": {
        "cs-fix": "php-cs-fixer fix . -vv || true"
    }
}
Consistent formatting styles like line endings, indentation style, and file encoding can be configured via an EditorConfig configuration residing in .editorconfig which will be used when supported by the IDE or text editor of choice.

Artifact naming and casing

Like sourcecode formatting and naming, repository artifacts should also follow a predictable naming scheme. All documentation files should have a consistent extension like .md or .rst and the casing should be consistent throughout the package repository. Comparing
├── <package-name>
│   ├── changelog.md
│   ├── code_of_conduct.md
│   ├── ...
│   ├── .github
│   │   └── ...
│   ├── LICENSE
│   ├── Readme.md
│   ├── roadmap.rst
to
├── <package-name>
│   ├── CHANGELOG.md
│   ├── CODE_OF_CONDUCT.md
│   ├── ...
│   ├── .github
│   │   └── ...
│   ├── LICENSE.md
│   ├── README.md
│   ├── ROADMAP.md
I would favour the later one anytime for it's much easier reading flow and pattern matchableness. The easier reading flow is achieved by the upper casing of the *.md files which also clearly communicates their documentation character.

The configuration files for tools which except the .dist file extension per default should all have such one like shown next.
├── <package-name>
│   ├── build.xml.dist
│   ├── phpunit.xml.dist
│   ├── ruleset.xml.dist
│   ├── ...

Commit message format

Next to the package's changelog, incrementally growing in the CHANGELOG.md file, the Git commit messages are an important source of change communication. Therefor they should also follow a consistent format which improves the reading flow while also leaving a professional impression. This format can be documented once again in the CONTRIBUTING.md file or even better be provided via a .gitmessage file residing in the package's Git repository.

Once more a Composer script, named configure-commit-template here, can ease configuration and if configured Git will use it's content when committing without the -m|--message and -F|--file option.

composer.json
{
    "__comment": "omitted other configuration",
    "scripts": {
        "configure-commit-template": "git config --add commit.template .gitmessage"
    }
}
To enforce commit message formatting adherence to the rules described by Chris Beams on a Git hook level, the git-lint-validators utility by Billie Thompson can be helpful.

Versioning

Release versions should follow the semantic versioning specification aka SemVer, once again there's nothing new here. When using version numbers in the sourcecode or CLI binaries, these should be in sync with the set Git tag. Tools like RMT or self-written tools should be utilised for this mundane task.

The next shown code illustrates such a simple self-written tool named application-version. It's main purpose is to set the provided version number in the CLI application's binary and avoid an application version and Git tag mismatch.

bin/application-version
#!/usr/bin/env php
<?php
$binApplicationName = '<bin-application-name>';
$binFile = __DIR__ . DIRECTORY_SEPARATOR . $binApplicationName;
list($void, $binFileRelative) = explode($binApplicationName, $binFile, 2);
$shortBinFilePath = $binApplicationName . $binFileRelative;

$options = getopt('v:ch', ['version:', 'current', 'verify-tag-match', 'help', 'current-raw']);

$help = <<<HELP
This command sets the version number in the {$shortBinFilePath} file:
Usage:
  application-version [options]
Options:
  -c, --current, --current-raw   The current version number
  --verify-tag-match             Verify application version and Git tag match
  -v, --version                  The version number to set
  -h, --help                     Display this help message
HELP;

if (array_key_exists('h', $options) || array_key_exists('help', $options)) {
    echo $help;
    exit(0);
}

/**
 * Return the application version.
 *
 * @param  string $binFile File holding the application version.
 * @return string
 */
function get_application_version($binFile) {
    $matches = [];
    $match = preg_match(
        '/(\d+\.)?(\d+\.)?(\*|\d+)/',
        file_get_contents($binFile),
        $matches
    );
    return trim($matches[0]);
}
/**
 * Return latest tagged version.
 *
 * @return string
 */
function get_latest_tagged_version() {
    exec('git describe --tags --abbrev=0', $output);
    return trim($output[0]);
}

if (array_key_exists('verify-tag-match', $options)) {
    $applicationVersion = 'v' . get_application_version($binFile);
    $latestGitTag = get_latest_tagged_version();
    if ($applicationVersion === $latestGitTag) {
        echo "The application version and Git tag match on {$latestGitTag}." . PHP_EOL;
        exit(0);
    }
    echo "The application version {$applicationVersion} and Git tag {$latestGitTag} don't match." . PHP_EOL;
    exit(1);
}

if (array_key_exists('current-raw', $options)) {
    echo get_application_version($binFile) . PHP_EOL;
    exit(0);
}

if (array_key_exists('c', $options) || array_key_exists('current', $options)) {
    $applicationVersion = 'v' . get_application_version($binFile);
    $latestGitTag = get_latest_tagged_version();
    echo "Current version set in {$shortBinFilePath} is {$applicationVersion}." . PHP_EOL;
    echo "Current tagged version {$latestGitTag}." . PHP_EOL;
    exit(0);
}

if ($options === []) {
    echo 'No options set.' . PHP_EOL;
    exit(1);
}

$version = isset($options['version']) ? trim($options['version']) : trim($options['v']);
$fileContent = file_get_contents($binFile);
$fileContent = preg_replace(
    '/(.*define.*VERSION.*)/',
    "define('VERSION', '$version');",
    $fileContent
);
file_put_contents($binFile, $fileContent);
echo "Set version in {$shortBinFilePath} to {$version}." . PHP_EOL;
exit(0);
The application-version tool could further be utilised in Travis CI builds, to avoid the earlier mentioned version differences, like shown in the next .travis.yml diggest. On an application version and Git tag mismatch the shown build script will break the build early.

.travis.yml
language: php

# omitted other configuration

script:
  # Verify application version and Git tag match
  - php bin/application-version --verify-tag-match
  # omitted other scripts

Lean builds

To speed up continuous integration builds, resource and time consuming extensions like Xdebug should be disabled when not required for measuring code coverage. The next shown before_script, tailored for Travis CI, is generated by Construct per default and might shave off a few build seconds and thereby provide a faster feedback.

.travis.yml
language: php

# omitted other configuration

before_script:
  - phpenv config-rm xdebug.ini || true
  # omitted other before_scripts
To reduce email traffic, the email notifications send by Travis CI should be reduced to a minimum like shown next, or dependent on your workflow the could be disabled at all.

.travis.yml
language: php

# omitted other configuration

notifications:
  email:
    on_success: never
Something I really would love to be supported by Travis CI is a feature to ignore a set of definable artifacts which could be configured in a .buildignore file or the like. This way wording or spelling changes on non build relevant artifacts like the README.md wouldn't trigger a build and misspend resources and energy. There's a related GitHub issue and here's hope it will be revisited in the near future.

Lean releases

To keep releases (or dists in Composer lingo) of PHP projects or micro-packages as lean as possible, their repositories should contain a complete and valid .gitattributes file. With such a file present all export-ignored files will be excluded from release archives and thereby save a significant amount of bandwith and energy.

The next code shows the content of such a .gitattributes file excluding non release relevant files like internal tools, configuration, and documentation artifacts. If for some reasons you require the complete source of a PHP project or micro-package you can bypass the default by using Composer's --prefer-source option.

.gitattributes
* text=auto eol=lf

.editorconfig export-ignore
.gitattributes export-ignore
.github/ export-ignore
.gitignore export-ignore
.gitmessage export-ignore
.php_cs export-ignore
.travis.yml export-ignore
bin/application-version export-ignore
bin/release-version export-ignore
bin/start-watchman export-ignore
CHANGELOG.md export-ignore
LICENSE.md export-ignore
phpunit.xml.dist export-ignore
README.md export-ignore
tests/ export-ignore
To validate the .gitattributes file of a PHP project or micro-package on the repository, Git HEAD, or build level the LeanPackageValidator CLI can be helpful.

Avoid badge posing

Badges, if used sparsely, are a handy tool for visualising some of the repository properties. It's definitely nice to immediately see the required PHP version, the current build status, or the latest version of the package as they save you manual look ups.

Badges showing the amount of downloads, code coverage, or the chosen license are in my opinion kind of poserish, they cause unnecessary requests to the badge service, and are in case of the license even obsolete.

Why should you care about the dopeness of a PHP package repository?

Creating and maintaining a dope PHP package repository might have a positive impact on several levels. It can earn you some valuable Brownie points or even provide a conversation gambit when doing job interviews, simply because you showcase professionalism.

Furthermore it's more likely to get valuable and high quality contributions from your second target audience when supportive documentation and tooling for things like issue creation, coding standards compliance, or Git commit message consistency are available.

It also might convince an end user, your main target audience, in using your package over a competitive one.

Le fini

So these were my recent insights and learnings on the anatomy of a dope PHP package repository. If you'r lucky to be attending this year's ZendCon, I recommend to catch Matthew Weier O'Phinney's session about Creating PHPantastic packages. I definitely be waiting for the related slides.

Happy packaging.

Friday, 22 April 2011

Enforcing target descriptions within build files with a Git hook

When automating mundane tasks of a project or development environment with a build tool like Phing or Ant, the driving build file will naturally accumulate several targets and tasks over time. To ease the build file acceptance within a team and at a later stage also the contribution rate by team members, it's crucial that all build targets have a description attribute to provide at least a rough outline of the build features at hand. When these attributes are in place the (potential) build file user will get such an outline by executing the build tool's list command (phing -l or ant -p). To get a better picture of the problem at hand imagine a project poorly covered with tests and your personal attitude towards extending it or just take a peek at the screenshot below showing a very poorly documented build file.

A poorly documented build file in Phing's list view

To overcome this accumulation of some sort of technical debt (i.e. poorly documented targets) there are various options at hand. The first one, not covered in this blog post, would be to add a pursuant test which verifies the existence of a description for every target/task of the build file under test. As it's very uncommon, at least from what I've heard, to have your build files covered by tests; the next thinkable approach would be to use a Git pre-commit hook to guard your repository/ies against the creeping in of such poorly documented build files.

The next listing shows such a Git hook (also available via GitHub) scribbled away in PHP, which detects any build file(s) following a common build file naming schema (i.e. build.xml|build.xml.dist|personal-build.xml|…) , prior to the actual commit. For every target element in the detected build file(s) it's then verified that it has a description attribute and that it's actual content is long enough to carry some meaning. If one of those two requirements aren't met, the commit is rejected while revealing the build file smells to the committer, so she can fix it, as shown in the outro screenshot. Happy build file sniffing.

#!/usr/bin/php
<?php
define('DEPENDENT_EXTENSION', 'SimpleXML');

if (!extension_loaded(DEPENDENT_EXTENSION)) {
    $consoleMessage = sprintf(
        "Skipping build file checks as the '%s' extension isn't available.", 
        DEPENDENT_EXTENSION
    );
    echo $consoleMessage . PHP_EOL;
    exit(0);
}

define('MIN_TARGET_DESCRIPTION_LENGTH', 10);
define('TARGET_DESCRIPTION_ATTRIBUTE', 'description');
define('TARGET_NAME_ATTRIBUTE', 'name');
define('CHECK_DESCRIPTION_LENGTH', true);

$possibleBuildFileNames = array(
    'build.xml.dist',
    'build.xml-dist',
    'build-dist.xml',
    'build.xml',
    'personal-build.xml'
);

$violations = getAllBuildFileViolationsOfCommit($possibleBuildFileNames);
fireBackPossibleViolationsAndExitAccordingly($violations);

function getAllBuildFileViolationsOfCommit(array $possibleBuildFileNames)
{
    $filesOfCommit = array();
    $gitCommand = 'git diff --cached --name-only';
    
    exec($gitCommand, $filesOfCommit, $commandReturnCode);
    
    $allViolations = array();
    foreach ($filesOfCommit as $file) {
      if (in_array(basename($file), $possibleBuildFileNames)) {
          $violations = checkBuildFileForViolations($file);
          if (count($violations) > 0) {
            $allViolations[$file] = $violations;
          }
      }
    }

    return $allViolations;
}

/**
 *  @param  array $allViolations
 *  @return void
 */
function fireBackPossibleViolationsAndExitAccordingly(array $allViolations)
{
    if (count($allViolations) > 0) {
        foreach ($allViolations as $buildFile => $violations) {

            $buildFileConsoleMessageHeader = sprintf("Build file '%s':", $buildFile);
            echo $buildFileConsoleMessageHeader . PHP_EOL;

            foreach ($violations as $violationMessage) {
                $buildFileConsoleMessageLine = sprintf(" + %s", $violationMessage);
                echo $buildFileConsoleMessageLine . PHP_EOL;
            }
        }
        if (count($allViolations) > 1) {
            $rejectCommitConsoleMessage = sprintf(
                "Therefore rejecting the commit of build files [ %s ].", 
                implode(', ', array_keys($allViolations))
            );
        } else {
            $rejectCommitConsoleMessage = sprintf(
                "Therefore rejecting the commit of build file [ %s ].", 
                implode(', ', array_keys($allViolations))
            );
        }

        echo $rejectCommitConsoleMessage . PHP_EOL;
        exit(1);
    }
    exit(0);
}
/**
 *  @param  string $buildfile
 *  @return array
 */
function checkBuildFileForViolations($buildFile) {
    if (!file_exists($buildFile)) {
        return array();
    }

    $buildfileXml = file_get_contents($buildFile);
    $buildXml = new SimpleXMLElement($buildfileXml);
    $allBuildTargets = $buildXml->xpath("//target");
    $violations = array();

    if (count($allBuildTargets) > 0) {

        $targetsWithNoDescription = $targetsWithTooShortDescription = array();

        foreach ($allBuildTargets as $buildTarget) {

            $actualTragetAttributes = $buildTarget->attributes();
            $allUsedTragetAttributes = array();
            $actualTargetName = null;

            foreach ($actualTragetAttributes as $attribute => $value) {
                $allUsedTragetAttributes[] = $attribute;

                if ($attribute === TARGET_NAME_ATTRIBUTE) {
                    $actualTargetName = $value;
                }

                if (CHECK_DESCRIPTION_LENGTH === true && 
                    $attribute === TARGET_DESCRIPTION_ATTRIBUTE && 
                    strlen($value) < MIN_TARGET_DESCRIPTION_LENGTH) 
                {
                    $targetsWithTooShortDescription[] = $actualTargetName;
                }
            }   

            if (!in_array(TARGET_DESCRIPTION_ATTRIBUTE, $allUsedTragetAttributes)) {
                $targetsWithNoDescription[] = $actualTargetName;
            }
        }
        if (count($targetsWithNoDescription) > 0) {
            if (count($targetsWithNoDescription) > 1) {
                $violations[] = sprintf(
                    "Build targets [ %s ] don't have mandatory descriptions.", 
                    implode(', ', $targetsWithNoDescription)
                );
            } else {
                $violations[] = sprintf(
                    "Build target [ %s ] doesn't have a mandatory description.", 
                    implode(', ', $targetsWithNoDescription)
                );
            }
        }

        if (count($targetsWithTooShortDescription) > 0) {
            if (count($targetsWithTooShortDescription) > 1) {
                $violations[] = sprintf(
                    "Build targets [ %s ] don't have an adequate target description length.", 
                    implode(', ', $targetsWithTooShortDescription),
                    MIN_TARGET_DESCRIPTION_LENGTH
                );
            } else {
                $violations[] = sprintf(
                    "Build target [ %s ] doesn't have an adequate target description length.", 
                    implode(', ', $targetsWithTooShortDescription),
                    MIN_TARGET_DESCRIPTION_LENGTH
                );
            }
        }
    }
    return $violations;
}
Non-Descriptive Phing build files rejected by a Git hook

Saturday, 20 November 2010

Measuring & displaying Phing build times with buildhawk

Recently I installed a Ruby gem called buildhawk which allows to measure and display the build times of Rake driven builds. As I like the idea behind this tool a lot but mostly use Phing for build orchestration, it was time to explore the possibility to interconnect them both. In this blog post I'll show an implementation of an apposite Phing Logger gathering the buildhawk compatible build times via git note(s) and how to put the interplay between those two tools to work.

Logging on

As mentioned above the build time of each build is stored as a git note and associated to the repository's HEAD, reflecting the current state of the system under build (SUB), which assumes that the SUB is versioned via Git. The next shown Phing Logger (i.e. BuildhawkLogger) grabs the overall build time by hooking into the buildFinished method of the extended DefaultLogger class, transforms it into a buildhawk specific format and finally adds it as a git note.
<?php

require_once 'phing/listener/DefaultLogger.php';

/**
 *  Writes a build event to the console and store the build time as a git notes in the   
 *  project's repository HEAD.
 *
 *  @author    Raphael Stolt <raphael.stolt@gmail.com>
 *  @see       BuildEvent
 *  @link      https://github.com/xaviershay/buildhawk Buildhawk on GitHub
 *  @package   phing.listener
 */
class BuildhawkLogger extends DefaultLogger {
    
    /**
     *  @var string
     */
    private $_gitNotesCommandResponse = null;

    /**
     *  Behaves like the original DefaultLogger, plus adds the total build time 
     *  as a git note to current repository HEAD.
     *
     *  @param  BuildEvent $event
     *  @see    BuildEvent::getException()
     *  @see    DefaultLogger::buildFinished
     *  @link   http://www.kernel.org/pub/software/scm/git/docs/git-notes.html
     */
    public function buildFinished(BuildEvent $event) {
        parent::buildFinished($event);
        if ($this->_isProjectGitDriven($event)) {
            $error = $event->getException();
            if ($error === null) {
                $buildtimeForBuildhawk = $this->_formatBuildhawkTime(
                    Phing::currentTimeMillis() - $this->startTime
                );
                if (!$this->_addBuildTimeAsGitNote($buildtimeForBuildhawk)) {
                    $message = sprintf(
                        "Failed to add git note due to '%s'",
                        $this->_gitNotesCommandResponse
                    );
                    $this->printMessage($message, $this->err, Project::MSG_ERR);
                }
            }
        }
    }
    
    /**
     * Checks (rudimentary) if the project is Git driven
     *
     *  @param  BuildEvent $event
     *  @return boolean
     */
    private function _isProjectGitDriven(BuildEvent $event)
    {
        $project = $event->getProject();
        $projectRelativeGitDir = sprintf(
            '%s/.git', $project->getBasedir()->getPath()
        );
        return file_exists($projectRelativeGitDir) && is_dir($projectRelativeGitDir);
    }
    
    /**
     *  Formats a time micro integer to buildhawk readable format.
     *
     *  @param  integer The time stamp
     */
    private function _formatBuildhawkTime($micros) {
        return sprintf("%0.3f", $micros);
    }
    
    /**
     *  Adds the build time as a git note to the current repository HEAD
     *
     *  @param  string  $buildTime The build time of the build
     *  @return mixed   True on sucess otherwise the command failure response
     */
    private function _addBuildTimeAsGitNote($buildTime) {
        $gitNotesCommand = sprintf(
            "git notes --ref=buildtime add -f -m '%s' HEAD 2>&1",
            $buildTime
        );
        $gitNotesCommandResponse = exec($gitNotesCommand, $output, $return);
        if ($return !== 0) {
            $this->_gitNotesCommandResponse = $gitNotesCommandResponse;
            return false;
        }
        return true;
    }
}

Putting the Logger to work

As the buildhawk logger is available via GitHub you can easily grab it by issuing sudo curl -s http://gist.github.com/raw/707868/BuildhawkLogger.php -o $PHING_HOME/listener/BuildhawkLogger.php. The next step, making the build times loggable, is achieved by using the -logger command line argument of the Phing Cli and specifying the buildhawk logger name or the path to it. In case you want the buildhawk logger to be used per default (it behaves like the default logger if the SUB isn't Git driven/managed) you can also add it to the Phing shell script.

The next console command issued in the directory of the SUB shows a Phing call utilizing the BuildhawkLogger, assumed it has been installed at $PHING_HOME/listener/BuildhawkLogger.php and not been made the default logger.
phing -logger phing.listener.BuildhawkLogger

Looking at them Phing build times

Now it's time to switch to buildhawk and let it finally perform it's designated task, rendering an with the commit SHAs, commit messages, and build times fed Erb template into an informative, viewable HTML page. To install it you simply have to run sudo gem install buildhawk and you're good to go.

The next console command shows the buildhawk call issued in the SUB's directory to produce it's build time report page.
buildhawk --title 'Examplr' > examplr-build-times.html
The outro screenshot below gives you a peek at a rendered build time report.Buildhawk report for a Phing driven build

Thursday, 3 June 2010

Growling PHPUnit's test status

PHPUnit Growl TestListenerTwo years ago I blogged about a Xinc (R.I.P?) plugin that growls each build status for any via Xinc continuously integrated project. Since I'm using PHPUnit more and more lately, especially in continuous testing sessions (sprints without hitting the continuous integration server), my dependence on a fast and more visual feedback loop rose. In this post I'll provide an easy solution that meets these requirements by utilizing PHPUnit's test listener feature.

What's the motivation, yo?

While doing story or feature sprints embedded in a continuous testing approach I first used a combination of stakeout.rb and PHPUnit's --colors option to radiate the tests status, but soon wasn't that satisfied with the chosen route as it happened that the console window got superimposed with other opened windows (e.g. API Browser, TextMate etc.) especially on my 13,3" MacBook.

To overcome this misery I decided to utilize PHPUnit's ability to write custom test listeners and to implement one that radiates the test status in a more prominent and sticky spot via Growl.

Implementing the Growl test listener

Similar to the ticket listener plugin mechanism I blogged about earlier PHPUnit also provides one for test listeners. This extension mechanism allows to bend the test result formatting and output to the given needs and scenarios a developer might face and therefore is a perfect match.

To customize the test feedback and visualization the test listener has to implement the provided PHPUnit_Framework_Testlistener interface. A few keystrokes later I ended up with the next shown implementation, which is also available via a GitHub gist, supporting the previous stated requirements.
<?php

class PHPUnit_Extensions_TestListener_GrowlTestListener 
    implements PHPUnit_Framework_Testlistener
{
    const TEST_RESULT_COLOR_RED = 'red';
    const TEST_RESULT_COLOR_YELLOW = 'yellow';
    const TEST_RESULT_COLOR_GREEN = 'green';
    
    private $_errors = array();
    private $_failures = array();
    private $_incompletes = array();
    private $_skips = array();
    private $_tests = array();
    private $_suites = array();
    private $_endedSuites = 0;
    private $_assertionCount = 0;
    private $_startTime = 0;

    private $_successPicturePath = null;
    private $_incompletePicturePath = null;
    private $_failurePicturePath = null;

    /**
     * @param string $successPicturePath
     * @param string $incompletePicturePath
     * @param string $failurePicturePath
     */
    public function __construct($successPicturePath, $incompletePicturePath, 
        $failurePicturePath)
    {
        $this->_successPicturePath = $successPicturePath;
        $this->_incompletePicturePath = $incompletePicturePath;
        $this->_failurePicturePath = $failurePicturePath;
    }

    public function addError(PHPUnit_Framework_Test $test, Exception $e, $time)
    {
        $this->_errors[] = $test->getName();
    }
    
    public function addFailure(PHPUnit_Framework_Test $test, 
        PHPUnit_Framework_AssertionFailedError $e, $time) 
    {     
        $this->_failures[] = $test->getName();
    }
    
    public function addIncompleteTest(PHPUnit_Framework_Test $test, 
        Exception $e, $time)
    {
        $this->_incompletes[] = $test->getName();
    }
    
    public function addSkippedTest(PHPUnit_Framework_Test $test, 
        Exception $e, $time) 
    {
        $this->_skips[] = $test->getName();
    }
    
    public function startTest(PHPUnit_Framework_Test $test)
    {
    
    }
    
    public function endTest(PHPUnit_Framework_Test $test, $time) 
    { 
        $this->_tests[] = array('name' => $test->getName(), 
            'assertions' => $test->getNumAssertions()
        );
        $this->_assertionCount+= $test->getNumAssertions();
    }
    
    public function startTestSuite(PHPUnit_Framework_TestSuite $suite)
    {
        if (count($this->_suites) === 0) {
            PHP_Timer::start();
        }
        $this->_suites[] = $suite->getName();
    }
    
    public function endTestSuite(PHPUnit_Framework_TestSuite $suite)
    {
        $this->_endedSuites++;
        
        if (count($this->_suites) <= $this->_endedSuites)
        {
            $testTime = PHP_Timer::secondsToTimeString(
                PHP_Timer::stop());

            if ($this->_isGreenTestResult()) {
                $resultColor = self::TEST_RESULT_COLOR_GREEN;
            }
            if ($this->_isRedTestResult()) {
                $resultColor = self::TEST_RESULT_COLOR_RED;
            }
            if ($this->_isYellowTestResult()) {
                $resultColor = self::TEST_RESULT_COLOR_YELLOW;
            }

            $suiteCount = count($this->_suites);
            $testCount = count($this->_tests);
            $failureCount = count($this->_failures);
            $errorCount = count($this->_errors);
            $incompleteCount = count($this->_incompletes);
            $skipCount = count($this->_skips);

            $resultMessage = '';

            if ($suiteCount > 1) {
                $resultMessage.= "Suites: {$suiteCount}, ";
            }
            $resultMessage.= "Tests: {$testCount}, ";
            $resultMessage.= "Assertions: {$this->_assertionCount}";

            if ($failureCount > 0) {
                $resultMessage.= ", Failures: {$failureCount}";
            } 

            if ($errorCount > 0) {
                $resultMessage.= ", Errors: {$errorCount}";
            }

            if ($incompleteCount > 0) {
                $resultMessage.= ", Incompletes: {$incompleteCount}";
            }

            if ($skipCount > 0) {
                $resultMessage.= ", Skips: {$skipCount}";
            }
            $resultMessage.= " in {$testTime}.";
            $this->_growlnotify($resultColor, $resultMessage);
        }
    }

    /**
     * @param string $resultColor
     * @param string $message
     * @param string $sender The name of the application that sends the notification
     * @throws RuntimeException When growlnotify is not available
     */
    private function _growlnotify($resultColor, $message = null, $sender = 'PHPUnit')
    {
        if ($this->_isGrowlnotifyAvailable() === false) {
            throw new RuntimeException('The growlnotify tool is not available');
        }
        $notificationImage = $this->_getNotificationImageByResultColor(
            $resultColor);
        $command = "growlnotify -w -s -m '{$message}' "
                 . "-n '{$sender}' "
                 . "-p 2 --image {$notificationImage}";
        exec($command, $response, $return);
    }

    /**
     * @return boolean
     */
    private function _isGrowlnotifyAvailable()
    {
        exec('growlnotify -v', $reponse, $status);
        return ($status === 0);
    }

    /**
     * @param string $color 
     * @return string
     */
    private function _getNotificationImageByResultColor($color)
    {
        switch ($color) {
            case self::TEST_RESULT_COLOR_RED:
                return $this->_failurePicturePath;
                break;
            case self::TEST_RESULT_COLOR_GREEN:
                return $this->_successPicturePath;
                break;
            default:
                return $this->_incompletePicturePath;
        }
    }

    /**
     * @return boolean
     */
    private function _isGreenTestResult()
    {
        return count($this->_errors) === 0 && 
               count($this->_failures) === 0 &&
               count($this->_incompletes) === 0 &&
               count($this->_skips) === 0;
    }

    /**
     * @return boolean
     */
    private function _isRedTestResult()
    {
        return count($this->_errors) > 0 ||
               count($this->_failures) > 0;
    }

    /**
     * @return boolean
     */
    private function _isYellowTestResult()
    {
        return count($this->_errors) === 0 &&
               count($this->_failures) === 0 &&
               (count($this->_incompletes) > 0 ||
                count($this->_skips) > 0);
    }
}

Hooking the Growl test listener into the PHPUnit ecosystem

To make use of the just outlined test listener it's necessary to add an entry to PHPUnit's XML configuration file telling PHPUnit which test listener class to utilize and where it's located in the file system. In a next step the images for the three possible Growl notifications have to be added to the local file system, and as the Growl test listener constructor takes these as arguments they have also to be injected in the PHPUnit XML configuration file (i.e. phpunit-offline.xml). Take a peek yourself how this is done in the next listing.
<phpunit backupGlobals="false"
         backupStaticAttributes="true"
         bootstrap="bootstrap.php"
         colors="true"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         processIsolation="true"
         stopOnFailure="true"
         syntaxCheck="true"
         testSuiteLoaderClass="PHPUnit_Runner_StandardTestSuiteLoader">
  <testsuites> 
    <testsuite name="Zend_Service_GitHub Offline Testsuite">
      <directory>Zend/Service/GitHub</directory>
      <directory>Zend/Service</directory>
    </testsuite>
  </testsuites>
  <groups>
    <include>
      <group>offline</group>
    </include>
  </groups>
  <listeners>
    <listener class="PHPUnit_Extensions_TestListener_GrowlTestListener" 
              file="/Users/stolt/Work/GrowlTestListener.php">
     <arguments>
       <string>$HOME/Pictures/pass.png</string>
       <string>$HOME/Pictures/pending.png</string>
       <string>$HOME/Pictures/fail.png</string>
     </arguments>
    </listener>
  </listeners>
</phpunit>

Putting the Growl test listener to work

Attention shameless plug! As an example application for a continuous testing session I chose a Zend Framework Service component I'm currently working on. To set up the continuously testing workflow, stakeout.rb is still my #1 choice, but in a recent blog post Andy Stanberry shows another tool dubbed Kicker which seems to be coequal. The following console snippet shows in a concrete scenario how to utilize stakeout.rb to watch for any changes on the Zend_Service_GitHub component or it's backing tests which immediately trigger the test suite execution if one is detected.
stakeout.rb 'phpunit --configuration phpunit-offline.xml' **/*.{php} ../Zend/**/*.php ../Zend/Service/**/*.php
In the classic TDD cycle we start with a failing test. Creating the test, adding the assertions that the system under test (SUT) has to fulfill and saving the according test class automatically triggers the test suite execution which ends up in the next shown Growl notification.

Growl notice for failed tests

Nest a très important client call comes in and since we are clever, a quick TextMate shortcut marks the currently worked on test as incomplete. This step might be a bit controversy as it's also suggested to leave the last worked on test broken, but I got to show you the pending/incomplete Growl notification ;D

Growl notice for incomplete tests

After finishing the 'interruptive' client call aka context switch we can continue to work on the feature of the SUT until it fulfills the expected behavior which will be radiated via the next shown Growl notification. Happy Growl flavored testing!

Growl notice for successful tests

* As you might notice in the shown Growl notification images there's a test suite count of 9 while we are only operating on a single one, this seems to be a possible PHPUnit bug, or just a misconfiguration of my testing environment.

In case you got a solution for this problem feel free to add an illuminating comment.