Growling PHPUnit's test status

Two years ago I blogged about a Xinc (R.I.P?) plugin that growls each build status for any via Xinc continuously integrated project. Since I'm using PHPUnit more and more lately, especially in continuous testing sessions (sprints without hitting the continuous integration server), my dependence on a fast and more visual feedback loop rose. In this post I'll provide an easy solution that meets these requirements by utilizing PHPUnit's test listener feature.

What's the motivation, yo?

While doing story or feature sprints embedded in a continuous testing approach I first used a combination of stakeout.rb and PHPUnit's --colors option to radiate the tests status, but soon wasn't that satisfied with the chosen route as it happened that the console window got superimposed with other opened windows (e.g. API Browser, TextMate etc.) especially on my 13,3" MacBook.

To overcome this misery I decided to utilize PHPUnit's ability to write custom test listeners and to implement one that radiates the test status in a more prominent and sticky spot via Growl.

Implementing the Growl test listener

Similar to the ticket listener plugin mechanism I blogged about earlier PHPUnit also provides one for test listeners. This extension mechanism allows to bend the test result formatting and output to the given needs and scenarios a developer might face and therefore is a perfect match.

To customize the test feedback and visualization the test listener has to implement the provided PHPUnit_Framework_Testlistener interface. A few keystrokes later I ended up with the next shown implementation, which is also available via a GitHub gist, supporting the previous stated requirements.

<?php

class PHPUnit_Extensions_TestListener_GrowlTestListener 
    implements PHPUnit_Framework_Testlistener
{
    const TEST_RESULT_COLOR_RED = 'red';
    const TEST_RESULT_COLOR_YELLOW = 'yellow';
    const TEST_RESULT_COLOR_GREEN = 'green';
    
    private $_errors = array();
    private $_failures = array();
    private $_incompletes = array();
    private $_skips = array();
    private $_tests = array();
    private $_suites = array();
    private $_endedSuites = 0;
    private $_assertionCount = 0;
    private $_startTime = 0;

    private $_successPicturePath = null;
    private $_incompletePicturePath = null;
    private $_failurePicturePath = null;

    /**
     * @param string $successPicturePath
     * @param string $incompletePicturePath
     * @param string $failurePicturePath
     */
    public function __construct($successPicturePath, $incompletePicturePath, 
        $failurePicturePath)
    {
        $this->_successPicturePath = $successPicturePath;
        $this->_incompletePicturePath = $incompletePicturePath;
        $this->_failurePicturePath = $failurePicturePath;
    }

    public function addError(PHPUnit_Framework_Test $test, Exception $e, $time)
    {
        $this->_errors[] = $test->getName();
    }
    
    public function addFailure(PHPUnit_Framework_Test $test, 
        PHPUnit_Framework_AssertionFailedError $e, $time) 
    {     
        $this->_failures[] = $test->getName();
    }
    
    public function addIncompleteTest(PHPUnit_Framework_Test $test, 
        Exception $e, $time)
    {
        $this->_incompletes[] = $test->getName();
    }
    
    public function addSkippedTest(PHPUnit_Framework_Test $test, 
        Exception $e, $time) 
    {
        $this->_skips[] = $test->getName();
    }
    
    public function startTest(PHPUnit_Framework_Test $test)
    {
    
    }
    
    public function endTest(PHPUnit_Framework_Test $test, $time) 
    { 
        $this->_tests[] = array('name' => $test->getName(), 
            'assertions' => $test->getNumAssertions()
        );
        $this->_assertionCount+= $test->getNumAssertions();
    }
    
    public function startTestSuite(PHPUnit_Framework_TestSuite $suite)
    {
        if (count($this->_suites) === 0) {
            PHP_Timer::start();
        }
        $this->_suites[] = $suite->getName();
    }
    
    public function endTestSuite(PHPUnit_Framework_TestSuite $suite)
    {
        $this->_endedSuites++;
        
        if (count($this->_suites) <= $this->_endedSuites)
        {
            $testTime = PHP_Timer::secondsToTimeString(
                PHP_Timer::stop());

            if ($this->_isGreenTestResult()) {
                $resultColor = self::TEST_RESULT_COLOR_GREEN;
            }
            if ($this->_isRedTestResult()) {
                $resultColor = self::TEST_RESULT_COLOR_RED;
            }
            if ($this->_isYellowTestResult()) {
                $resultColor = self::TEST_RESULT_COLOR_YELLOW;
            }

            $suiteCount = count($this->_suites);
            $testCount = count($this->_tests);
            $failureCount = count($this->_failures);
            $errorCount = count($this->_errors);
            $incompleteCount = count($this->_incompletes);
            $skipCount = count($this->_skips);

            $resultMessage = '';

            if ($suiteCount > 1) {
                $resultMessage.= "Suites: {$suiteCount}, ";
            }
            $resultMessage.= "Tests: {$testCount}, ";
            $resultMessage.= "Assertions: {$this->_assertionCount}";

            if ($failureCount > 0) {
                $resultMessage.= ", Failures: {$failureCount}";
            } 

            if ($errorCount > 0) {
                $resultMessage.= ", Errors: {$errorCount}";
            }

            if ($incompleteCount > 0) {
                $resultMessage.= ", Incompletes: {$incompleteCount}";
            }

            if ($skipCount > 0) {
                $resultMessage.= ", Skips: {$skipCount}";
            }
            $resultMessage.= " in {$testTime}.";
            $this->_growlnotify($resultColor, $resultMessage);
        }
    }

    /**
     * @param string $resultColor
     * @param string $message
     * @param string $sender The name of the application that sends the notification
     * @throws RuntimeException When growlnotify is not available
     */
    private function _growlnotify($resultColor, $message = null, $sender = 'PHPUnit')
    {
        if ($this->_isGrowlnotifyAvailable() === false) {
            throw new RuntimeException('The growlnotify tool is not available');
        }
        $notificationImage = $this->_getNotificationImageByResultColor(
            $resultColor);
        $command = "growlnotify -w -s -m '{$message}' "
                 . "-n '{$sender}' "
                 . "-p 2 --image {$notificationImage}";
        exec($command, $response, $return);
    }

    /**
     * @return boolean
     */
    private function _isGrowlnotifyAvailable()
    {
        exec('growlnotify -v', $reponse, $status);
        return ($status === 0);
    }

    /**
     * @param string $color 
     * @return string
     */
    private function _getNotificationImageByResultColor($color)
    {
        switch ($color) {
            case self::TEST_RESULT_COLOR_RED:
                return $this->_failurePicturePath;
                break;
            case self::TEST_RESULT_COLOR_GREEN:
                return $this->_successPicturePath;
                break;
            default:
                return $this->_incompletePicturePath;
        }
    }

    /**
     * @return boolean
     */
    private function _isGreenTestResult()
    {
        return count($this->_errors) === 0 && 
               count($this->_failures) === 0 &&
               count($this->_incompletes) === 0 &&
               count($this->_skips) === 0;
    }

    /**
     * @return boolean
     */
    private function _isRedTestResult()
    {
        return count($this->_errors) > 0 ||
               count($this->_failures) > 0;
    }

    /**
     * @return boolean
     */
    private function _isYellowTestResult()
    {
        return count($this->_errors) === 0 &&
               count($this->_failures) === 0 &&
               (count($this->_incompletes) > 0 ||
                count($this->_skips) > 0);
    }
}

Hooking the Growl test listener into the PHPUnit ecosystem

To make use of the just outlined test listener it's necessary to add an entry to PHPUnit's XML configuration file telling PHPUnit which test listener class to utilize and where it's located in the file system. In a next step the images for the three possible Growl notifications have to be added to the local file system, and as the Growl test listener constructor takes these as arguments they have also to be injected in the PHPUnit XML configuration file (i.e. phpunit-offline.xml). Take a peek yourself how this is done in the next listing.

<phpunit backupGlobals="false"
         backupStaticAttributes="true"
         bootstrap="bootstrap.php"
         colors="true"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         processIsolation="true"
         stopOnFailure="true"
         syntaxCheck="true"
         testSuiteLoaderClass="PHPUnit_Runner_StandardTestSuiteLoader">
  <testsuites> 
    <testsuite name="Zend_Service_GitHub Offline Testsuite">
      <directory>Zend/Service/GitHub</directory>
      <directory>Zend/Service</directory>
    </testsuite>
  </testsuites>
  <groups>
    <include>
      <group>offline</group>
    </include>
  </groups>
  <listeners>
    <listener class="PHPUnit_Extensions_TestListener_GrowlTestListener" 
              file="/Users/stolt/Work/GrowlTestListener.php">
     <arguments>
       <string>$HOME/Pictures/pass.png</string>
       <string>$HOME/Pictures/pending.png</string>
       <string>$HOME/Pictures/fail.png</string>
     </arguments>
    </listener>
  </listeners>
</phpunit>

Putting the Growl test listener to work

Attention shameless plug! As an example application for a continuous testing session I chose a Zend Framework Service component I'm currently working on. To set up the continuously testing workflow, stakeout.rb is still my #1 choice, but in a recent blog post Andy Stanberry shows another tool dubbed Kicker which seems to be coequal. The following console snippet shows in a concrete scenario how to utilize stakeout.rb to watch for any changes on the Zend_Service_GitHub component or it's backing tests which immediately trigger the test suite execution if one is detected.

stakeout.rb 'phpunit --configuration phpunit-offline.xml' **/*.{php} ../Zend/**/*.php ../Zend/Service/**/*.php

In the classic TDD cycle we start with a failing test. Creating the test, adding the assertions that the system under test (SUT) has to fulfill and saving the according test class automatically triggers the test suite execution which ends up in the next shown Growl notification.



Nest a très important client call comes in and since we are clever, a quick TextMate shortcut marks the currently worked on test as incomplete. This step might be a bit controversy as it's also suggested to leave the last worked on test broken, but I got to show you the pending/incomplete Growl notification ;D



After finishing the 'interruptive' client call aka context switch we can continue to work on the feature of the SUT until it fulfills the expected behavior which will be radiated via the next shown Growl notification. Happy Growl flavored testing!



* As you might notice in the shown Growl notification images there's a test suite count of 9 while we are only operating on a single one, this seems to be a possible PHPUnit bug, or just a misconfiguration of my testing environment.

In case you got a solution for this problem feel free to add an illuminating comment.