Scraping websites with Zend_Dom_Query

Today I stumbled upon an interesting and reportable scenario where I had to extract information of the weekly published Drum and Bass charts provided by BBC 1Xtra. As this information currently isn't available in any consumer friendly format like for example a RSS feed, I had to go that scraping route but didn't want to hustle with a regex approach. Since version 1.6.0 the Zend_Dom_Query component has been added to the framework mainly to support functional testing of MVC applications, but it also can be used for rolling custom website scrapers in a snap. Woot, perfect match!

The following code snippets are showing the Bbc_DnbCharts_Scraper class I came up with and an example of its usage. The class utilizes curl to read the website holding the desired data, which will be passed to Zend_Dom_Query to execute queries upon it. For querying the former loaded XHTML Document Object Model it's possible to either utilize XPath or CSS selectors. So I had to pick my poison, and decided to go with the CSS selectors as them were best suited for the document to query and will be more familiar to most jQuery or Prototype users. The query returns a result set of all matching DOMElements which are further unpuzzled via a private helper method returning just the desired charts data as shown in the closing listing. As you can see the implementation of the scraping can be done with a minimum of effort and these are exactly the moments I love the Zend Framework for.

<?php
require_once('Zend/Dom/Query.php');
/** 
 * 'Class-level' PHPDoc Block   
 */
class Bbc_DnbCharts_Scraper
{
    private $_url = null;
    private $_xhtml = null;

    /**
     * @param string $url
     */
    public function __construct($url)
    {
        $this->_url = $url;
    }
    /**
     * Scrapes off the drum and bass charts content from the BBC 1Xtra website.     
     *
     * @return array
     * @throws Exception
     */
    public function scrape()
    {
        try {
            $dom = new Zend_Dom_Query($this->_getXhtml());
        } catch (Exception $e) {
            throw $e;        
        }
        $results = $dom->query('div.chart div');
        $chartDetails = array();
        foreach ($results as $index => $result) {
            /* @var $result DOMElement */
            if ($result->nodeValue !== '') { //filter out <br /> element
                $chartDetails[] = $result->nodeValue;
            }
        }
        return $this->_unpuzzleChartDetails($chartDetails, true);
    }
    /**
     * Unpuzzles the chart details and groups them by their chart position, 
     * if desired with associative keys.
     *  
     * @param array $details
     * @param boolean $associative
     * @return array
     */
    private function _unpuzzleChartDetails(array $details, $associative = false)
    {
        if (0 === count($details)) {
            return array();
        } else {  
            $nextChartRank = 2;
            $charts = array();
            $groupedChartDetails = array();
            
            foreach ($details as $index => $chartDetail)
            { 
                if ($index <= $nextChartRank) {
                    $groupedChartDetails[] = $chartDetail;
                }
                if ($index == $nextChartRank) {
                    $nextChartRank+=3;
                    $charts[] = $groupedChartDetails;
                    unset($groupedChartDetails);
                }
            }
            if ($associative) {
                $associatives = array('artist', 'tune', 'label');
                foreach ($charts as $chartsIndex => $chart) {
                    unset($charts[$chartsIndex]);
                    foreach ($chart as $chartIndex => $chartDetails) {
                        $charts[$chartsIndex][$associatives[$chartIndex]] = 
                            $chartDetails;
                    }
                }
            }
            return $charts;
        }
    }
    /**
     * Gets the XHTML document via curl 
     * 
     * @return string
     * @throws Exception
     */
    private function _getXhtml()
    {
        $curl = curl_init();
        if (!$curl) {
            throw new Exception('Unable to init curl. ' . curl_error($curl));
        }
        curl_setopt($curl, CURLOPT_URL, $this->_url);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
        // Faking user agent
        curl_setopt($curl, CURLOPT_USERAGENT, 
            'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)');
        $xhtml = curl_exec($curl);
        if (!$xhtml) {
            throw new Exception('Unable to read XHTML. ' . curl_error($curl));
        }
        curl_close($curl);
        return $xhtml; 
    }
}

// Usage demo
$scraper = new Bbc_DnbCharts_Scraper('http://www.bbc.co.uk/1xtra/drumbass/chart/');
$charts = $scraper->scrape();

The closing code snippet shows an extract of the Drum and Bass charts from BBC 1Xtra scraped off around the 16th October 2008.

Array
(
    [0] => Array
        (
            [artist] => Chase & Status Ft Plan B
            [tune] => Pieces
            [label] => Ram Records
        )

    ...
    
    [9] => Array
        (
            [artist] => Zen
            [tune] => Full Effect
            [label] => Flipmode Audio
        )

)