Screen Scraping with Goutte

We all have been in situations were we need content or information from a connected website, but have no access to a REST Api or any other backend feed.

In these cases screen scraping is the only option to get needed information to finalize an integration. You can do that directly in CURL, but that can be tedious. Far easier to use a nicely packaged solution that combines a component that simulates web browser behavior and a component that eases DOM navigation for HTML and XML documents. Meet Goutte!

STEP 0

Install via composer.

composer require fabpot/goutte

STEP 1

Login into a website and navigate to the page that has your needed information

$client = new Client();


$crawler = $client->request('GET', 'https://www.page/login.php');

 
// select the form and fill in some values

$form = $crawler->selectButton('Login')->form();


$form['f_loginname'] = 'HelloMe';


$form['f_loginpass'] = 'securepass';

 
// submit that form

$crawler = $client->submit($form);

 
// go to next page    

$crawler = $client->request('GET', 'https://www.page.de/overview.php');

STEP 2

Get the data you need.

// loop over html and filter out what you need
 

$crawler->filter('table.clients tr')->each(function ($node) {

 
  $node->filter('td')->each(function ($sub_node) {

    echo $sub_node->html();

  }

}

Goutte @ Github
BrowserKit Documentation
DOM Crawler Documentation

Enjoy coding …