Screen Scraping with Goutte

We all have been in situations were we need content or information from a connected website, but have no access to a REST Api or any other backend feed.

In these cases screen scraping is the only option to get needed information to finalize an integration. You can do that directly in CURL, but that can be tedious. Far easier to use a nicely packaged solution that combines a component that simulates web browser behavior and a component that eases DOM navigation for HTML and XML documents. Meet Goutte!

STEP 0

Install via composer.

composer require fabpot/goutte

1	composer require fabpot/goutte

STEP 1

$client = new Client();
$crawler = $client->request('GET', 'https://www.page/login.php');

// select the form and fill in some values
$form = $crawler->selectButton('Login')->form();
$form['f_loginname'] = 'HelloMe';
$form['f_loginpass'] = 'securepass';

// submit that form
$crawler = $client->submit($form);

// go to next page    
$crawler = $client->request('GET', 'https://www.page.de/overview.php');

$client = new Client();

$crawler = $client->request('GET', 'https://www.page/login.php');

// select the form and fill in some values

$form = $crawler->selectButton('Login')->form();

$form['f_loginname'] = 'HelloMe';

$form['f_loginpass'] = 'securepass';

// submit that form

$crawler = $client->submit($form);

// go to next page

$crawler = $client->request('GET', 'https://www.page.de/overview.php');

STEP 2

Get the data you need.

// loop over html and filter out what you need

$crawler->filter('table.clients tr')->each(function ($node) {

  $node->filter('td')->each(function ($sub_node) {
    echo $sub_node->html();
  }
}

// loop over html and filter out what you need