Taking time to write about development

Reading a websites contents using PHP and XPath

Following on from my previous post Scraping data with PHP and cURL today I’m going to show you about reading a websites contents using PHP and XPath.

One of the biggest problems I have faced when dealing with a scraped web page (read: block of HTML) is how to go about getting specific text elements out of it. Several years ago I would’ve gone with regex. This was a hassle to me and I’ve still yet to get my head around how it correctly works. These days though, I have discovered a useful little thing called XPath.

Basically, how it works is, you put the HTML in a DOMDocument and then you can use XPath to navigate through elements and attributes of the document getting the data of them in the process.

Using PHP and XPath

Now if we were to consider the following HTML:

And now, lets say I wanted to get the text of the rightDiv. This is how we would go about it:

Simply, it will load the $content (HTML) into a DOMDocument and then using XPath it looks for the for the div with the id of ‘mainDiv’ and then inside of ‘mainDiv’ it looks for the div with the id of ‘rightDiv’. Finally it then gets the text of the ‘p’ element.

Now that is one of the more straight forward scenarios and is made easier by the fact that most of the different elements have custom IDs like ‘mainDiv’ or ‘rightDiv’. If we were to look at the same HTML block again but this time without IDs, this is how it would look.

This time lets get the src of the image in the former leftDiv.

As you can see, the difference this time is that instead of using the divs ID, we used the elements number. Inside the body, it looks for the first div, and then inside of that it again looks for the first child div before returning the src of the img element.

Again though, this is probably another pretty simple solution but definitely gives you an idea of how XPath works and some of the ways it can be used. The next scenario is basically the one I came face to face with, and that scenario is one in which we have multiple divs with the same name. Consider this html:

So here we have a webpage that displays a bunch of people with information such as their name, age, weight, and picture. Now as someone who is interested in getting the information of all of these people, I need to scrape the website and then using XPath I can loop through each person and get it.

To do this, we get everything inside the ‘container’ div and then loop through each individual ‘person’ div. Then using XPath on the ‘person’ div we can use the above examples to get the information we want. Finally we store all the results in an array.

There you have it. If you run a print_r on the results array you should have yourself a bunch values containing the information of Bob, Stacy, and John.

Now a totally awesome cheat for this solution is to use the Google Chrome XPath Helper extension. All you need to do is hold down the shift key as you hover over webpage elements. This in turn will return the XPath query which is a HUGE help!

As always feel free to comment and or improve on the above. Also, don’t forget to follow me @JAGracie.

, , , ,

2 thoughts on “Reading a websites contents using PHP and XPath

Leave a Reply

Your email address will not be published. Required fields are marked *