Development - Devhour https://www.devhour.net Taking time to write about development Tue, 12 Mar 2024 10:01:14 +0000 en-US hourly 1 https://wordpress.org/?v=6.4.3 https://www.devhour.net/wp-content/uploads/2024/03/cropped-devhourlogo-32x32.png Development - Devhour https://www.devhour.net 32 32 Reading a websites contents using PHP and XPath https://www.devhour.net/reading-a-websites-contents-using-php-and-xpath/?utm_source=rss&utm_medium=rss&utm_campaign=reading-a-websites-contents-using-php-and-xpath https://www.devhour.net/reading-a-websites-contents-using-php-and-xpath/#respond Mon, 16 Sep 2013 12:32:00 +0000 https://www.devhour.net/?p=32 Following on from my previous post Scraping data with PHP and cURL today I’m going to show you about reading a websites contents using PHP and XPath. One of the biggest problems I have faced when dealing with a scraped web page (read: block of HTML) is how to go about getting specific text elements out of it. Several years […]

The post Reading a websites contents using PHP and XPath first appeared on Devhour.

]]> Following on from my previous post Scraping data with PHP and cURL today I’m going to show you about reading a websites contents using PHP and XPath.

One of the biggest problems I have faced when dealing with a scraped web page (read: block of HTML) is how to go about getting specific text elements out of it. Several years ago I would’ve gone with regex. This was a hassle to me and I’ve still yet to get my head around how it correctly works. These days though, I have discovered a useful little thing called XPath.

Basically, how it works is, you put the HTML in a DOMDocument and then you can use XPath to navigate through elements and attributes of the document getting the data of them in the process.

Using PHP and XPath

Now if we were to consider the following HTML:

<html>   
  <head>
    <title>Test</title>
  </head>
  <body>
    <div id="mainDiv">
      <div id="leftDiv">
        <p class="bodyText">This is left</p>
        <img src="images/test.jpg" />
      </div>
      <div id="rightDiv">
        <p class="bodyText">This is right</p>
      </div>
    </div>
    <div id="footer">
      <p class="smallText">This is the footer</p>
    </div>
  </body>
</html>
HTML

And now, lets say I wanted to get the text of the rightDiv. This is how we would go about it:

<?php
  // $content is the content you scraped (via curl for example)
  $dom = new DOMDocument();
  @$dom->loadHTML($content)
  $xpath = new DOMXPath($dom);
  $rightDivText = $xpath->query("//html/body/div[@id='mainDiv']/div[@id='rightDiv']/p/text()"); 
  
  // Returns: This is right
  echo $rightDivText->item(0)->nodeValue;
?>
PHP

Simply, it will load the $content (HTML) into a DOMDocument and then using XPath it looks for the for the div with the id of ‘mainDiv’ and then inside of ‘mainDiv’ it looks for the div with the id of ‘rightDiv’. Finally it then gets the text of the ‘p’ element.

Now that is one of the more straight forward scenarios and is made easier by the fact that most of the different elements have custom IDs like ‘mainDiv’ or ‘rightDiv’. If we were to look at the same HTML block again but this time without IDs, this is how it would look.

<html>
  <head>
    <title>Test</title>
  </head>
  <body>
    <div>
      <div>
        <p class="bodyText">This is left</p>
        <img src="images/test.jpg" />
      </div>

      <div>
        <p class="bodyText">This is right</p>
      </div>
    </div>

    <div id="footer">
      <p class="smallText">This is the footer</p>
    </div>
  </body>
</html>
HTML

This time lets get the src of the image in the former leftDiv.

<?php
// $content is the content you scraped (via curl for example)
$dom = new DOMDocument();  @$dom->loadHTML($content)
$xpath = new DOMXPath($dom);
$imgSrc = $xpath->query("//html/body/div[1]/div[1]/img/@src");

// Returns: images/test.jpg
echo $imgSrc->item(0)->nodeValue;
?>
PHP

As you can see, the difference this time is that instead of using the divs ID, we used the elements number. Inside the body, it looks for the first div, and then inside of that it again looks for the first child div before returning the src of the img element.

Again though, this is probably another pretty simple solution but definitely gives you an idea of how XPath works and some of the ways it can be used. The next scenario is basically the one I came face to face with, and that scenario is one in which we have multiple divs with the same name. Consider this html:

<html>
  <head>
    <title>Test</title>
  </head>
  <body>
    <div id="container">
      <div id="person">
        <p>Bob</p>
        <p>25</p>
        <p>180lbs</p>
        <img src="images/p1.jpg" />
      </div>
      <div id="person">
        <p>Stacy</p>
        <p>26</p>
        <p>187lbs</p>
        <img src="images/p2.jpg" />
      </div>
      <div id="person">
        <p>John</p>
        <p>21</p>
        <p>255lbs</p>
        <img src="images/p3.jpg" />
      </div>
    </div>
    <div id="footer"><p class="smallText">This is the footer</p></div>
  </body>
</html>
HTML

So here we have a webpage that displays a bunch of people with information such as their name, age, weight, and picture. Now as someone who is interested in getting the information of all of these people, I need to scrape the website and then using XPath I can loop through each person and get it.

To do this, we get everything inside the ‘container’ div and then loop through each individual ‘person’ div. Then using XPath on the ‘person’ div we can use the above examples to get the information we want. Finally we store all the results in an array.

 <?php
$dom = new DOMDocument();
@$dom->loadHTML($content);
$tempDom   = new DOMDocument();
$xpath     = new DOMXPath($dom);
$container = $xpath->query("//div[@id='container']");
foreach ($container as $item) {
    $tempDom->appendChild($tempDom->importNode($item, true));
}
$tempDom->saveHTML();
$peopleXpath = new DOMXPath($tempDom);
$peopleDiv   = $peopleXpath->query("div[@id='person']");
$results     = array();
foreach ($peopleDiv as $people) {
    $newDom = new DOMDocument;
    $newDom->appendChild($newDom->importNode($people, true));
    $personXpath = new DOMXPath($newDom);
    $name        = trim($personXpath->query("p[1]/text()")->item(0)->nodeValue);
    $age         = trim($resultXpath->query("p[2]/text()")->item(0)->nodeValue);
    $weight      = trim($resultXpath->query("p[3]/text()")->item(0)->nodeValue);
    $image       = trim($resultXpath->query("img[1]/@src")->item(0)->nodeValue);
    $results[]   = array(
        'name' => $name,
        'age' => $age,
        'weight' => $weight,
        'image' => $image
    );
}
?>
PHP

There you have it. If you run a print_r on the results array you should have yourself a bunch values containing the information of Bob, Stacy, and John.

Now a totally awesome cheat for this solution is to use the Google Chrome XPath Helper extension. All you need to do is hold down the shift key as you hover over webpage elements. This in turn will return the XPath query which is a HUGE help!

As always feel free to comment and or improve on the above. Also, don’t forget to follow me @JAGracie.

The post Reading a websites contents using PHP and XPath first appeared on Devhour.

]]> https://www.devhour.net/reading-a-websites-contents-using-php-and-xpath/feed/ 0 Raspberry Pi and Motion (Weather App Pt 2) https://www.devhour.net/raspberry-pi-and-motion-weather-app-pt-2/?utm_source=rss&utm_medium=rss&utm_campaign=raspberry-pi-and-motion-weather-app-pt-2 https://www.devhour.net/raspberry-pi-and-motion-weather-app-pt-2/#respond Mon, 06 May 2013 12:12:00 +0000 https://www.devhour.net/?p=17 Part 2 in this series is going to show you how to setup Raspberry Pi and Motion in order to snap a picture every 10 minutes or so. If you haven’t already read Part 1 in this series to check out how to set up a Raspberry Pi temperature sensor with the DS18B20 sensor. Having setup the […]

The post Raspberry Pi and Motion (Weather App Pt 2) first appeared on Devhour.

]]> Part 2 in this series is going to show you how to setup Raspberry Pi and Motion in order to snap a picture every 10 minutes or so.

If you haven’t already read Part 1 in this series to check out how to set up a Raspberry Pi temperature sensor with the DS18B20 sensor.

Having setup the temperature sensor and tested that it can read the temperature correctly, I can now move on to setting up our camera to capture an image of the weather every 10 minutes.

Fortunately I did have a spare web cam lying around but you could get away with spending a couple of dollars and getting a fairly cheap web camera that’ll do the trick.

Now to do this I am going to use a pretty cool piece of software called Motion. Motion is a piece of software which uses video4linux device to detect motion and can be used to take both videos and pictures. It also has a pretty powerful trigger action settings which allows you to run shell commands after/before a certain event in order to perform extra actions (eg Send an email or upload the video/picture). I will make use of these in the next part of this series in order to grab the current temperature after the image has been taken.

One of the cool things people have used Motion for is to make a basic home security system. Have a webcam pointed at the driveway or main entrance to your house and then have motion take a video or picture of anyone who approaches. Upload the video to a remote location and send you an email letting you know you have a visitor. Works quite well and for ~$50 is a cheap solution.

For now though I am going simpler. Tell Motion to take a picture of whatever the web camera is pointing at every 10 minutes and move the picture to our images folder (later used for the website I make). In my case I will be pointing my web cam out the window in order to see the weather.

Setup Raspberry Pi and Motion

Firstly, install and setup motion on the Raspberry Pi.

# Update and upgrade the Raspberry Pi
sudo apt-get update
sudo apt-get upgrade

# Install motion
sudo apt-get install motion
Bash

Then edit the config file.

# Tell the service to start in daemon mode
sudo nano /etc/default/motion
start_motion_daemon=yes

# Make these changes to the motion config file
sudo nano /etc/motion/motion.conf
daemon on
snapshot_interval 600
output_normal off
ffmpeg_cap_new off
target_dir /home/pi/motion/images
INI

The above changes made to the config are only a handful of the options you will see inside the file. The general idea of the changes I made are to turn off video recording and to tell motion to take a snapshot every 10 minutes and to then save the file to the images directory. Feel free to play around but these settings will get me doing what I need to do.

Start Motion on the Raspberry Pi

You probably want to start motion in debug mode the first time around just to make sure everything is as it should be.

sudo motion -n
Bash

This will display output to the screen and every 10 minutes you should see snapshots getting saved to the output directory.

Check the directory and download the images to make sure you are getting the desired output and if all is as expected you can now start the motion service.

sudo service motion start
Bash

That’s it! Raspberry Pi and Motion is now set up. Pretty easy and straight forward. Like I said though, this is one of the more basic uses of motion and it can be used to do a lot more. Have a play around and let me know what you come up with. I’m always interested!

The next part of this tutorial series I will be showing you how to write a python script which will be executed using the motion action trigger.

So stay tuned and as always you can follow me on twitter @JAGracie

The post Raspberry Pi and Motion (Weather App Pt 2) first appeared on Devhour.

]]> https://www.devhour.net/raspberry-pi-and-motion-weather-app-pt-2/feed/ 0 The Dev Hour begins https://www.devhour.net/the-dev-hour-begins/?utm_source=rss&utm_medium=rss&utm_campaign=the-dev-hour-begins https://www.devhour.net/the-dev-hour-begins/#respond Wed, 10 Apr 2013 08:11:00 +0000 https://www.devhour.net/?p=11 The first few years of my working life has seen me fall slightly behind in the world of web development so I’m taking the time to get a blog together to write down my findings and any interesting little tidbits (/titbits) I find along the way. Bare with me as I try to catch up […]

The post The Dev Hour begins first appeared on Devhour.

]]> The first few years of my working life has seen me fall slightly behind in the world of web development so I’m taking the time to get a blog together to write down my findings and any interesting little tidbits (/titbits) I find along the way.

Bare with me as I try to catch up (may find it impossible as this part of the world moves at a rate of knots). I’ll probably be talking about pieces of code that have been around for some time now. Feel free to help out with a comment or 2 to help get me up to speed, as I’ll appreciate any help I come across.

The post The Dev Hour begins first appeared on Devhour.

]]> https://www.devhour.net/the-dev-hour-begins/feed/ 0