PHP - Devhour https://www.devhour.net Taking time to write about development Tue, 12 Mar 2024 10:01:42 +0000 en-US hourly 1 https://wordpress.org/?v=6.5.2 https://www.devhour.net/wp-content/uploads/2024/03/cropped-devhourlogo-32x32.png PHP - Devhour https://www.devhour.net 32 32 Reading a websites contents using PHP and XPath https://www.devhour.net/reading-a-websites-contents-using-php-and-xpath/?utm_source=rss&utm_medium=rss&utm_campaign=reading-a-websites-contents-using-php-and-xpath https://www.devhour.net/reading-a-websites-contents-using-php-and-xpath/#respond Mon, 16 Sep 2013 12:32:00 +0000 https://www.devhour.net/?p=32 Following on from my previous post Scraping data with PHP and cURL today I’m going to show you about reading a websites contents using PHP and XPath. One of the biggest problems I have faced when dealing with a scraped web page (read: block of HTML) is how to go about getting specific text elements out of it. Several years […]

The post Reading a websites contents using PHP and XPath first appeared on Devhour.

]]> Following on from my previous post Scraping data with PHP and cURL today I’m going to show you about reading a websites contents using PHP and XPath.

One of the biggest problems I have faced when dealing with a scraped web page (read: block of HTML) is how to go about getting specific text elements out of it. Several years ago I would’ve gone with regex. This was a hassle to me and I’ve still yet to get my head around how it correctly works. These days though, I have discovered a useful little thing called XPath.

Basically, how it works is, you put the HTML in a DOMDocument and then you can use XPath to navigate through elements and attributes of the document getting the data of them in the process.

Using PHP and XPath

Now if we were to consider the following HTML:

<html>   
  <head>
    <title>Test</title>
  </head>
  <body>
    <div id="mainDiv">
      <div id="leftDiv">
        <p class="bodyText">This is left</p>
        <img src="images/test.jpg" />
      </div>
      <div id="rightDiv">
        <p class="bodyText">This is right</p>
      </div>
    </div>
    <div id="footer">
      <p class="smallText">This is the footer</p>
    </div>
  </body>
</html>
HTML

And now, lets say I wanted to get the text of the rightDiv. This is how we would go about it:

<?php
  // $content is the content you scraped (via curl for example)
  $dom = new DOMDocument();
  @$dom->loadHTML($content)
  $xpath = new DOMXPath($dom);
  $rightDivText = $xpath->query("//html/body/div[@id='mainDiv']/div[@id='rightDiv']/p/text()"); 
  
  // Returns: This is right
  echo $rightDivText->item(0)->nodeValue;
?>
PHP

Simply, it will load the $content (HTML) into a DOMDocument and then using XPath it looks for the for the div with the id of ‘mainDiv’ and then inside of ‘mainDiv’ it looks for the div with the id of ‘rightDiv’. Finally it then gets the text of the ‘p’ element.

Now that is one of the more straight forward scenarios and is made easier by the fact that most of the different elements have custom IDs like ‘mainDiv’ or ‘rightDiv’. If we were to look at the same HTML block again but this time without IDs, this is how it would look.

<html>
  <head>
    <title>Test</title>
  </head>
  <body>
    <div>
      <div>
        <p class="bodyText">This is left</p>
        <img src="images/test.jpg" />
      </div>

      <div>
        <p class="bodyText">This is right</p>
      </div>
    </div>

    <div id="footer">
      <p class="smallText">This is the footer</p>
    </div>
  </body>
</html>
HTML

This time lets get the src of the image in the former leftDiv.

<?php
// $content is the content you scraped (via curl for example)
$dom = new DOMDocument();  @$dom->loadHTML($content)
$xpath = new DOMXPath($dom);
$imgSrc = $xpath->query("//html/body/div[1]/div[1]/img/@src");

// Returns: images/test.jpg
echo $imgSrc->item(0)->nodeValue;
?>
PHP

As you can see, the difference this time is that instead of using the divs ID, we used the elements number. Inside the body, it looks for the first div, and then inside of that it again looks for the first child div before returning the src of the img element.

Again though, this is probably another pretty simple solution but definitely gives you an idea of how XPath works and some of the ways it can be used. The next scenario is basically the one I came face to face with, and that scenario is one in which we have multiple divs with the same name. Consider this html:

<html>
  <head>
    <title>Test</title>
  </head>
  <body>
    <div id="container">
      <div id="person">
        <p>Bob</p>
        <p>25</p>
        <p>180lbs</p>
        <img src="images/p1.jpg" />
      </div>
      <div id="person">
        <p>Stacy</p>
        <p>26</p>
        <p>187lbs</p>
        <img src="images/p2.jpg" />
      </div>
      <div id="person">
        <p>John</p>
        <p>21</p>
        <p>255lbs</p>
        <img src="images/p3.jpg" />
      </div>
    </div>
    <div id="footer"><p class="smallText">This is the footer</p></div>
  </body>
</html>
HTML

So here we have a webpage that displays a bunch of people with information such as their name, age, weight, and picture. Now as someone who is interested in getting the information of all of these people, I need to scrape the website and then using XPath I can loop through each person and get it.

To do this, we get everything inside the ‘container’ div and then loop through each individual ‘person’ div. Then using XPath on the ‘person’ div we can use the above examples to get the information we want. Finally we store all the results in an array.

 <?php
$dom = new DOMDocument();
@$dom->loadHTML($content);
$tempDom   = new DOMDocument();
$xpath     = new DOMXPath($dom);
$container = $xpath->query("//div[@id='container']");
foreach ($container as $item) {
    $tempDom->appendChild($tempDom->importNode($item, true));
}
$tempDom->saveHTML();
$peopleXpath = new DOMXPath($tempDom);
$peopleDiv   = $peopleXpath->query("div[@id='person']");
$results     = array();
foreach ($peopleDiv as $people) {
    $newDom = new DOMDocument;
    $newDom->appendChild($newDom->importNode($people, true));
    $personXpath = new DOMXPath($newDom);
    $name        = trim($personXpath->query("p[1]/text()")->item(0)->nodeValue);
    $age         = trim($resultXpath->query("p[2]/text()")->item(0)->nodeValue);
    $weight      = trim($resultXpath->query("p[3]/text()")->item(0)->nodeValue);
    $image       = trim($resultXpath->query("img[1]/@src")->item(0)->nodeValue);
    $results[]   = array(
        'name' => $name,
        'age' => $age,
        'weight' => $weight,
        'image' => $image
    );
}
?>
PHP

There you have it. If you run a print_r on the results array you should have yourself a bunch values containing the information of Bob, Stacy, and John.

Now a totally awesome cheat for this solution is to use the Google Chrome XPath Helper extension. All you need to do is hold down the shift key as you hover over webpage elements. This in turn will return the XPath query which is a HUGE help!

As always feel free to comment and or improve on the above. Also, don’t forget to follow me @JAGracie.

The post Reading a websites contents using PHP and XPath first appeared on Devhour.

]]> https://www.devhour.net/reading-a-websites-contents-using-php-and-xpath/feed/ 0 Automatically post to a Page using PHP Facebook SDK https://www.devhour.net/automatically-post-to-a-page-using-php-facebook-sdk/?utm_source=rss&utm_medium=rss&utm_campaign=automatically-post-to-a-page-using-php-facebook-sdk https://www.devhour.net/automatically-post-to-a-page-using-php-facebook-sdk/#respond Tue, 10 Sep 2013 12:30:00 +0000 https://www.devhour.net/?p=30 During my weekends/time off I tend to watch quite a lot of sport, with the English Premier League (EPL) being one of my favourite things to support. As such, I created and run an EPL facebook page dedicated  to covering fixtures, results, and other news. Now this all good fun and I enjoy doing it, but week after […]

The post Automatically post to a Page using PHP Facebook SDK first appeared on Devhour.

]]> During my weekends/time off I tend to watch quite a lot of sport, with the English Premier League (EPL) being one of my favourite things to support. As such, I created and run an EPL facebook page dedicated  to covering fixtures, results, and other news.

Now this all good fun and I enjoy doing it, but week after week, month after month of posting all the fixtures the day before and posting the results the day after gets a little tedious, especially if I’m short on time. So being the geek I am, I decided that I could easily automate these, and so I did, using the PHP facebook SDK.

It was actually pretty straight forward, the hardest bit was trying to find websites which would give me the information in a straight forward and easy way. In the end, I decided to go with the offical Premier League website, as not only is it the correct information it is also laid out in a way which makes it easy to retrieve.

So putting my previous posts into good use I used curl to scrape the website and Xpath to dissect it into the important parts (namely the home and away teams, the location, and the final score).

Once I have this important information I can then go away and use the PHP facebook SDK to post a status to my EPL News feed which I will show you how to do now.

Firstly, you will need to create a facebook application which will be integrated in a website. After you’ve created the app you will want to fill out the website URL and hit save. This URL is going to be where the PHP app you are going to create will be stored (ie http://devhour.net/fbapp).

EPLNewsFeed Facebook Application
EPLNewsFeed Facebook Application

Next up you’ll need to run the following URL to authorize your app to manage your pages and other such permissions. Remember to replace [APP_ID] and [SITE_URL] with both your facebook APP ID and the website you entered above. You will be prompted to grant access to your app (click Allow).

https://graph.facebook.com/oauth/authorize?client_id=[APP_ID]&scope=offline_access,publish_stream,manage_pages&redirect_uri=[SITE_URL]

It will then redirect you back to the URL of the site you specified in the app. If you look closely though you’ll notice that on the end of the url is ?code=aBuNcHoFgIbBeRiSh. Copy everything after the ?code= and keep it in a safe place for now. You’re going to need it in a second.

The next step is to get the long lasting access token which will allow you to keep posting without having to authenticate yourself every time (this is vital to making this whole script automatic). So again go to the following URL and replace [APP_ID], [APP_SECRET], [SITE_URL], and [CODE] (the one you just previously copied).

https://graph.facebook.com/oauth/access_token?client_id=[APP_ID]&redirect_uri=[SITE_URL]&client_secret=[APP_SECRET]&code=[CODE]

If all went well you’ll be redirected back to your website and on the screen you should see:

access_token=…….

Store this somewhere because it’s very important 🙂

Now the fun can start. Firstly, you can test the access token by going to the Facebook Graph API explorer, pasting the token into the Access token field and trying the following:

Facebook Graph API Explorer
Facebook Graph API Explorer

This will post a test message to your page which will indicate that the access token you have is good to go. Now on to the code…

Before we write any code on our end, you’re going to have to download the PHP facebook SDK from the github repo. Download and store these files somewhere handy and feel free to jump into the examples are have a bit of a play.

On our end however, the first thing we are going to is create a php file and reference the facebook sdk and enter some variable details (fill out as necessary).

 <?php
require "src/facebook.php";
$PAGE_ID         = ""; // The page you want to post to (you must be a manager) 
$FACEBOOK_APP_ID = ""; // Your facebook app ID
$FACEBOOK_SECRET = ""; // Your facebook secret
$ACCESS_TOKEN    = ""; // The access token you receieved above
$PAGE_TOKEN      = ""; // Leave this blank. It will be set later
PHP

Now we go ahead and create a new facebook instance using our newly created app details.

$facebook = new Facebook(array('appId' => $FACEBOOK_APP_ID, 'secret' => $FACEBOOK_SECRET, 'cookie' => true,));  
$post = array('access_token' => $ACCESS_TOKEN);
PHP

The next little piece of code will loop through the pages that you manage and find the PAGE_TOKEN which matches the PAGE_ID you specified. This will allow you to post to that pages wall as the page rather than you.

try {  
  $res = $facebook->api('/me/accounts','GET',$post);   
  if (isset($res['data'])) {
    foreach ($res['data'] as $account) {
      if ($PAGE_ID == $account['id']) { 
        $PAGE_TOKEN = $account['access_token']; 
        break; 
      }
    }
  }
} 
catch (Exception $e){  
 echo $e->getMessage();
}
PHP

Taking all the information we have so far received we can now go ahead and post to the pages wall.

$message = "This is a test message";
$post = array('access_token' => $PAGE_TOKEN, 'message' => $message);
try{
  $res = $facebook->api("/$PAGE_ID/feed","POST",$post);
}
catch (Exception $e){
  echo $e->getMessage();
}
?>
PHP

And that’s it. It’s actually really straight forward and easy to do. Feel free to play around with it and I’d be interested in seeing what you can come up with.

As always, follow me on twitter: @JAGracie

The post Automatically post to a Page using PHP Facebook SDK first appeared on Devhour.

]]> https://www.devhour.net/automatically-post-to-a-page-using-php-facebook-sdk/feed/ 0 Sending messages to an Eggdrop via PHP https://www.devhour.net/sending-messages-to-an-eggdrop-via-php/?utm_source=rss&utm_medium=rss&utm_campaign=sending-messages-to-an-eggdrop-via-php https://www.devhour.net/sending-messages-to-an-eggdrop-via-php/#respond Thu, 23 May 2013 12:19:00 +0000 https://www.devhour.net/?p=22 So if you frequent IRC often like I do, you may notice some eggdrop bots displaying messages which come from a website. The most common times you will see this sort of script in action is when they are hooked into forum posts or torrent listings. You will more than likely have seen messages saying […]

The post Sending messages to an Eggdrop via PHP first appeared on Devhour.

]]> So if you frequent IRC often like I do, you may notice some eggdrop bots displaying messages which come from a website. The most common times you will see this sort of script in action is when they are hooked into forum posts or torrent listings. You will more than likely have seen messages saying “New forum topic…X Y by Z” or “[PRE] such.and.such.movie.torrent”. Today I’m going to show you how to do that by sending messages to an eggdrop via PHP.

It’s actually relatively easy (as you are about to see) and really only requires a few lines of PHP code on the web end and the same again in TCL on the eggdrop. The general idea is that the eggdrop will open up a socket and sit listening waiting for data to come through before displaying the message. For the PHP end it is as simple as connecting to said socket and sending through some data.

First things first, the TCL script sock-listen.tcl:

set output_chan "#chan"
# eggdrop port to bindset 
port 7878
# eggdrop ip to bind
set host xxx.xxx.xxx.xxx
set serverSocket [socket -server main -myaddr $host $port]
proc main { sock host port } {
  fconfigure $sock -buffering line
  fileevent $sock readable [action $sock $host $port]
}
proc action { chan host port } {
  global output_chan
  if {![eof $chan]} {
    set soc_data [gets $chan]
    if {$soc_data != ""} {
      putquick "PRIVMSG $output_chan :$host | $port $soc_data"
    }
  } { close $chan }
}
putlog "sock-listen.tcl loaded"
TCL

That’s it! Simple enough. Create a socket which will sit and listen waiting for something to connect and send through data. Once a connection is made call the ‘action’ function which will  send a message to the channel with the host, port and socket data.

To get that up and running just copy sock-listen.tcl into the scripts folder of your eggdrop and include it in your config file. Don’t forget to change the channel, IP address and port. Once you rehash or reconnect the bot the socket will automatically start listening.

Now to send messages from PHP is very straight forward, just open up a socket and connect to the IP/port specified in the TCL script. Send through a message and watch it come out the other end (IRC channel).

To do this we create a php file called send.php and place the following code:

<?php
//eggdrop port
$egg_port = 7878;
//eggdrop ip
$egg_ip = 'xxx.xxx.xxx.xxx';
// message to send
$msg = 'Sent from a PHP page';
$sock = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
socket_connect($sock, $egg_ip, $egg_port) or die ('failed to connect!');
socket_write($sock, $msg);
socket_close($sock);
?>
PHP

Again, easy enough. Make sure you set the correct IP address and port and then upload the file. Point your browser to it and watch the magic happen. You should see the following message appear on the IRC channel:

xxx.xxx.xxx.xxx | 7878 | Sent from a PHP page

To make it easier I’ve added a small form which will allow you to customize the message to your liking. Again send.php:

<?php
//eggdrop port$egg_port = 7878;
//eggdrop ip
$egg_ip = 'xxx.xxx.xxx.xxx';

if (isset($_POST['message'])) {
  $msg = $_POST['message'];   
  $sock = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
  socket_connect($sock, $egg_ip, $egg_port) or die ('failed to connect!');
  socket_write($sock, $msg);
  socket_close($sock);
  header("Location: send.php");
}
?>
<form action='send.php' method='POST'>
<input type='text' name='message' cols='20'><br>
<input type='submit' value='send'>
</form>
PHP

It isn’t all that secure but gives you a general idea of what you can do. To implement it into the examples I posted at the start all you really need to do is edit the submit/post PHP files and place the socket code above with a customized message in.

This isn’t limited to just PHP. Something similar could easily be set up in a number or any other languages as well. Simply by opening the socket and sending the data.

Something simple for  the day. Have a play around and let me know if you make any improvements. Feel free to ask any questions, I’ll do my best to answer them :)

Also feel free to follow me on twitter: @JAGracie

The post Sending messages to an Eggdrop via PHP first appeared on Devhour.

]]> https://www.devhour.net/sending-messages-to-an-eggdrop-via-php/feed/ 0 Scraping data with PHP and cURL https://www.devhour.net/scraping-data-with-php-and-curl/?utm_source=rss&utm_medium=rss&utm_campaign=scraping-data-with-php-and-curl https://www.devhour.net/scraping-data-with-php-and-curl/#respond Sun, 12 May 2013 12:17:00 +0000 https://www.devhour.net/?p=20 I’m working on another idea which I hope to release soon which involves scraping websites using PHP and cURL. I don’t want to give too much away before I release the website so I won’t go into too much detail. However, what I can tell you is that it required me to go out and […]

The post Scraping data with PHP and cURL first appeared on Devhour.

]]> I’m working on another idea which I hope to release soon which involves scraping websites using PHP and cURL.

I don’t want to give too much away before I release the website so I won’t go into too much detail. However, what I can tell you is that it required me to go out and  get a lot of data from external websites using variables passed through from a form on my end.

I originally started out using a piece of python software called Scrapy which worked very well, but the logistics of using that and either storing the data or displaying it on a webpage became too much of a hassle so I instead opted to go for PHP and cURL.

For the PHP side I’m using the framework Codeigniter which is a very easy and very speedy framework which is perfect for what I wanted to do.

The basic flow of how everything works is:

  1. The form is filled out and submitted
  2. Data from the form is sent to the external website in a cURL request
  3. The webpage content is then returned
  4. From there the data can be formatted and displayed accordingly

To do this with PHP and cURL is a fairly straight forward process and I’ll show you how to go about it. The only real issue you may come across is that when forms come into play you need to make sure each and every form element is included in the call.

$url = 'http://www.website.com/login.php';
$postdata = array('username' => "Jamie",'password' => "password");
$ch = curl_init();
if($ch){
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
  curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt'); // set cookie file to given file   
  curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt'); // set same file as cookie jar   
  $content = curl_exec($ch);
  $headers = curl_getinfo($ch);
  curl_close($ch);
  // Debug option
  // print_r($headers);
  if($headers['http_code'] == 200){
    echo $content;
  }
}
PHP

That’s the entire call and will return the html contents of website.com/login.php. I’ll go through the above code piece by piece and give a run down on each of the different parts.

$url = 'http://www.website.com/login.php';
$postdata = array('username' => "Jamie", 'password' => "password");
PHP

Firstly, the url variable should be self explanatory and the postdata is just a simple array which contains the form elements that are required to login with (in this case a username and password).

$ch = curl_init();if($ch){
PHP

Create a new curl object and if all is well continue on.

curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt'); // set cookie file to given file
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt'); // set same file as cookie jar
PHP

These are all curl options which I am going to use. You can view the rest of the different options over at the php website. The ones we are using and basically all to do with logging in. Storing the cookies and passing through the post data are the main ones to take note of.

$content = curl_exec($ch);
$headers = curl_getinfo($ch);
curl_close($ch);
PHP

Last but not least, execute the curl request passing through the options we used, set the returned content to a variable and also grab the headers before finally closing the curl object.

As you will note in the original code I use the headers variable as a debug option. This is very handy, in particular the header_code which can be very useful. If you ever find that something isn’t working, double check that you are getting a 200 code and not a 400/501.

From there you can grab/scrape the content and data to your hearts content. A great thing is now that you have received and stored the cookies from logging in, you have access to ‘authenticated only’ sections of the website. So you can go away and run more curl requests to get those areas of the website.

I was about to end it there but one other important piece I have come across is that some forms that you fill out will actually re-direct you to different parts of the website after submit. It’s fairly easy to identify because you will get a header code of 302 and the great thing is that you also get a redirect_url in the headers. All you need to do is make another curl request using the redirect url you received.

// original curl request up here
if ($headers['http_code'] == 302){
  $ch = @curl_init();
  curl_setopt($ch, CURLOPT_URL, $headers['redirect_url']);
  curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt'); // set cookie file to given file   
  curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt'); // set same file as cookie jar   
  $content = curl_exec($ch);
}
PHP

Once you have gotten the content you require you then need to get access to the specific data or text you’re after. I’m going to show you how to do this using xpath in another post, so keep your eyes out for that.

Like always, if you have any comments or questions feel free to post and I’ll do my best to answer ’em.

Follow me on twitter @JAGracie

The post Scraping data with PHP and cURL first appeared on Devhour.

]]> https://www.devhour.net/scraping-data-with-php-and-curl/feed/ 0