PLEASE NOTE: “As an Amazon Associate, [Need Coffee] earns from qualifying purchases." You know we make money from Amazon links,
and I know you know this, but they make us say it anyway. More info, click here.

Guide to Writing Webbots, Spiders, and Screen Scrapers

Create automated agents with PHP/CURL to harness the Web

April 2, 2007, San Francisco–The Internet is bigger and better than what a mere browser allows. “Webbots, Spiders, and Screen Scrapers: A Guide to Developing Internet Agents with PHP/CURL” (No Starch Press, April 2007, is for programmers and businesspeople who want to take full advantage of the vast resources available on the Web. As author Michael Schrenk demonstrates, there’s no reason to let browsers limit the online experience–especially when it’s so easy to automate online tasks to suit individual needs.

This new book begins by outlining the deficiencies of browsers, then explains how these deficiencies can be exploited in the design and deployment of task-specific webbots–customized programs that aggregate different sources, filter content for relevant data, and automate online transactions.

Inside “Webbots, Spiders, and Screen Scrapers,” readers learn how to write fault-tolerant webbots and spiders that:

-download entire websites and parse data from web pages
-manage cookies and decode encrypted files
-automate form submissions and send and receive email
-send SMS alerts to cell phones
-unlock password-protected websites
-automatically bid in online auctions
-exchange data with FTP and NNTP servers

[ad#longpost]Sample projects reinforce these new skills so readers can create simple Web applications to track online prices, create anonymous browsing environments, archive online data, and more. In addition, the author’s website ( provides readers with sample scripts and code libraries, as well as a place to test their own webbots.

“It can be difficult to learn how to design, develop, and deploy webbots,” said No Starch Press founder Bill Pollock. “Mike Schrenk has been living and breathing this stuff for many years and is the perfect teacher to share his accumulated wisdom.”

As “Webbots, Spiders, and Screen Scrapers” illustrates, some tasks are just too tedious—or too important—to leave to humans. With guidance from author Schrenk on how to automate online life, readers won’t let a browser limit the way they use the Internet again.

Table of contents:
Sample chapter:

ABOUT THE AUTHOR: Michael Schrenk develops webbots and spiders for clients across North America. He has written for Computerworld and Web Techniques magazines and has taught college courses on Web usability and Internet marketing. He’s also an occasional speaker at DEFCON.

Webbots, Spiders, and Screen Scrapers: A Guide to Developing Internet Agents with PHP/CURL by Michael Schrenk Apr 07, 328pp, US$39.95, ISBN-10 1-59327-120-4; ISBN-13 978-1-59327-120-6 Available in bookstores everywhere, from, or from No Starch Press (,, 800.420.7240).

ABOUT NO STARCH PRESS: Founded in 1994, No Starch Press ( is one of the few remaining independent computer book publishers. We publish the finest in geek entertainment–unique books on technology, with a focus on open source, security, hacking, programming, and alternative operating systems. Our titles have personality, our authors are passionate, and our books tackle topics that people care about. No Starch Press titles have been selected for the prestigious Communication Arts Design Annual and STEP inside 100, and have won the Ippy Award from Independent Publisher magazine. Visit for more information and our complete catalog. (And most No Starch Press books use RepKover, a lay-flat binding that won’t snap shut.)