Follows links across the whole web in resumable batches. Each Run now crawls a batch (about a minute) and saves progress — click again (or let cron run) to keep going. Polite: respects robots.txt and caps pages per site.
# Keep crawling automatically -- one batch every 5 minutes (crontab -e on the server):
*/5 * * * * php /var/www/oneselfapps.com/deepcrawler.php run
*/5 * * * * php /var/www/oneselfapps.com/deepcrawler.php run