In order to use the standalone Magento shell script set the “Enabled” configuration option to “No”. A new entry will need to be added into the crontab. Below is an example cron job, [magento root] should be replaced with your actual Magento root. Please note this script must be executed as a shell script and NOT a web request.
There are two common ways to setup the cron job. First is to run the crawler every five minutes, if there is another instance running, the most recent version will quit allowing the existing instance to finish. This ensures that your site is constantly being warmed. However if an instance dies and is unable to remove the lock file, the warmer will not warm the site until the lock file is removed.
*/5 * * * * php [magento root]/shell/cache-warmer.php
The second way is to space out each crawl instance with enough time to complete. The exact time is dependent on the set concurrency and the number of pages being crawled. The example below runs every two hours at 5 minutes past the hour. If there is an existing instance running, it will quit allowing the new instance to take over. With this method, you will never need to worry about removing a lock file, however you will want to give enough time for the instance to complete before the new instance started.
5 */2 * * * php [magento root]/shell/cache-warmer.php --force