Configure the LiteSpeed crawler to pre-warm LSCache

LSCache only helps the second visitor. The first visitor to any URL pays the full PHP-render bill — and on a content site with thousands of long-tail URLs, that first visitor is almost always Googlebot. Pre-warming the cache with the LiteSpeed crawler flips that: a background job walks your sitemap, hits each URL once, and leaves a warm cache entry behind. By the time a human (or a bot that matters) lands, the response is a single-digit-millisecond cache hit.

The crawler ships free with the LSCache for WordPress plugin, but on most shared and reseller hosts it's disabled at the server level. This guide covers the WHM-side allow flag, the per-site crawler config, the sitemap and role-simulation pieces, and the four things that quietly stop it from running.

Prerequisites

You need LiteSpeed Web Server Enterprise (or OpenLiteSpeed) on the box — the crawler won't work on Apache even with the LSCache plugin installed. If you're still on Apache, start with installing LSWS on a cPanel box.

Also required:

• An active LSWS licence — the crawler is bundled, no add-on
• LSCache enabled and verifying hits — see enabling LSCache on cPanel
• A working XML sitemap on each target site (Yoast, Rank Math, All in One SEO, or the WordPress core sitemap at /wp-sitemap.xml all work)
• WHM root for the server-side step

Step 1 — Allow the crawler at the server level

The LSCache plugin can queue crawl jobs all it likes, but if LSWS doesn't allow the crawler endpoint, none of them will execute. The default for new LSWS installs on cPanel is disabled — LiteSpeed ships it off so a careless tenant can't accidentally self-DoS a shared box.

In WHM → LiteSpeed Web Server → LSWS Configuration → Cache Settings → expand the Crawler section:

• Allow Crawler — Yes
• Max Crawler Threads — 3 (start conservative; raise to 5 on dedicated boxes)
• Crawler Throttle (microseconds) — 1000000 (1 second between URL hits per thread)
• Crawler Load Limit — 1.5 (suspend crawler when 1-minute load average exceeds this; tune to roughly the CPU count of the box)

Save and click Graceful Restart.

To verify the server now accepts crawler runs:

grep -i crawler /usr/local/lsws/conf/cache.conf

You should see enableCrawler 1 and the throttle values you set.

Step 2 — Configure the crawler per site

Server-side allow is necessary but not sufficient — each WordPress install configures its own crawl schedule, URL list, and role simulations.

In the WordPress admin → LiteSpeed Cache → Crawler → General Settings:

• Crawler — ON
• Delay (microseconds) — 500000 (0.5s; the server-side throttle still caps it)
• Run Duration (seconds) — 400 (the crawler runs in bursts; longer durations crawl more URLs per cron tick but consume more memory)
• Interval Between Runs (seconds) — 600 (10 minutes between bursts; for fast- changing sites drop to 300)
• Crawl Interval (seconds) — 302400 (3.5 days — how often each URL is re-warmed)
• Threads — 3 (must be ≤ the server-side Max Crawler Threads)
• Timeout (seconds) — 30
• Server Load Limit — 1 (per-site load ceiling; the server-side limit still applies)

Then under Sitemap Settings:

• Custom Sitemap — paste the absolute URL of the XML sitemap, e.g. https://example.com/wp-sitemap.xml or https://example.com/sitemap_index.xml for Yoast
• Drop Domain from Sitemap — ON (LSCache stores URLs without the host prefix internally)
• Sitemap Timeout — 120

Click Save Changes, then click Refresh Crawler List. The plugin fetches the sitemap, parses URLs, and shows a count — confirm it matches roughly what your sitemap contains. If it shows zero, the sitemap URL is wrong or returns a non-200.

Step 3 — Add role simulations for logged-in or member content

By default the crawler hits each URL as an anonymous visitor. If the site has members-only pages, a WooCommerce cart, or any per-role personalisation, you need the crawler to also visit as each role — otherwise the public cache warms but the logged-in cache stays cold.

In LiteSpeed Cache → Crawler → Simulation Settings:

• Role Simulation — add the user ID of one user per role you want crawled (typically: a subscriber, a member, a customer). The crawler will load each sitemap URL once per simulated role.
• Cookie Simulation — for non-WordPress role cookies (membership plugins, A/B testing tools), add cookie_name / cookie_value pairs.

Each added simulation multiplies the crawl workload. A site with 5,000 URLs and three role simulations is a 20,000-request job — size your Interval Between Runs and Run Duration accordingly.

Step 4 — Trigger the first crawl and watch it run

The crawler runs via WP-Cron by default. On low-traffic sites WP-Cron only fires on page loads, so the crawler can stall for hours waiting for someone to hit the site. Replace WP-Cron with a real system cron under the cPanel user:

*/5 * * * * cd /home/USER/public_html && /usr/local/bin/php -q wp-cron.php >/dev/null 2>&1

Then disable internal WP-Cron in wp-config.php:

define('DISABLE_WP_CRON', true);

Watch the crawler activity:

tail -f /home/USER/public_html/wp-content/litespeed/crawler/*.log

Or from the WordPress admin, LiteSpeed Cache → Crawler → Crawler Cron. Status flags you'll see:

• Waiting — crawler queued, will run on next cron tick
• Position — current index in the URL list (e.g. 1247 / 5832)
• Threads — active worker count
• Last Result — Success, or an error message

Verify cache is warming. Pick a URL the crawler just visited and request it twice:

curl -I https://example.com/some-post/
curl -I https://example.com/some-post/

Both responses should include x-litespeed-cache: hit — even the first, because the crawler primed it.

When the crawler isn't running

Four failures account for almost every "crawler enabled but nothing's happening" ticket:

• Server-side Allow Crawler is off. Check /usr/local/lsws/conf/cache.conf for enableCrawler 1. If you set it in WHM and the file still says 0, the LSWS restart didn't apply — run /usr/local/lsws/bin/lswsctrl restart manually.
• WP-Cron isn't firing. No system cron, no traffic, no wp-cron.php execution. Confirm with crontab -u USER -l and tail /var/log/cron.
• Sitemap returns garbage or 404. Some security plugins block bot access to sitemaps. Fetch the sitemap URL yourself: curl -A 'lsrunner' https://example.com/wp-sitemap.xml.
• Imunify360 or ModSecurity blocks the crawler User-Agent. The crawler hits URLs with User-Agent: lsrunner. If WAF rules see that as suspicious, requests get 403'd. Add lsrunner to the WAF allowlist — see tuning Imunify360 WAF false positives.

Common gotchas

• The crawler counts against LVE entry processes. Each thread = one EP for the user. On CloudLinux a site with EP=20 and a 5-thread crawler config will throttle live visitors during a crawl burst. Either cap crawler threads at half the EP limit or raise EP on heavy WordPress users.
• QUIC.cloud CDN can mask crawler effectiveness. If the site is fronted by QUIC.cloud, the crawler warms the origin but the user still hits the CDN edge — a cold CDN edge will still show a slow first byte. Configure CDN pre-warming separately in the QUIC.cloud dashboard.
• Long sitemaps with stale URLs waste budget. If half your sitemap is 404s or redirects, half your crawler runs warm nothing. Audit the sitemap quarterly.

Next steps

• If cache hit rate is still low after pre-warming, walk through fixing LSCache disabled on WordPress — the crawler can't help when something else is busting the cache key.
• For sites behind a CDN, pair the crawler with QUIC.cloud edge caching so the warming extends past your origin.
• On busy shared boxes, tune CloudLinux LVE limits before enabling crawler fleet-wide so a runaway crawl doesn't take a customer's site down.