User-agent: * Disallow: /wp-admin/ Disallow: /wp-content/uploads/catalog/ # Block the /pdfs/directory from being indexed and scraped from the links to the .pdfs Allow: /wp-admin/admin-ajax.php