Add draft robots.txt

[herzog31]

• Add draft.robots.txt as a blueprint for customers.

Test URLs:

• Before: https://main--aem-boilerplate-commerce--hlxsites.hlx.live/
• After: https://robots2--aem-boilerplate-commerce--hlxsites.hlx.live/

[aem-code-sync]

Hello, I'm the AEM Code Sync Bot and I will run some actions to deploy your branch and validate page speed.
In case there are problems, just click a checkbox below to rerun the respective action.

• Re-run PSI checks
• Re-sync branch

Commits

• e70fe75 🔄 (latest)
• 17ecd05 ✅
• 32acbcf ✅

[aem-code-sync]

Page	Scores	Audits	Google
/

[herzog31]

@hannessolo @dicagno Can you think of anything else that we should add to this file?

[hannessolo]

Does this take into account luma bridge pages? Should it also disallow pages like /account ?

[herzog31]

@hannessolo Good point. All of those pages should already be excluded via metadata <meta name="robots" content="noindex, nofollow"> e.g. https://main--aem-boilerplate-commerce--hlxsites.hlx.live/customer/account. We could move those over to the robots.txt.

[justinconabree]

Hi @herzog31 here's what we've seen are good paths to exclude in our projects:

/drafts/**
/enrichment/**
/experiments/**
/fragments/**
/tools/**
/nav
/footer

Note this was taken from the exclude from helix-query feeding our sitemap so the pattern might be different.

[herzog31]

Thanks @justinconabree, I added some of your suggestions 👍