Robots.txt is a text document that site proprietors can make to advise web crawler bots how to creep and list pages on their website. You can have various lines of directions to permit or forbid explicit URLs and add numerous sitemaps.
If you don't have a robots.txt file, then search engines will still crawl and index your website. This can slow down your website indexing rate. You can fix this by refusing search bots from endeavoring to creep pointless pages like your WordPress administrator pages, module documents, and topics organizer.
How do I use robots txt in WordPress?
1. Create or edit robots.txt in the WordPress Dashboard
2. Log in to your WordPress website
3. Click on 'SEO
4. Click on 'Tools
5. Click on 'File Editor
6. Make the changes to your file
7. Save your changes