You can set search engine meta tag robots content value from the Magento admin panel.
Just Log in with the Admin panel,
-
-
- Go To Content -> Design -> Configuration,
Edit your Theme, - From the Other Settings Section,
- Click on Search Engine Robots,
- Choose the value from Default Robots,
- Save Configuration.
- Go To Content -> Design -> Configuration,
-
You can see the default robots value from the <head> tag with meta name robots,
<meta name=”robots” content=”INDEX,FOLLOW”/>
Available robots value will be one of these,
-
-
-
INDEX, FOLLOW
-
NOINDEX, FOLLOW
-
INDEX, NOFOLLOW
-
NOINDEX, NOFOLLOW
-
-
This Meta tag robot set the content value (INDEX, FOLLOW) from the above configuration.
You can see it from the code level, vendor/magento/framework/View/Page/Config.php
Check the getRobots() method, which will set the content value based on the current theme of the store. You can change it based on your custom requirement for a specific page by creating an after-plugin.