How to set search engine robots meta tag content value in Magento 2?

You can set search engine meta tag robots content value from the Magento admin panel.

Just Log in with the Admin panel,

      • Go To Content -> Design -> Configuration,
        Edit your Theme,
      • From the Other Settings Section,
      • Click on Search Engine Robots,
      • Choose the value from Default Robots,
      • Save Configuration.

You can see the default robots value from the <head> tag with meta name robots,
<meta name=”robots” content=”INDEX,FOLLOW”/>

Available robots value will be one of these,

      1. INDEX, FOLLOW
      2. NOINDEX, FOLLOW
      3. INDEX, NOFOLLOW
      4. NOINDEX, NOFOLLOW

This Meta tag robot set the content value (INDEX, FOLLOW) from the above configuration.
You can see it from the code level, vendor/magento/framework/View/Page/Config.php

Check the getRobots() method, which will set the content value based on the current theme of the store. You can change it based on your custom requirement for a specific page by creating an after-plugin.