How to use noindex, follow on duplicate pages for search engines

Question

I can easily set NO INDEX, FOLLOW on particular page through layout update.

But what I want is, I need to set NO INDEX, FOLLOW when the page loads after filtering from layered navigation on product listing page.

For Eg : https://test.com/recording-studio/keyboard-controllers.html?number_of_keys=37&price=-150

This is duplicate of https://test.com/recording-studio/keyboard-controllers.html. So need not to be indexed.

Any solutions for this ?

Answer 1

There are two options to accomplish this. First of all, you can use canonical URLs. This will tell the search engines that, although they are on a different URL, the canonical URL is the actual URL of the current page. This can be done within Magento by going to System > Configuration > Catalog > Catalog and then set the options Use Canonical Link Meta Tag For Categories and Use Canonical Link Meta Tag For Products under the tab Search Engine Optimisation to "Yes".

Second of all, you can define this in your robots.txt, which you can place in your project's root folder. This will not add a "noindex, nofollow" tag to your page, but it will tell search engines to ignore those pages.

An example of the robots.txt that I always use can be found here.

Of course you could also add a meta tag in your page, but that won't work for specific parameters (?price=xxx, etc.). For that you could use this page (also shared by Arunendra).

Answer 2

You can ignore dynamic urls using robots.txt just you need to add this line in this file you can find this file on your root folder:

Disallow: /*?*

For more see here.