Много тем пересмотрел, сравнивал разные файлы Роботс…
В некоторых сайтов вообще три записи, например:
User-agent: *
Disallow: /app/
Disallow: /design/themes/responsive/media/images
Allow: /design/themes/responsive/media/images/*
Disallow: /js/addons/image_zoom
Allow: /js/addons/image_zoom/*
Disallow: /store_closed.html
Disallow: /*var/cache/misc/1/ab__so_seohide/
Disallow: /*index.php?dispatch=
Disallow: /*subcats=
Disallow: /*currency=
Disallow: /*sort_by=
Disallow: /*layout=
Disallow: /*items_per_page=
Disallow: /*productfeatures-addproduct
Disallow: /*features_hash
Disallow: /*=discussion
Sitemap: https://site.com/sitemap.xml
В некоторых очень много разных…
Собрал вот такой файл Роботс, подскажите, все ли здесь хорошо? может есть что-то лишнее или чего-то не хватает?
User-agent: *
Disallow: /cgi-bin/
Disallow: /design/
Disallow: /var/
Disallow: /app/
Disallow: /images/thumbnails/
Disallow: /store_closed.html
Disallow: /404.html
Disallow: /init.php
Disallow: /php.ini
Disallow: /prepare.php
Disallow: /sendmail/
Disallow: /*ui=
Disallow: /*component
Disallow: /*content
Disallow: /*sort_by=
Disallow: /*layout=
Disallow: /*variant_id=
Disallow: /*display=
Disallow: /*return_url=
Disallow: /*more_filters=
Disallow: /*features_hash=
Disallow: /*subcats=
Disallow: /*?sef_rewrite=
Disallow: /*tags.summary*
Disallow: /*wishlist.view*
Disallow: /*?selected_section=
Disallow: /*?subcats=
Disallow: /*?filter_product=
Disallow: /*askquestion
Disallow: /login?
Disallow: /profiles-update
Disallow: /index.php?dispatch=products.quick_view
Disallow: /index.php?dispatch=auth.recover_password
Disallow: /index.php?dispatch=events.search
Disallow: /index.php?dispatch=auth.login_form
Disallow: /index.php?dispatch=gift_certificates.add
Disallow: /index.php?dispatch=news.list
Disallow: /index.php?dispatch=orders.search
Disallow: /index.php?dispatch=profiles.update
Disallow: /index.php?dispatch=profiles.add
Disallow: /index.php?dispatch=checkout.cart
Disallow: /index.php*
Disallow: /*?features_hash*
Disallow: /cart
Disallow: /profiles-add
Disallow: /wishlist
Disallow: /compare
Disallow: /orders
Allow: /*.css
Allow: /*.js
Host: https://site.com
Sitemap: https://site.com/sitemap.xml