Como veremos na t\u00f3pico sobre o Googlebot<\/a>, os rob\u00f4s dos buscadores s\u00e3o aplicativos que navegam pela internet atrav\u00e9s dos links encontrados nas p\u00e1ginas, em busca de conte\u00fado a ser indexado e exibido nos resultados de busca. Por\u00e9m, voc\u00ea pode optar por n\u00e3o ter algumas de suas p\u00e1ginas exibidas nos resultados de busca, como por exemplo:<\/p>\n Como o pr\u00f3prio nome j\u00e1 diz, robots.txt<\/strong> \u00e9 um arquivo no formato .txt (bloco de notas). Funciona como um filtro para os rob\u00f4s dos sites de busca e faz com que os webmasters controlem permiss\u00f5es de acesso a determinadas p\u00e1ginas ou pastas dos sites. O robots.txt controla qual informa\u00e7\u00e3o de um site deve ou n\u00e3o deve ser indexada pelos sites de busca. A sintaxe do arquivo \u00e9 bem simples, e deve ser colocada pelo webmaster respons\u00e1vel pelo site na raiz da hospedagem. O pr\u00f3prio Google usa um arquivo em http:\/\/www.google.com\/robots.txt<\/a>, e navegar por ele \u00e9 no m\u00ednimo curioso.<\/p>\n O arquivo robots.txt tem o papel de criar uma pol\u00edtica de acesso aos Robots. Para a execu\u00e7\u00e3o dessas tarefas, h\u00e1 palavras reservadas, ou seja, palavras com a fun\u00e7\u00e3o de comandos que permitir\u00e3o ou n\u00e3o o acesso a determinados diret\u00f3rios ou p\u00e1ginas de um site. Vejamos os principais comandos do arquivo robots.txt:<\/p>\n <\/p>\n A fun\u00e7\u00e3o do comando user-agent<\/strong> \u00e9 listar quais rob\u00f4s devem seguir as regras indicadas no arquivo robots.txt. Supondo que voc\u00ea deseje somente que o mecanismo de busca do Google siga as defini\u00e7\u00f5es definidas no arquivo robots.txt,basta indicar o User-agent como Googlebot. Eis as principais op\u00e7\u00f5es:<\/p>\n <\/p>\n O comando instrui os sites de busca sobre quais diret\u00f3rios ou p\u00e1ginas n\u00e3o devem ser inclu\u00eddas no \u00edndice. Exemplos:<\/p>\n <\/p>\n O comando Allow<\/strong> orienta aos robots<\/em> qual diret\u00f3rio ou p\u00e1gina deve ter o conte\u00fado indexado. Diret\u00f3rios e p\u00e1ginas s\u00e3o por defini\u00e7\u00e3o sempre permitidos. Assim, este comando deve ser utilizado apenas em situa\u00e7\u00f5es em que o webmaster bloqueou o acesso a um diret\u00f3rio por meio do comando Disallow, mas gostaria de ter indexado um arquivo ou sub-diret\u00f3rio dentro do diret\u00f3rio bloqueado. Note por exemplo no robots.txt do Google, logo no in\u00edcio, as duas linhas abaixo. O Allow permite que seja indexado o diret\u00f3rio \/about abaixo do diret\u00f3rio \/catalogs.<\/p>\n <\/a><\/p>\n Uma outra fun\u00e7\u00e3o permitia pelo robots.txt \u00e9 a indica\u00e7\u00e3o do caminho e nome do sitemap em formato XML<\/a> do site. A ferramenta para Webmasters do Google, por\u00e9m, oferece um maior controle e visibilidade para a mesma fun\u00e7\u00e3o – comunicar ao Google onde est\u00e1 o ou os arquivos sitemap<\/strong>. Note como o Google submete, em seu robots.txt, diversos sitemaps:<\/p>\n Como veremos abaixo em exemplos reais de robots.txt, \u00e9 muito f\u00e1cil acessar o conte\u00fado de arquivos robots.txt de qualquer site, inclusive de concorrentes. Assim, cuidado com o que \u00e9 inclu\u00eddo nesse arquivo. Evite colocar arquivos confidenciais. Nesses casos, o ideal \u00e9 utilizar a meta tag robots (meta name = “robots”), explicada no t\u00f3pico de meta tags<\/a> deste tutorial.<\/p>\n Exemplo<\/strong>: o webmaster n\u00e3o deseja que o conte\u00fado do diret\u00f3rio\/docs seja indexado pelos robots, ent\u00e3o, bloqueou o acesso ao diret\u00f3rio \/docs com o comando “Disallow: \/docs”<\/em> no arquivo robots.txt. Dentro desse diret\u00f3rio, por\u00e9m, existe um sub-diret\u00f3rio chamado “public”, que deve ter seu conte\u00fado indexado. Para que isso aconte\u00e7a, basta usar no arquivo robots.txt a instru\u00e7\u00e3o “Allow: \/docs\/public\/”<\/em>.<\/p>\n Para olhar exemplos de arquivos robots.txt<\/strong>, saia navegando pela internet e inclua o arquivo \/robots.txt na ra\u00edz dos sites visitados para verificar se eles utilizam o arquivo robots.txt. Veja abaixo alguns exemplos:<\/p>\n <\/p>\n <\/p>\n Sobre o autor<\/strong>: Fundador da SEO Marketing, Bernhard Schultze<\/a> \u00e9 l\u00edder do comit\u00ea de Search da APADI (Associa\u00e7\u00e3o Paulista de Ag\u00eancias Digitais) e instrutor do curso de SEO da Digitalks. Perfil Google Plus: +Bernhard Schultze<\/a><\/p>\n <\/p>\n <\/p>\n http:\/\/www.seomarketing.com.br\/robots.txt.php<\/p>\n http:\/\/www.google.com\/robots.txt<\/p>\n <\/p>\n Saiba como evitar que determinadas p\u00e1ginas do site apare\u00e7am no Google Como veremos na t\u00f3pico sobre o Googlebot, os rob\u00f4s dos buscadores s\u00e3o aplicativos que navegam pela internet atrav\u00e9s dos links encontrados nas p\u00e1ginas, em busca de conte\u00fado a ser indexado e exibido nos resultados…<\/p>\n","protected":false},"author":2,"featured_media":3617,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_angie_page":false,"page_builder":"","footnotes":""},"categories":[36],"tags":[],"class_list":["post-3615","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-seo"],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/posts\/3615","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/comments?post=3615"}],"version-history":[{"count":0,"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/posts\/3615\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/media\/3617"}],"wp:attachment":[{"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/media?parent=3615"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/categories?post=3615"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/tags?post=3615"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
O que \u00e9 robots.txt<\/h2>\n
Sintaxe do Robots.txt<\/h2>\n
User-agent<\/h3>\n
\n
Disallow<\/h3>\n
\n
Allow<\/h3>\n
Disallow: \/catalogs \nAllow: \/catalogs\/about\n<\/pre>\n
Sitemap<\/h3>\n
Sitemap: http:\/\/www.google.com\/hostednews\/sitemap_index.xml \nSitemap: http:\/\/www.google.com\/sitemaps_webmasters.xml \nSitemap: http:\/\/www.google.com\/ventures\/sitemap_ventures.xml \nSitemap: http:\/\/www.gstatic.com\/dictionary\/static\/sitemaps\/sitemap_index.xml \nSitemap: http:\/\/www.gstatic.com\/earth\/gallery\/sitemaps\/sitemap.xml \nSitemap: http:\/\/www.gstatic.com\/s2\/sitemaps\/profiles-sitemap.xml \nSitemap: http:\/\/www.gstatic.com\/trends\/websites\/sitemaps\/sitemapindex.xml<\/pre>\n
Cuidados com o arquivo robots.txt<\/h2>\n
Aplica\u00e7\u00e3o de Robots.txt<\/h2>\n
Exemplos reais de Robots.txt<\/h2>\n
\n
User-agent: *\nDisallow: \/search\nDisallow: \/sdch\nDisallow: \/groups\nDisallow: \/images\nDisallow: \/catalogs\nAllow: \/catalogs\/about\nAllow: \/catalogs\/p?\nDisallow: \/catalogues\nAllow: \/newsalerts\nDisallow: \/news\nAllow: \/news\/directory\nDisallow: \/nwshp\nDisallow: \/setnewsprefs?\nDisallow: \/index.html?\nDisallow: \/?\nAllow: \/?hl=\nDisallow: \/?hl=*&\nAllow: \/?hl=*&gws_rd=ssl$\nDisallow: \/?hl=*&*&gws_rd=ssl\nAllow: \/?gws_rd=ssl$\nAllow: \/?pt1=true$\nDisallow: \/addurl\/image?\nAllow: \/mail\/help\/\nDisallow: \/mail\/\nDisallow: \/pagead\/\nDisallow: \/relpage\/\nDisallow: \/relcontent\nDisallow: \/imgres\nDisallow: \/imglanding\nDisallow: \/sbd\nDisallow: \/keyword\/\nDisallow: \/u\/\nDisallow: \/univ\/\nDisallow: \/cobrand\nDisallow: \/custom\nDisallow: \/advanced_group_search\nDisallow: \/googlesite\nDisallow: \/preferences\nDisallow: \/setprefs\nDisallow: \/swr\nDisallow: \/url\nDisallow: \/default\nDisallow: \/m?\nDisallow: \/m\/\nAllow: \/m\/finance\nDisallow: \/wml?\nDisallow: \/wml\/?\nDisallow: \/wml\/search?\nDisallow: \/xhtml?\nDisallow: \/xhtml\/?\nDisallow: \/xhtml\/search?\nDisallow: \/xml?\nDisallow: \/imode?\nDisallow: \/imode\/?\nDisallow: \/imode\/search?\nDisallow: \/jsky?\nDisallow: \/jsky\/?\nDisallow: \/jsky\/search?\nDisallow: \/pda?\nDisallow: \/pda\/?\nDisallow: \/pda\/search?\nDisallow: \/sprint_xhtml\nDisallow: \/sprint_wml\nDisallow: \/pqa\nDisallow: \/palm\nDisallow: \/gwt\/\nDisallow: \/purchases\nDisallow: \/bsd?\nDisallow: \/linux?\nDisallow: \/mac?\nDisallow: \/microsoft?\nDisallow: \/unclesam?\nDisallow: \/answers\/search?q=\nDisallow: \/local?\nDisallow: \/local_url\nDisallow: \/shihui?\nDisallow: \/shihui\/\nDisallow: \/froogle?\nDisallow: \/products?\nDisallow: \/froogle_\nDisallow: \/product_\nDisallow: \/products_\nDisallow: \/products;\nDisallow: \/print\nDisallow: \/books\/\nDisallow: \/bkshp?*q=*\nDisallow: \/books?*q=*\nDisallow: \/books?*output=*\nDisallow: \/books?*pg=*\nDisallow: \/books?*jtp=*\nDisallow: \/books?*jscmd=*\nDisallow: \/books?*buy=*\nDisallow: \/books?*zoom=*\nAllow: \/books?*q=related:*\nAllow: \/books?*q=editions:*\nAllow: \/books?*q=subject:*\nAllow: \/books\/about\nAllow: \/booksrightsholders\nAllow: \/books?*zoom=1*\nAllow: \/books?*zoom=5*\nDisallow: \/ebooks\/\nDisallow: \/ebooks?*q=*\nDisallow: \/ebooks?*output=*\nDisallow: \/ebooks?*pg=*\nDisallow: \/ebooks?*jscmd=*\nDisallow: \/ebooks?*buy=*\nDisallow: \/ebooks?*zoom=*\nAllow: \/ebooks?*q=related:*\nAllow: \/ebooks?*q=editions:*\nAllow: \/ebooks?*q=subject:*\nAllow: \/ebooks?*zoom=1*\nAllow: \/ebooks?*zoom=5*\nDisallow: \/patents?\nDisallow: \/patents\/download\/\nDisallow: \/patents\/pdf\/\nDisallow: \/patents\/related\/\nDisallow: \/scholar\nDisallow: \/citations?\nAllow: \/citations?user=\nDisallow: \/citations?*cstart=\nAllow: \/citations?view_op=new_profile\nAllow: \/citations?view_op=top_venues\nDisallow: \/complete\nDisallow: \/s?\nDisallow: \/sponsoredlinks\nDisallow: \/videosearch?\nDisallow: \/videopreview?\nDisallow: \/videoprograminfo?\nAllow: \/maps?*output=classic*\nAllow: \/maps\/api\/js?\nAllow: \/maps\/d\/\nDisallow: \/maps?\nDisallow: \/mapstt?\nDisallow: \/mapslt?\nDisallow: \/maps\/stk\/\nDisallow: \/maps\/br?\nDisallow: \/mapabcpoi?\nDisallow: \/maphp?\nDisallow: \/mapprint?\nDisallow: \/maps\/api\/js\/\nDisallow: \/maps\/api\/staticmap?\nDisallow: \/mld?\nDisallow: \/staticmap?\nDisallow: \/places\/\nAllow: \/places\/$\nDisallow: \/maps\/preview\nDisallow: \/maps\/place\nDisallow: \/help\/maps\/streetview\/partners\/welcome\/\nDisallow: \/help\/maps\/indoormaps\/partners\/\nDisallow: \/lochp?\nDisallow: \/center\nDisallow: \/ie?\nDisallow: \/sms\/demo?\nDisallow: \/katrina?\nDisallow: \/blogsearch?\nDisallow: \/blogsearch\/\nDisallow: \/blogsearch_feeds\nDisallow: \/advanced_blog_search\nDisallow: \/uds\/\nDisallow: \/chart?\nDisallow: \/transit?\nDisallow: \/mbd?\nDisallow: \/extern_js\/\nDisallow: \/xjs\/\nDisallow: \/calendar\/feeds\/\nDisallow: \/calendar\/ical\/\nDisallow: \/cl2\/feeds\/\nDisallow: \/cl2\/ical\/\nDisallow: \/coop\/directory\nDisallow: \/coop\/manage\nDisallow: \/trends?\nDisallow: \/trends\/music?\nDisallow: \/trends\/hottrends?\nDisallow: \/trends\/viz?\nDisallow: \/trends\/embed.js?\nDisallow: \/trends\/fetchComponent?\nDisallow: \/notebook\/search?\nDisallow: \/musica\nDisallow: \/musicad\nDisallow: \/musicas\nDisallow: \/musicl\nDisallow: \/musics\nDisallow: \/musicsearch\nDisallow: \/musicsp\nDisallow: \/musiclp\nDisallow: \/browsersync\nDisallow: \/call\nDisallow: \/archivesearch?\nDisallow: \/archivesearch\/url\nDisallow: \/archivesearch\/advanced_search\nDisallow: \/base\/reportbadoffer\nDisallow: \/urchin_test\/\nDisallow: \/movies?\nDisallow: \/codesearch?\nDisallow: \/codesearch\/feeds\/search?\nDisallow: \/wapsearch?\nDisallow: \/safebrowsing\nAllow: \/safebrowsing\/diagnostic\nAllow: \/safebrowsing\/report_badware\/\nAllow: \/safebrowsing\/report_error\/\nAllow: \/safebrowsing\/report_phish\/\nDisallow: \/reviews\/search?\nDisallow: \/orkut\/albums\nAllow: \/jsapi\nDisallow: \/views?\nDisallow: \/c\/\nDisallow: \/cbk\nAllow: \/cbk?output=tile&cb_client=maps_sv\nDisallow: \/recharge\/dashboard\/car\nDisallow: \/recharge\/dashboard\/static\/\nDisallow: \/translate_a\/\nDisallow: \/translate_c\nDisallow: \/translate_f\nDisallow: \/translate_static\/\nDisallow: \/translate_suggestion\nDisallow: \/profiles\/me\nAllow: \/profiles\nDisallow: \/s2\/profiles\/me\nAllow: \/s2\/profiles\nAllow: \/s2\/oz\nAllow: \/s2\/photos\nAllow: \/s2\/search\/social\nAllow: \/s2\/static\nDisallow: \/s2\nDisallow: \/transconsole\/portal\/\nDisallow: \/gcc\/\nDisallow: \/aclk\nDisallow: \/cse?\nDisallow: \/cse\/home\nDisallow: \/cse\/panel\nDisallow: \/cse\/manage\nDisallow: \/tbproxy\/\nDisallow: \/imesync\/\nDisallow: \/shenghuo\/search?\nDisallow: \/support\/forum\/search?\nDisallow: \/reviews\/polls\/\nDisallow: \/hosted\/images\/\nDisallow: \/ppob\/?\nDisallow: \/ppob?\nDisallow: \/adwordsresellers\nDisallow: \/accounts\/ClientLogin\nDisallow: \/accounts\/ClientAuth\nDisallow: \/accounts\/o8\nAllow: \/accounts\/o8\/id\nDisallow: \/topicsearch?q=\nDisallow: \/xfx7\/\nDisallow: \/squared\/api\nDisallow: \/squared\/search\nDisallow: \/squared\/table\nDisallow: \/toolkit\/\nAllow: \/toolkit\/*.html\nDisallow: \/globalmarketfinder\/\nAllow: \/globalmarketfinder\/*.html\nDisallow: \/qnasearch?\nDisallow: \/app\/updates\nDisallow: \/sidewiki\/entry\/\nDisallow: \/quality_form?\nDisallow: \/labs\/popgadget\/search\nDisallow: \/buzz\/post\nDisallow: \/compressiontest\/\nDisallow: \/analytics\/reporting\/\nDisallow: \/analytics\/admin\/\nDisallow: \/analytics\/web\/\nDisallow: \/analytics\/feeds\/\nDisallow: \/analytics\/settings\/\nAllow: \/alerts\/manage\nAllow: \/alerts\/remove\nDisallow: \/alerts\/\nAllow: \/alerts\/$\nDisallow: \/ads\/search?\nDisallow: \/ads\/plan\/action_plan?\nDisallow: \/ads\/plan\/api\/\nDisallow: \/phone\/compare\/?\nDisallow: \/travel\/clk\nDisallow: \/hotelfinder\/rpc\nDisallow: \/hotels\/rpc\nDisallow: \/flights\/rpc\nDisallow: \/commercesearch\/services\/\nDisallow: \/evaluation\/\nDisallow: \/chrome\/browser\/mobile\/tour\nDisallow: \/compare\/*\/apply*\nDisallow: \/forms\/perks\/\nDisallow: \/baraza\/*\/search\nDisallow: \/baraza\/*\/report\nDisallow: \/shopping\/suppliers\/search\nDisallow: \/ct\/\nDisallow: \/edu\/cs4hs\/\nDisallow: \/trustedstores\/s\/\nDisallow: \/trustedstores\/tm2\nDisallow: \/trustedstores\/verify\nDisallow: \/adwords\/proposal\nDisallow: \/shopping\/product\/\nDisallow: \/shopping\/seller\nDisallow: \/shopping\/reviewer\nDisallow: \/about\/careers\/apply\/\nDisallow: \/about\/careers\/applications\/\nDisallow: \/landing\/signout.html\nAllow: \/gb\/images\nAllow: \/gb\/js\nDisallow: \/gallery\/\nAllow: \/chromecast\/setup$\nAllow: \/chromecast\/setup\/$\nSitemap: http:\/\/www.gstatic.com\/culturalinstitute\/sitemaps\/www_google_com_culturalinstitute\/sitemap-index.xml\nSitemap: https:\/\/www.google.com\/edu\/sitemap.xml\nSitemap: https:\/\/www.google.com\/work\/sitemap.xml\nSitemap: http:\/\/www.google.com\/hostednews\/sitemap_index.xml\nSitemap: http:\/\/www.google.com\/maps\/views\/sitemap.xml\nSitemap: http:\/\/www.google.com\/sitemaps_webmasters.xml\nSitemap: http:\/\/www.google.com\/ventures\/sitemap_ventures.xml\nSitemap: http:\/\/www.gstatic.com\/dictionary\/static\/sitemaps\/sitemap_index.xml\nSitemap: http:\/\/www.gstatic.com\/earth\/gallery\/sitemaps\/sitemap.xml\nSitemap: http:\/\/www.gstatic.com\/s2\/sitemaps\/profiles-sitemap.xml\nSitemap: http:\/\/www.gstatic.com\/trends\/websites\/sitemaps\/sitemapindex.xml\nSitemap: http:\/\/www.google.com\/adwords\/sitemap.xml\nSitemap: http:\/\/www.google.com\/drive\/sitemap.xml\nSitemap: http:\/\/www.google.com\/docs\/sitemaps.xml\nSitemap: http:\/\/www.google.com\/sheets\/sitemaps.xml\nSitemap: http:\/\/www.google.com\/slides\/sitemaps.xml\nSitemap: http:\/\/www.google.com\/forms\/sitemaps.xml\n<\/pre>\n","protected":false},"excerpt":{"rendered":"