Primero estoy empezando configurando mi archivo robots.txt.
This has not worked, because google and yahoo keep poking their heads in, (notification by email that someone visited)
So I'm thinking of switching it to this:
Insertar CODE, HTML o PHP:
User-agent: Slurp
User-agent: Googlebot
User-agent: msnbot
User-agent: Mediapartners-Google
User-agent: Adsbot-Google
User-agent: ia_archiver-web.archive.org
Disallow: /forum/
Insertar CODE, HTML o PHP:
User-agent: *
Disallow: /
User-agent: *
Disallow: /herring/
This has not worked, because google and yahoo keep poking their heads in, (notification by email that someone visited)
So I'm thinking of switching it to this:
Insertar CODE, HTML o PHP:
User-agent: Slurp
User-agent: Googlebot
User-agent: msnbot
User-agent: Mediapartners-Google
User-agent: Adsbot-Google
User-agent: ia_archiver-web.archive.org
Disallow: /forum/
Disallow: /herring/
Insertar CODE, HTML o PHP:
User-agent: *
Disallow: /
Disallow: /herring/
Disallow: /forum/
¿La segunda entrada parece correcta?