# robots specifications (Apr 25, 1995, 10:55) # we leave open all pages in the top directory of the server # but we disallow all directories (physical and pass-through) # On 23-Oct-2000 the Disallow for /CATS/ was removed / BSN # On 23-May-2009 we took the final step to disallow all / BSN User-agent: * Disallow: / Disallow: /ACTREP/ Disallow: /COURSES/ Disallow: /DOCUMENTATION/ Disallow: /HEP/ Disallow: /HET/ Disallow: /LIBRARY/ Disallow: /NBARCHIVES/ Disallow: /NBI-HISTORY/ Disallow: /NEWS/ Disallow: /SOFTWARE/ Disallow: /TEST/ Disallow: /TOOLS/ Disallow: /PROTECTED/ Disallow: /ACLUSTERS/ Disallow: /ASTRO/ Disallow: /CONDMAT/ Disallow: /GENTHEO/ Disallow: /METEOMOSS/ Disallow: /NEURALNET/ Disallow: /NUCEXP/ Disallow: /NUCTHEO/ Disallow: /COMPUTATION/ Disallow: /NBIfAFG/ Disallow: /RF/ Disallow: /cgi-bin/ Disallow: /icons/ Disallow: /home/mfk/mfk/WWW