User-agent: * Allow: / Disallow: /admin/ Disallow: /writable/ Disallow: /tmp/ Disallow: /search?q= Disallow: /?q= # Avoid duplicate content from query strings Disallow: /colleges?type= Disallow: /colleges?faculty= # Crawl delay for polite bots Crawl-delay: 2 # Sitemap location Sitemap: https://pune.college/sitemap.xml # Google-specific rules User-agent: Googlebot Allow: / Disallow: /admin/ Disallow: /writable/ # Bing User-agent: Bingbot Allow: / Crawl-delay: 5