The rules apply to every crawler (User-Agent: *). When rules overlap, the most specific path wins — so /api/records/ stays open for Google Scholar even though the rest of /api/ is blocked. robots.txt is a politeness standard, not security: private areas are also protected by sign-in.
Allowed — please index
/The public website and home page./repositoryBrowse and search the open-access collection./records/Each research record page (with Google Scholar and Dublin Core metadata)./api/records/Public full-text files — the citation_pdf_url Google Scholar needs to index each paper./api/oaiThe OAI-PMH endpoint, so harvesters that honour robots.txt can collect metadata./authorsAuthor directory and profile pages./documentsPublic downloads: prospectus, almanac, fee structure./aboutAbout the repository./interoperabilityThis explainer and the human-readable endpoint views./privacyPolicy pages./end-user-agreementPolicy pages./accessibilityPolicy pages./feedbackFeedback form.
Blocked — please stay out
/adminAdministrator portal — private./dashboardStaff portal — private./studentStudent portal — private./accountReader account pages — private./api/Internal application endpoints (the public ones above are explicitly allowed)./loginSign-in has no search value./registerAccount forms have no search value./verify-emailOne-time links./change-passwordAccount security page./forgot-passwordAccount security page./reset-passwordOne-time links.
robots.txtOpen raw
User-Agent: * Allow: / Allow: /repository Allow: /records/ Allow: /api/records/ Allow: /api/oai Allow: /authors Allow: /documents Allow: /about Allow: /interoperability Allow: /privacy Allow: /end-user-agreement Allow: /accessibility Allow: /feedback Disallow: /admin Disallow: /dashboard Disallow: /student Disallow: /account Disallow: /api/ Disallow: /login Disallow: /register Disallow: /verify-email Disallow: /change-password Disallow: /forgot-password Disallow: /reset-password Host: https://repository.slads.sc.tz Sitemap: https://repository.slads.sc.tz/sitemap.xml
Sitemap pointer
The last line tells crawlers where the full list of public pages is.
View the sitemap →