Bixeebot: Bixee's Web Crawler for JobsBixeebot is a web-crawling robot. It visits websites containing jobs, education, travel, shopping, finance, autos, news, music and collects snippets of the data to build a searchable index for the Bixee Search |
How to prevent Bixee from crawling your siteIf you wish to exclude your website from Bixee's index, you can place a file at the root of your server called robots.txt. This is the standard protocol that most web crawlers observe for excluding a web server or directory from an index. More information on robots.txt is available here: http://www.robotstxt.org/wc/norobots.html. Please note that Bixeebot does not interpret a 401/403 response ("Unauthorized"/"Forbidden") to a robots.txt fetch as a request not to crawl any pages on the site.To remove your site from Bixee only and prevent just Bixeebot from crawling your site in the future, place the following robots.txt file in your server root: User-agent: Bixeebot Disallow: / Each port must have its own robots.txt file. In particular, if you serve content via both http and https, you'll need a separate robots.txt file for each of these protocols. For example, to allow Bixeebot to index all http pages but no https pages, you'd use the robots.txt files below. For your http protocol (http://yourserver.com/robots.txt): User-agent: Bixeebot Allow: / For the https protocol (https://yourserver.com/robots.txt): User-agent: Bixeebot Disallow: /
Incase if you feel that your request for removal of your site from the Bixee index is urgent or previously crawled data (before you have put in your robots.txt file) still exists in Bixee's index kindly email us immediately.
We will do our best to take action on your request asap. How to submit your site to Bixee for indexingBixeebot normally discovers a web site automatically and starts indexing it unless explicitly prevented from doing so, as described above.However if you want your site to be crawled and indexed urgently kindly email us. We will try to include your site in our index asap. |