Home » General » Alex » Block Spiders from your website
Some spiders do not respect the robots.txt. In these cases you can use the Apache rewrite engine to block them.
Some rule examples can be found below.RewriteCond %{HTTP_USER_AGENT} ^Yahoo! Slurp [OR]RewriteCond %{HTTP_USER_AGENT} ^.*slurp.*$ [OR]RewriteCond %{HTTP_USER_AGENT} ^TurnitinBot [OR]RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]RewriteCond %{HTTP_USER_AGENT} ^BoardReader [OR]RewriteCond %{HTTP_USER_AGENT} ^Bot [OR]RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]RewriteCond %{HTTP_USER_AGENT} cometsearch [OR]RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]RewriteCond %{HTTP_USER_AGENT} ^ConveraCrawler [OR]RewriteCond %{HTTP_USER_AGENT} ^Download Demon [OR]RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]RewriteCond %{HTTP_USER_AGENT} ^EmailCollector [OR]RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]RewriteCond %{HTTP_USER_AGENT} ^Express WebPictures [OR]RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]RewriteCond %{HTTP_USER_AGENT} ^GornKer [OR]RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]RewriteCond %{HTTP_USER_AGENT} ^Image Stripper [OR]RewriteCond %{HTTP_USER_AGENT} ^Image Sucker [OR]RewriteCond %{HTTP_USER_AGENT} Indy Library [NC,OR]RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]RewriteCond %{HTTP_USER_AGENT} ^Internet Ninja [OR]RewriteCond %{HTTP_USER_AGENT} ^Irvine [OR]RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]RewriteCond %{HTTP_USER_AGENT} ^JOC Web Spider [OR]RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]RewriteCond %{HTTP_USER_AGENT} ^Mass Downloader [OR]RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]RewriteCond %{HTTP_USER_AGENT} ^MIDown tool [OR]RewriteCond %{HTTP_USER_AGENT} ^Mister PiX [OR]RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]RewriteCond %{HTTP_USER_AGENT} ^Net Vampire [OR]RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]RewriteCond %{HTTP_USER_AGENT} ^oegp [OR]RewriteCond %{HTTP_USER_AGENT} ^Offline Explorer [OR]RewriteCond %{HTTP_USER_AGENT} ^Offline Navigator [OR]RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]RewriteCond %{HTTP_USER_AGENT} ^Papa Foto [OR]RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]RewriteCond %{HTTP_USER_AGENT} dloader(NaverRobot) [OR]RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]RewriteCond %{HTTP_USER_AGENT} ^SearchExpress [OR]RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]RewriteCond %{HTTP_USER_AGENT} ^Teleport Pro [OR]RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]RewriteCond %{HTTP_USER_AGENT} ^W3CRobot [OR]RewriteCond %{HTTP_USER_AGENT} ^Web Image Collector [OR]RewriteCond %{HTTP_USER_AGENT} ^Web Sucker [OR]RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]RewriteCond %{HTTP_USER_AGENT} ^WebBandit [OR]RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]RewriteCond %{HTTP_USER_AGENT} ^WebGo IS [OR]RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]RewriteCond %{HTTP_USER_AGENT} ^Website eXtractor [OR]RewriteCond %{HTTP_USER_AGENT} ^Website Quester [OR]RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]RewriteCond %{HTTP_USER_AGENT} ^WSB [OR]RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]RewriteCond %{HTTP_USER_AGENT} ^Xaldon WebSpider [OR]RewriteCond %{HTTP_USER_AGENT} ^Zeus [OR]RewriteCond %{HTTP_USER_AGENT} ^ZyBorgRewriteRule ^.*$ http://127.0.0.1/ [R,L]
Alexander Gräf
Portfolio
Book Me
Get in touch
Live Chat
I am a full-stack developer. My expertise include:
I love programming, design and know my way around server architecture as well. I would never feel complete, with one of these missing.
I have a broad range of interests, that’s why I constantly dive into new technologies and expand my knowledge where ever required. Technologies are evolving fast and I enjoy using the latest.
Apart from that, I am a peace loving guy who tries to have people around him that think the same. I truly believe in the principle: “If you help someone, someone will help you, when you need it."