Block all bots/crawlers/spiders for a special directory with htaccess

You need to have mod_rewrite enabled. Placed it in .htaccess in that folder. If placed elsewhere (e.g. parent folder) then RewriteRule pattern need to be slightly modified to include that folder name). RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC] RewriteRule .* – [R=403,L] I have entered only few bots — you add any other yourself (letter … Read more

How do search engines deal with AngularJS applications?

Use PushState and Precomposition The current (2015) way to do this is using the JavaScript pushState method. PushState changes the URL in the top browser bar without reloading the page. Say you have a page containing tabs. The tabs hide and show content, and the content is inserted dynamically, either using AJAX or by simply … Read more