‘Volatile crawling activity’ is an ongoing struggle for internet sites, but website owners are fighting back.