Disturbing!

[azekeil]

As a direct result of my last post, I've discovered LJ has about 10 'spiders' monitoring new posts for images and then grabbing them from the servers they're hosted on. That means there are at least 10 bots downloading every new post; the amount of traffic this generates must be considerable! I wonder if this is an LJ-provided function, or if they're individually going to grab their own copies of the pages. Now I think about it, it MUST be LJ-provided, surely, because I don't know of a function or way to grab all new pages.

UPDATE: Apparently LJ does support the use of bots, and even hosts XML showing the URLs of the latest X pictures. Mystery solved, although it's a little anti-social for people just to copy and host the PHP code to do this. Kills home web servers for a while...