Disturbing!

[azekeil]

As a direct result of my last post, I've discovered LJ has about 10 'spiders' monitoring new posts for images and then grabbing them from the servers they're hosted on. That means there are at least 10 bots downloading every new post; the amount of traffic this generates must be considerable! I wonder if this is an LJ-provided function, or if they're individually going to grab their own copies of the pages. Now I think about it, it MUST be LJ-provided, surely, because I don't know of a function or way to grab all new pages.

UPDATE: Apparently LJ does support the use of bots, and even hosts XML showing the URLs of the latest X pictures. Mystery solved, although it's a little anti-social for people just to copy and host the PHP code to do this. Kills home web servers for a while...

Comments

[zeke-hubris.livejournal.com]

I've seen a couple of things in lj_nifty that show all recently posted images. I'm sure some of them were privately done projects.

[azekeil.livejournal.com]

I've updated the post with my findings..

[ev1ldonut.livejournal.com]

You can stop that happening, by preventing your posts from appearing on the "latest updates" page. I forget the console command though...

[growf.livejournal.com]

Be aware that images in private or locked posts appear on the XML feed regardless.

I've seen a number of images on 'LJ Recent Images' pages that the owners definitely didn't mean to circulate.

[dylan.livejournal.com]

Love your icon dude, that's a great photo :)

[yellowraver.livejournal.com]

So does that mean that if you post pictures in friends only or private posts, they are then available to the world via that pictures page?

Scary stuff :(

[yellowraver.livejournal.com]

If you find it, please let us all know :)

[ev1ldonut.livejournal.com]

Found it...

Go here and type the command "set latest_optout yes" (without the quotes)

[dennyd.livejournal.com]

Whoa, that's not good.

[yellowraver.livejournal.com]

Thanks you are a star :)