Picture Server Storing Millions of jpg's!

[James McDonald]

Shawn wrote:
> Hi guys,
>
> I have a system of IP camera's that are supposed to FTP snapshots upon
> motion sensing at a number of remote sites.  The cameras seem to send
> far more pictures even when things are static and I've ended up with
> litterally millions of 10-20K jpg's, not all of which I need to save as
> it were.
>
> I am having a great deal of difficulty managing theses pictures.  The
> machines can  seem to handle the shear numbers.  The hardware is an
> olde Quad XENON 500 with 4G's of RAM and a 300G 6 drive RAID 5 array.
> EXT3 and XFS didn't seem to help.  Any suggestions for FS or
> application(s) that may be helpful in this sort of situation?
>
> Shawn
>   
Do some testing on the camera's so they sense only what they are 
supposed to. If you are interested only in human activity perhaps you 
have insects etc triggering the uploads.

Each time you ftp something up do you have the camera's authenticating 
themselves is PAM a bottle neck?

The photos are stored on the file system with what sort of file structure?

Is java (resource hog) anywhere in the mix?

How are you querying the shots. I.e. I want to see all photos from 
camera number X between the times of Y AM and ZPM.

When you say shear numbers how many shots is that a second? Knowing that 
will help us to understand if you are saturating your bandwidth.

Is the RAID 5 software or hardware? Hardware RAID is best for 
performance. Are the discs IDE, SAS, SCSI? This will tell us whether you 
are trying to push a golf ball through a straw.

What sort of CPU/Mem usage are you seeing on the boxes?