Picture Server Storing Millions of jpg's!

[Shawn]

On Thu, 03 Jan 2008 14:34:42 +1100
James McDonald <james at jamesmcdonald.id.au> wrote:

> Shawn wrote:
> > Hi guys,
> >
> > I have a system of IP camera's that are supposed to FTP snapshots
> > upon motion sensing at a number of remote sites.  The cameras seem
> > to send far more pictures even when things are static and I've
> > ended up with litterally millions of 10-20K jpg's, not all of which
> > I need to save as it were.
> >
> > I am having a great deal of difficulty managing theses pictures.
> > The machines can  seem to handle the shear numbers.  The hardware
> > is an olde Quad XENON 500 with 4G's of RAM and a 300G 6 drive RAID
> > 5 array. EXT3 and XFS didn't seem to help.  Any suggestions for FS
> > or application(s) that may be helpful in this sort of situation?
> >
> > Shawn
> >   
> Do some testing on the camera's so they sense only what they are 
> supposed to. If you are interested only in human activity perhaps you 
> have insects etc triggering the uploads.
> 
> Each time you ftp something up do you have the camera's
> authenticating themselves is PAM a bottle neck?
> 
> The photos are stored on the file system with what sort of file
> structure?
> 
> Is java (resource hog) anywhere in the mix?
> 
> How are you querying the shots. I.e. I want to see all photos from 
> camera number X between the times of Y AM and ZPM.
> 
> When you say shear numbers how many shots is that a second? Knowing
> that will help us to understand if you are saturating your bandwidth.
> 
> Is the RAID 5 software or hardware? Hardware RAID is best for 
> performance. Are the discs IDE, SAS, SCSI? This will tell us whether
> you are trying to push a golf ball through a straw.
> 
> What sort of CPU/Mem usage are you seeing on the boxes?

The structure is a single directory on most of the cameras.  They
simply download a snapshot if they detect motion, which they do even if
a shawdow from a stationary object moves with the sun, no sensitivity
adjustment.  The cameras were a stop gap and cheap, budgetary
restraints.  I had planned to run a cron job every day to break up the
files into subdirectories but I am swamped with other duties and it
never seems to hit the top of the pile as it were.

The RAID is SAS and hardware.  The CPU mem usage isn't bad.  It simply
the number of files.

Shawn