Picture Server Storing Millions of jpg's!

[James McDonald]

Shawn wrote:
> On Thu, 03 Jan 2008 14:34:42 +1100
> James McDonald <james at jamesmcdonald.id.au> wrote:
>
>   
>> Shawn wrote:
>>     
>>> Hi guys,
>>>
>>> I have a system of IP camera's that are supposed to FTP snapshots
>>> upon motion sensing at a number of remote sites.  The cameras seem
>>> to send far more pictures even when things are static and I've
>>> ended up with litterally millions of 10-20K jpg's, not all of which
>>> I need to save as it were.
>>>
>>> I am having a great deal of difficulty managing theses pictures.
>>> The machines can  seem to handle the shear numbers.  The hardware
>>> is an olde Quad XENON 500 with 4G's of RAM and a 300G 6 drive RAID
>>> 5 array. EXT3 and XFS didn't seem to help.  Any suggestions for FS
>>> or application(s) that may be helpful in this sort of situation?
>>>
>>> Shawn
>>>   
>>>       
>> Do some testing on the camera's so they sense only what they are 
>> supposed to. If you are interested only in human activity perhaps you 
>> have insects etc triggering the uploads.
>>
>> Each time you ftp something up do you have the camera's
>> authenticating themselves is PAM a bottle neck?
>>
>> The photos are stored on the file system with what sort of file
>> structure?
>>
>> Is java (resource hog) anywhere in the mix?
>>
>> How are you querying the shots. I.e. I want to see all photos from 
>> camera number X between the times of Y AM and ZPM.
>>
>> When you say shear numbers how many shots is that a second? Knowing
>> that will help us to understand if you are saturating your bandwidth.
>>
>> Is the RAID 5 software or hardware? Hardware RAID is best for 
>> performance. Are the discs IDE, SAS, SCSI? This will tell us whether
>> you are trying to push a golf ball through a straw.
>>
>> What sort of CPU/Mem usage are you seeing on the boxes?
>>     
>
> The structure is a single directory on most of the cameras.  They
> simply download a snapshot if they detect motion, which they do even if
> a shawdow from a stationary object moves with the sun, no sensitivity
> adjustment.  The cameras were a stop gap and cheap, budgetary
> restraints. 
> I had planned to run a cron job every day to break up the
> files into subdirectories but I am swamped with other duties and it
> never seems to hit the top of the pile as it were.
>   
This would be good because having 100k filenames in a dir makes any 
software that will read them have a lot of difficulty in just reading in 
the the array of names

Probably best to create that script you mention and then use a year / 
month / day  directory structure with full time 
yyyymmddhhmmss-camera_no.jpeg filenames ....


> The RAID is SAS and hardware.  The CPU mem usage isn't bad.  It simply
> the number of files.
>   
You still haven't mentioned specifically how the server or whoever is 
trying to access the photos is failing. 'can't handle' in what sense?