Picture Server Storing Millions of jpg's!

[Kurt Wall]

On Thu, Jan 03, 2008 at 03:57:56AM +0000, Shawn wrote:
> 
> The structure is a single directory on most of the cameras.  They
> simply download a snapshot if they detect motion, which they do even if
> a shawdow from a stationary object moves with the sun, no sensitivity
> adjustment.  The cameras were a stop gap and cheap, budgetary
> restraints.  I had planned to run a cron job every day to break up the
> files into subdirectories but I am swamped with other duties and it
> never seems to hit the top of the pile as it were.
> 
> The RAID is SAS and hardware.  The CPU mem usage isn't bad.  It simply
> the number of files.

Shawn,

Not to tell you your business, but I guess I'd move writing that
cron job to the top of the list. Dealing with the number of files
should be reasonably straightforward, but definitely demands handling
in an automated fashion. I'd go with XFS for it because it was 
designed with large filesystems in mind and "millions" of files
definitely qualifies as a large filesystem.

To get you started, here's a bletcherous hack I used to organize
a couple thousand JPEGs into directories by year/month/day. It assumes
that the images contain EXIF data and that the date is accurate.
The bash script invokes a PHP script using the PHP CLI interpreter (I
did say it was a bletcherous hack, didn't I?).

<?php
/*
 * Read the EXIF header from an image, ignoring the thumbnail,
 * and return the date/time the image was created as an array:
 * "date" => EXIF.DateTimeOriginal
 * "time" => EXIF.DateTimeOriginal
 */
function get_img_date($image) {
	$exif_hdr = exif_read_data($image, 0, true, false);
	if (false !== $exif_hdr) {
		$pic_date = $exif_hdr['EXIF']['DateTimeOriginal'];
		$img_data['date'] = date("Y/m/d", strtotime($pic_date));
		$img_data['time'] = date("h:i A", strtotime($pic_date));
		return($img_data);
	} else {
		return(false);
	}
}
$d = get_img_date($argn);
echo $d['date'] . "\n";
?>

#!/bin/bash
# sort_imgs.sh - Sort images into date-based directories

# Deal with file names that contain spaces
IFS=$'\n'

# Make the target directories
for F in *JPG
do
	D=`echo $F | php -F get_img_date.php`
	if [ ! -d $D ]; then
		echo -n "Creating directory $D..."
		mkdir -p $D
		[ $? == 0 ] && echo "done"
	fi
done

# Copy the images into their destination directories
for F in *JPG
do
	D=`echo $F | php -F get_img_date.php`
	if [ -d $D ]; then
		echo -n "Copying $F to $D..."
		cp -p $F $D
		if [ $? == 0 ]; then
			 echo "done"
		fi
		rm $F
	else
		echo "$D doesn't exist!"
	fi
done

As always, if it breaks, you get to keep both pieces. It worked for
me, but I'm not responsible if it steals your girlfriend, eats your 
lunch, or mocks you in public.

Kurt
-- 
Meekness is uncommon patience in planning a worthwhile revenge.