mdraid RAID1 write performance issues

[Lonni J Friedman]

Greetings,
I have two different mdraid RAID1 arrays on a system at home, with
each array comprised of two identical SATA disks, which has existed
for many months without any problems.  One array is comprised of two
80GB disks, and the second is comprised of two identical 500GB disks.
Both arrays use the same SATA controller on the motherboard (with the
same kernel driver).  In the past few weeks I've noticed that write
performance to the array with the 500GB disks is absolutely horrible
most of the time.  Like 10MB/s bad.  The array with the 80GB disks is
reasonably good (about 60MB/s).  Read performance is good everywhere.
While I understand that there will be a performance penalty for RAID1
mirroring, it definitely shouldn't be quite this bad, and the fact
that the other array with the smaller disks is upwards of 5x faster
most of the time suggests that something isn't right somewhere.  Plus
I'm certain that the performance used to be significantly better
(around 40-50MB/s at least).  Both arrays have a single partition on
their disks, formatted xfs.

Another oddity to add to the situation is that occasionally and
unpredictably the performance will spike up to 'normal' levels.  For
example, I can copy large (5 - 40GB) files to the array and it will
hover around 8 or 10MB/s for a while, then spike to 40 or 50MB/s for a
few seconds, then fall back to 8MB/s for a minute or more.

I'm certain that nothing in the background is reading or writing to
the disks so often that its starving the disks for throughput.

I benchmarked the disks using:
dd if=/dev/zero of=foo1 bs=25M count=80

The only thing that's changed on this system over the past few months
is the kernel version (currently using a Fedora16 3.6.7 kernel), so
its quite possible that some weird regression or default setting
changed, but I'm not sure what that might be.  I'm hoping that there's
some /proc setting or something in hdparm that I can tweak to fix
this, as I'm at a complete loss on what might be causing this
misbehavior.

I've compared 'hdparm -I' output from the fast & slow disks, and I
don't see anything obvious that might be different between them.  Both
are set to udma6

I'm reasonably confident that there isn't anything physically wrong
with the disks or the array.  I've run smart tests on them, checked
/proc/mdstat and checked dmesg & messages, and there are no errors or
other abnormalities.  They're just slow for no obvious reason.

I also tried playing with different IO schedulers (via
/sys/block/sdf/queue/scheduler ), but none of them seemed to have any
impact on performance.  The default is cfq.

Does anyone have any suggestions or ideas on what's going on here?

thanks & happy holidays!


-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
L. Friedman                                    netllama at gmail.com
LlamaLand                       https://netllama.linux-sxs.org