tuning NFS for a write heavy environment

[Lonni J Friedman]

On Fri, Sep 26, 2008 at 4:31 AM, David A. Bandel <david.bandel at gmail.com> wrote:
> On Thu, Sep 25, 2008 at 8:47 PM, Lonni J Friedman <netllama at gmail.com> wrote:
>> Anyone have any experience tuning NFS (Linux only) for a write
>> intensive environment ?
>>
>> For background:
>> I've got a Linux NFS server, serving up /home from a 3 SATA disk
>> mdraid RAID5 array (underlying filesystem is XFS).  Alot of developers
>> are using this NFS mount for doing builds, so its almost continuously
>> under heavy write pressure.   The server usually has a bunch of nfsd
>> processes consuming most of the CPU time.
>
> How many nfs mounts are we talking about, and are the developers just
> writing files across this mount or are they actually trying to compile
> across the mount?

/home is the only NFS mount point being exported.  They're compiling C
code across the NFS mount.

>
> Can you add another _good_ ethernet card (full duplex) to the nfs
> server to help distribute the load?

The system isn't bandwidth limited.  It has a dedicated gigabit NIC
just for the NFS traffic, and its never coming even close to
saturating the interface.

>
>>
>> The server itself has a Athlon64X2-6400 (3.2Ghz) CPU, with 2GB of RAM.
>>  Its not swapping at all.
>
> Some rambling thoughts:
>
> Why should it swap?  The bottleneck is your network.  That should fill
> your buffers pretty good.

I was just commenting that this isn't a swapping issue.  I'm not sure
that I understand why you think the network is the bottleneck?

> I have a school that has 16 computers that nfs boot across a network.
> In years past they only had /home mounted and had no problems.  Now
> the entire system is mounted that way, but they run remote X, so all
> the graphics come across the network.
>
> Not exactly the same, but again, your bottleneck will be the network
> -- hope you're not running cheap half-dup cards is all.

Its all gigabit eth, and I've confirmed that the NIC is running at
gigabit speed, with all the client systems also running at gigabit
speeds.  There are no ethernet interface errors on any of the systems.

>
> Try finding something like ntop and looking at your network load (or
> just watch the switch lights at peak load periods to see how many
> collisions you're having).

0.  I see no evidence that this is a network bottleneck.

>
> Check to see if your nfs exports are synchronous or async.  Async
> isn't as safe, but should give a performance boost as it won't block
> the way synchronous does.

Doesn't Linux default to async ?  Anyway, I'm using the default.

>
> Are they all writing to the same files? (file-locking issues?)

No, each person has their own account in /home, with their own files.



-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
L. Friedman                                    netllama at gmail.com
LlamaLand                       https://netllama.linux-sxs.org