page allocation failure

Net Llama! netllama
Fri Feb 9 11:01:01 PST 2007 

Actually, you should google on this a bit, as it looks like it might now 
be as simple as an OOM condition but some kind of driver buffer overflow, 
where it needs to store something in memory somewhere  and doesn't have 
enough room.  One report that I read was discussing someone using jumbo 
frames with a weird MTU.

At anyrate, I don't think this is a HW problem.

On Fri, 9 Feb 2007, C M Reinehr wrote:

> On Friday 09 February 2007 12:02, Net Llama! wrote:
>> That loooks like an OOM to me, not a HW problem.  How much RAM is in this
>> box?
>
> MemTotal:      1026024 kB
> SwapTotal:      979704 kB
>
> Right now, with no backups running:
>
> MemFree:        930624 kB
> SwapFree:       979704 kB
>
> I wonder if something could be preventing the swap from being used.
>
> # swapon -s
> Filename				Type		Size	Used	Priority
> /dev/md2                                partition	979704	0	-1
>
> It's quite possible that I did something wrong 18 months ago that only now is
> surfacing.
>
> TIA
>
> cmr
>> On Fri, 9 Feb 2007, C M Reinehr wrote:
>>> This morning I came in to the office and found one of my servers DOA.
>>> This server functions primarily as a backup server, performing a nightly
>>> network backup to disk & tape and has been working well for the past 18
>>> or so months until now. I've found it this way on the past two Friday
>>> mornings (today & last Friday). On Friday mornings the weekly cycle of
>>> level Full backups starts at 02:35.
>>>
>>> I'm guessing that this is an indication of failing hardware, probably
>>> memory, but would appreciate anyone's insights.
>>>
>>> Following is an except of the syslog. The full syslog, from the time the
>>> problem started until the system stopped responding, is attached.
>>>
>>> Thanks in advance!
>>>
>>> cmr
>>>
>>> Feb  9 02:54:26 Bilskirnir kernel: swapper: page allocation failure.
>>> order:0, mode:0x20
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> Feb  9 02:54:26 Bilskirnir kernel: Call Trace: <IRQ>
>>> <ffffffff8024ee08>{__alloc_pages+664}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff80264f76>{kmem_getpages+70}
>>> <ffffffff8020a04a>{apic_timer_interrupt+98}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff80265cd1>{cache_grow+177}
>>> <ffffffff80265ecc>{cache_alloc_refill+348}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff802662db>{__kmalloc+91} <ffffffff804b3fd6>{__alloc_skb+86}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff803e33ad>{e100_rx_alloc_skb+29}
>>> <ffffffff803e376f>{e100_rx_clean+143}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff803e3aa7>{e100_poll+55} <ffffffff803e2cd0>{e100_watchdog+0}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff804ba299>{net_rx_action+121} <ffffffff8022d98b>{__do_softirq+75}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff8020a6a6>{call_softirq+30} <ffffffff8020c361>{do_softirq+49}
>>> Feb  9 02:54:26 Bilskirnir kernel:        <ffffffff8020c317>{do_IRQ+71}
>>> <ffffffff80207a90>{default_idle+0}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff80209e14>{ret_from_intr+0} <EOI>
>>> <ffffffff8053517f>{thread_return+0}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff80207abe>{default_idle+46} <ffffffff80207bd4>{cpu_idle+68}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff806ff8a6>{start_kernel+358}
>>> <ffffffff806ff2ac>{x86_64_start_kernel+428}
>>> Feb  9 02:54:26 Bilskirnir kernel: Mem-info:
>>> Feb  9 02:54:26 Bilskirnir kernel: DMA per-cpu:
>>> Feb  9 02:54:26 Bilskirnir kernel: cpu 0 hot: high 0, batch 1 used:0
>>> Feb  9 02:54:26 Bilskirnir kernel: cpu 0 cold: high 0, batch 1 used:0
>>> Feb  9 02:54:26 Bilskirnir kernel: DMA32 per-cpu:
>>> Feb  9 02:54:26 Bilskirnir kernel: cpu 0 hot: high 186, batch 31 used:51
>>> Feb  9 02:54:26 Bilskirnir kernel: cpu 0 cold: high 62, batch 15 used:56
>>> Feb  9 02:54:26 Bilskirnir kernel: Normal per-cpu: empty
>>> Feb  9 02:54:26 Bilskirnir kernel: HighMem per-cpu: empty
>>> Feb  9 02:54:26 Bilskirnir kernel: Free pages:        5380kB (0kB
>>> HighMem) Feb  9 02:54:26 Bilskirnir kernel: Active:24574 inactive:214681
>>> dirty:82 writeback:9471 unstable:0 free:1345 slab:14425 mapped:8866
>>> pagetables:262 Feb  9 02:54:26 Bilskirnir kernel: DMA free:3992kB
>>> min:36kB low:44kB high:52kB active:88kB inactive:4396kB present:9384kB
>>> pages_scanned:25
>>> all_unreclaimable? no
>>> Feb  9 02:54:26 Bilskirnir kernel: lowmem_reserve[]: 0 994 994 994
>>> Feb  9 02:54:26 Bilskirnir kernel: DMA32 free:1388kB min:4012kB
>>> low:5012kB high:6016kB active:98208kB inactive:854328kB present:1018020kB
>>> pages_scanned:0 all_unreclaimable? no
>>> Feb  9 02:54:26 Bilskirnir kernel: lowmem_reserve[]: 0 0 0 0
>>> Feb  9 02:54:26 Bilskirnir kernel: Normal free:0kB min:0kB low:0kB
>>> high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
>>> all_unreclaimable? no Feb  9 02:54:26 Bilskirnir kernel:
>>> lowmem_reserve[]: 0 0 0 0
>>> Feb  9 02:54:26 Bilskirnir kernel: HighMem free:0kB min:128kB low:128kB
>>> high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
>>> all_unreclaimable? no
>>> Feb  9 02:54:26 Bilskirnir kernel: lowmem_reserve[]: 0 0 0 0
>>> Feb  9 02:54:26 Bilskirnir kernel: DMA: 0*4kB 1*8kB 1*16kB 0*32kB 0*64kB
>>> 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3992kB
>>> Feb  9 02:54:26 Bilskirnir kernel: DMA32: 1*4kB 1*8kB 0*16kB 1*32kB
>>> 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1388kB
>>> Feb  9 02:54:26 Bilskirnir kernel: Normal: empty
>>> Feb  9 02:54:26 Bilskirnir kernel: HighMem: emff80265cd1>{cache_grow+177}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff80265ecc>{cache_alloc_refill+348}
>>> <ffffffff802662db>{__kmalloc+91} Feb  9 02:54:26 Bilskirnir kernel:
>>>  <ffffffff804b3fd6>{__alloc_skb+86}
>>> <ffffffff803e33ad>{e100_rx_alloc_skb+29}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff803e376f>{e100_rx_clean+143} <ffffffff803e3aa7>{e100_poll+55}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff804ba299>{net_rx_action+121} <ffffffff8022d98b>{__do_softirq+75}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff8020a6a6>{call_softirq+30} <ffffffff8020c361>{do_softirq+49}
>>> Feb  9 02:54:26 Bilskirnir kernel:        <ffffffff8020c317>{do_IRQ+71}
>>> <ffffffff80207a90>{default_idle+0}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff80209e14>{ret_from_intr+0} <EOI>
>>> <ffffffff8053517f>{thread_return+0}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff80207abe>{default_idle+46} <ffffffff80207bd4>{cpu_idle+68}
>>> Feb  9 02:54:26 Bilskirnir kernel:
>>> <ffffffff806ff8a6>{start_kernel+358}
>>> <ffffffff806ff2ac>{x86_64_start_kernel+428}
>>> Feb  9 02:54:26 Bilskirnir kernel: Mem-info:
>>> Feb  9 02:54:26 Bilskirnir kernel: DMA per-cpu:
>>> Feb  9 02:54:26 Bilskirnir kernel: cpu 0 hot: high 0, batch 1 used:0
>>> Feb  9 02:54:26 Bilskirnir kernel: cpu 0 cold: high 0, batch 1 used:0
>>> Feb  9 02:54:26 Bilskirnir kernel: DMA32 per-cpu:
>>> Feb  9 02:54:26 Bilskirnir kernel: cpu 0 hot: high 186, batch 31 used:51
>>> Feb  9 02:54:26 Bilskirnir kernel: cpu 0 cold: high 62, batch 15 used:56
>>> Feb  9 02:54:26 Bilskirnir kernel: Normal per-cpu: empty
>>> Feb  9 02:54:26 Bilskirnir kernel: HighMem per-cpu: empty
>>> Feb  9 02:54:26 Bilskirnir kernel: Free pages:        5380kB (0kB
>>> HighMem) Feb  9 02:54:26 Bilskirnir kernel: Active:24574 inactive:214681
>>> dirty:82 writeback:9471 unstable:0 free:1345 slab:14425 mapped:8866
>>> pagetables:262 Feb  9 02:54:26 Bilskirnir kernel: DMA free:3992kB
>>> min:36kB low:44kB high:52kB active:88kB inactive:4396kB present:9384kB
>>> pages_scanned:25
>>> all_unreclaimable? no
>>> Feb  9 02:54:26 Bilskirnir kernel: lowmem_reserve[]: 0 994 994 994
>>> Feb  9 02:54:26 Bilskirnir kernel: DMA32 free:1388kB min:4012kB
>>> low:5012kB high:6016kB active:98208kB inactive:854328kB present:1018020kB
>>> pages_scanned:0 all_unreclaimable? no
>>> Feb  9 02:54:26 Bilskirnir kernel: lowmem_reserve[]: 0 0 0 0
>>> Feb  9 02:54:26 Bilskirnir kernel: Normal free:0kB min:0kB low:0kB
>>> high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
>>> all_unreclaimable? no Feb  9 02:54:26 Bilskirnir kernel:
>>> lowmem_reserve[]: 0 0 0 0
>>> Feb  9 02:54:26 Bilskirnir kernel: HighMem free:0kB min:128kB low:128kB
>>> high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
>>> all_unreclaimable? no
>>> Feb  9 02:54:26 Bilskirnir kernel: lowmem_reserve[]: 0 0 0 0
>>> Feb  9 02:54:26 Bilskirnir kernel: DMA: 0*4kB 1*8kB 1*16kB 0*32kB 0*64kB
>>> 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3992kB
>>> Feb  9 02:54:26 Bilskirnir kernel: DMA32: 1*4kB 1*8kB 0*16kB 1*32kB
>>> 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1388kB
>>> Feb  9 02:54:26 Bilskirnir kernel: Normal: empty
>>> Feb  9 02:54:26 Bilskirnir kernel: HighMem: empty
>>> Feb  9 02:54:26 Bilskirnir kernel: Swap cache: add 1, delete 1, find 0/0,
>>> race 0+0
>>> Feb  9 02:54:26 Bilskirnir kernel: Free swap  = 979700kB
>>> Feb  9 02:54:26 Bilskirnir kernel: Total swap = 979704kB
>>> Feb  9 02:54:26 Bilskirnir kernel: Free swap:       979700kB
>>> Feb  9 02:54:26 Bilskirnir kernel: 262128 pages of RAM
>>> Feb  9 02:54:26 Bilskirnir kernel: 5622 reserved pages
>>> Feb  9 02:54:26 Bilskirnir kernel: 235851 pages shared
>>> Feb  9 02:54:26 Bilskirnir kernel: 0 pages swap cached
>
>

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lonni J Friedman                        netllama at linux-sxs.org
LlamaLand				http://netllama.linux-sxs.org

More information about the Linux-users mailing list