any perl performance experts awake?

[Lonni J Friedman]

On Thu, Apr 11, 2013 at 9:32 AM, Andrew Gould <andrewlylegould at gmail.com> wrote:
>
> On Thu, Apr 11, 2013 at 11:13 AM, Lonni J Friedman <netllama at gmail.com>
> wrote:
>>
>>
>> >
>> > Does the script read in an entire data source file and parse each line?
>> > Or
>> > does is read one line at a time and parse/write it prior to reading the
>> > next
>> > line?  If the entire source file is being read into memory, could it be
>> > causing a bottleneck?
>>
>> The script reads in an entire data source file, parsing line by line,
>> putting the data into a hash (%hash_values).  Once that is completed,
>> the hash is passed to sqlInsert().  So everything is already read into
>> memory at the point in time when performance tanks.  I'd expect that
>> this would be the fast path, since its never needed to read from disk.
>>  All of my systems have 2+GB RAM, and the data in question is always
>> less than 30MB, so I can't imagine that this would be a swap issue, if
>> that's what you mean?   Unless querying a key/value pair in a hash is
>> not a good performance path in perl?
>> _______________________________________________
>
>
>
> The script is holding the input file (>150k rows?) and the hash in memory
> while it's reformatting the data and performing sqlinsert().  I was
> wondering whether the combination of processing and RAM utilization could be
> causing the slowdown.

Yes, that's how its behaving.  Is there a better way to do this in perl?