<div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote">On Thu, Apr 11, 2013 at 11:13 AM, Lonni J Friedman <span dir="ltr"><<a href="mailto:netllama@gmail.com" target="_blank">netllama@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
><br>
</div></div><div class="im">> Does the script read in an entire data source file and parse each line? Or<br>
> does is read one line at a time and parse/write it prior to reading the next<br>
> line? If the entire source file is being read into memory, could it be<br>
> causing a bottleneck?<br>
<br>
</div>The script reads in an entire data source file, parsing line by line,<br>
putting the data into a hash (%hash_values). Once that is completed,<br>
the hash is passed to sqlInsert(). So everything is already read into<br>
memory at the point in time when performance tanks. I'd expect that<br>
this would be the fast path, since its never needed to read from disk.<br>
All of my systems have 2+GB RAM, and the data in question is always<br>
less than 30MB, so I can't imagine that this would be a swap issue, if<br>
that's what you mean? Unless querying a key/value pair in a hash is<br>
not a good performance path in perl?<br>
<div class="HOEnZb"><div class="h5">_______________________________________________<br></div></div></blockquote><div><br></div><div><br></div><div style>The script is holding the input file (>150k rows?) and the hash in memory while it's reformatting the data and performing sqlinsert(). I was wondering whether the combination of processing and RAM utilization could be causing the slowdown.</div>
<div style><br></div><div> </div></div></div></div>