I needed to duplicate a strange problem in our application. It involved large volumes. So I wrote a script to simulate 17k records. My first test hung the app. After 4 or 5 hours, I gave up. Me and my boss split the work between us. I handled the database optimization, and he took care of the stuff in the app.The database piece turned out to be terribly inefficient code in one of the database triggers. I was surprised at some of the findings in the app. One problem was the sorting of the records. This used a bubble sort implemented by a developer no longer on the team. Turns out he ran the bubble sort on the data twice in order to sort by two columns. That was just too slow. Who doesn't use quicksort these days?
Another issue was the population of our spreadsheet. This is accomplished by adding data one cell at a time. At 17k rows by about 10 columns, you know we are in trouble. We might be capped because we are using a third party spreadsheet. I got to think there is a way to bulk load up large spreadsheets. I hope when we put all our speedups together, this 17k records loads in no time at all.
