I have written an algorithm (function) to read in a number of files from a list, manipulate and agglomerate those files, and then store as a single file. This works fine and processes each file in under a second and a half.
To improve the user experience I created a simple GUI to prep the algorithm (input file list, output file location, and processing options etc.). This works fine.
However, the algorithm takes significantly longer to execute on UI thread (Windows Forms Application), over a command line thread (Console Application). I can find no explanation as to why. The command line execution time is close to linear with the number of files. However the UI thread or BackgroundWorker thread execution is very non-linear and very quickly becomes too slow to be useful. See table below. (I killed the 100 file run after 2 hours).
Number of files 1, 3, 5, 13, 20
Command Line Execution Time 1s, 4s, 7s, 19s, 29s
UI BackgroundWorker Execution Time 1s, 7s, 22s, 309s, 441s
I thought this was to do with the BackgroundWorker class I was using to perform the processing, however on removing the backgroundWorker and calling the function directly from a button press caused it to be even slower (82 seconds for 5 files (and caused UI to lock)).
I have now created a standalone static test function, scattered with Stopwatches to performance test and discover the issue.
One thing is highlighted as getting slower and slower with every additional file.
byte canData = reader.ReadBytes(8);
"reader" is BinaryReader. This was used as both simple files, and ZipArchiveEntry files are being processed.
Other lines of code which are highlighted as slowing, are always memory allocations, either a "new" statement, or List<>.Add functions.
My understanding of BinaryReaders involves them buffering incoming information and therefore