Flow Support Forum

What is the maximum number of rows that Flow CSV viewer supports?

Can it load and plot CSV files with more than 1 million rows?

Hello, there is no upper limit other than your computer's memory. One million rows should be no problem on a modern computer unless each row has very many (several hundred) columns. I myself have tested files with more than a hundred million rows. It takes some time to load and prepare the plots, but then it works fine. Give it a try and let me know if you experience any problems! /Gustaf

Hi,

My laptop has 8G RAM and 2T storage. When I was trying to load a 3G CSV file, only one third of rows (100K) was successfully loaded in Flow CSV Editor. Do you know whether there is an option to increase the limit?

Many thanks,
Frank

Hi Frank,

That sounds like it could be a bug. There is no upper limit other than what fits in memory, and if memory is insufficient, you should get a clear "Out of memory" message - it should definitely not just silently load a part of the data and skip the rest. I guess it would be difficult for you to share the 3 GB data file, but any additional information you can give me about the application behavior would be helpful. I'll try to see if I can replicate problem on my side.

Sorry about the trouble!

Best regards,
Gustaf

Hi Gustaf,

Sure! I am using Flow CSV Editor on Windows 11, installed from Microsoft Store.

The first file I was trying to open has 1048K rows and 84 columns. The file size is about 3GB. I tried to open it in Flow CSV Editor and only first 300K rows was loaded.

The second file I was trying to open also has 1048K rows but 14 columns. The file size is 7.2MB, and only first 800K rows was loaded. There is no error message in the whole process.

Thanks,
Frank

Thank you for the additional information. I wasn't able to replicate the problem despite using larger files with both more rows and columns. So I think the most probable explanation is that there is something in your data that causes the CSV parser to either skip a lot of rows or exit prematurely for some reason.

1. The CSV parser skips rows that only contain whitespace characters (i.e. spaces and tabs), and it skips rows that start with a # character (if that setting is enabled). Can you check that your files don't have many empty rows or rows starting with #?
2. The parser may exit prematurely for many reasons, but for almost all cases, an error message will be shown. However, if the exact string value " EOF" (including the quotes and the space before EOF) is found, it may cause the parser to believe that the end of file has been reached. Is it possible that this string appears in your file?

If this doesn't give you a clue, would you mind sending me the first 10 and the last 10 rows of one of the failing CSV files to me in an email, info@waveworks (add .dk manually)? That way I could try to recreate a similar file and see if I can replicate the same problem.

Grateful for any help you can provide, so I can fix this bug!

Best regards,
Gustaf