Auto reload externally changed files does not work properly

[WalterMeu]

Hi Gustaf,
The Flow CSV Viewer is a very simple and easy to use tool.
It features "Auto reload externally changed files: Immediately when detected", which I would like to use.
Unfortunately, this feature does not behave as expected. It works fine in some cases, but it does work in some other cases.

To be precise, I see two issues. When I run a script that does something like the following (pseudo code)
open_file "testfile.csv" "append"
for i=1 to 1000
write_file "$i;$i+1;$i*$i"
wait 100ms
next
close_file
then this script opens a file and appends a new line every 100ms, and this will be done for about 100 seconds (which is enough time to test the issue).
When I try to open testfile.csv with Flow then I get the error message "Load CSV: Could not open testfile.csv: Cannot access file testfile.csv because it is in use by another process". The same error message will occur when I change the script such that it writes to stdout and I redirect the output of the scipt with "> testfile.csv". This indicates that the file open command that is used in Flow does not accept opening a file that is simultaneously written by another process. Note that changing the script such that it embeds the write_file command between open_file and close_file seems to work better, but after a while Flow will get the same error because somewhen it will hit the short time window when the file is being written. Other tools can handle this, so I am sure there is a solution to this.

The second issue seems to related to the "Date created" and "Date accessed" information of a file (this info can be displayed in Windows Explorer).
From the idea point of view, "Date created" tells the date/time when the file is created (and this date normally is not expected to change as long as the file exists), whereas "Date accessed" tells the date/time when the file is accessed (e.g. when a line is appended). After some tests (with scripts / text editors / ...) I found that when "Date created" remains and "Date accessed" is updated (which intuitively is the expected behaviour when a file is continuously updated), Flow does NOT automatically update the waveform! But Flow does an automatic update in the case when "Date created" is updated (e.g. one of my text editors updates both "Date created" and "Date accessed" when I update and save a file)! I guess this should be simple to be fixed. All you need to do is to check for an update of "Date accessed" rather than "Date created".

Please put these issues on your list of improvements, and I hope you find the chance to fix this somewhen in the near future - because the way Flow behaves now is of limited use for me.

Regards,
Walter

[Waveworks]

Hi Walter,

Thank you for the detailed error report, and sorry about the trouble with the auto reload functionality!

To address the first issue about file access, as far as I know, the Windows operating system doesn't allow opening a file that's still open in another process. I think you need to close the file between each write, which I understand you also have tried. The reason why it still doesn't work relates more to the second issue: Flow actually doesn't look at the file date at all; it's using a .NET feature called FileSystemWatcher, which sends events to Flow whenever a file is changed. However I had to implement a workaround for a problem in FileSystemWatcher which essentially means that Flow can't cope with files changing too frequently, as in your case, and that leads to the file access issue.

I should be able to make a better workaround some day, but I'm afraid the next Flow release is not imminent. While waiting for a fix, I suggest that you try buffering one second's worth of data in your script (so 10 lines in your case I guess) and hence only write the file once every second, closing the file after each write. This should give Flow enough time to reload properly between each write and thus avoid the file access problem.

Again sorry for the trouble, but I hope the above suggestion works for you, although I understand it's not an ideal solution.

Best regards,
Gustaf

[WalterMeu]

Hi Gustaf,

Thanks for looking into this.

I see that I need to live with the workaround for the moment, but I already exptected it...

Regards,
Walter

[WalterMeu]

Hi Gustaf,

There is one more thing that came into my mind.

You wrote "[...] as far as I know, the Windows operating system doesn't allow opening a file that's still open in another process. [...]".

Well, I think there must be a way to open a file for reading that is still open in another process, because always when I open a file with my script and close it only some minutes later after all the data was written, of even if I have a script writes to stdout and I redirect the output of the scipt with "> testfile.csv", then I am always able to open the file during this time with a standard text editor or with the DOS "type" command.

Regards,
Walter

[Waveworks]

Hi again Walter,

Thank you for the added insight. That's an interesting experiment - I'll try it out when I get a chance. Indeed if a text editor or "type" can open the file, Flow should of course also be able to open it.

Best regards,
Gustaf