Copying files open for writing – possible data loss
Hello everybody,
so I've been struggling with this problem with WinSCP. I'm using it to transfer files from a financial application to a banking system and every now and then accountants would complain that some information is missing from those files. It turned out that sometimes transfer would be attempted on a file that is still being generated (it takes on average 3-4 minutes to generate a file) and in such case WinSCP copies a part of the file generated sofar, and doesn't issue any warning (even though I use the -delete switch, meaning that I'm actually moving the file). This is a classical case of data loss.
I asked Martin whether this could be fixed, and his answer was "Maybe if more people ask for it.". So, here I am, asking if you could support me in this case.
Description of an issue:
WinSCP allows to copy a file that is opened for writing. This can be useful at times (for example you can copy an Excel sheet that is open), but it can lead to data loss: imagine a situation where a program fetches data from database and saves it to file. If you copy a file while it's not yet ready, you'll get a copy that is missing some part of data, but since you got no warning, you may think that the file is complete. This is even trickier when a transfer is done automatically, periodically from script (which WinSCP is capable of doing and thus is predestined to support).
Proposed solution:
1) On attempt to copy a file that is opened for writing, issue a warning, with possibility to continue or skip.
2) Point 1 should be configurable – with possibility to disable it, but enabled by default (I don't insist on the last one).
3) On attempt to move a file that is opened for writing, issue a warning, with possibility to continue or skip. If user presses "skip", file shouldn't be copied at all (treat "move" as one logical operation, not two separate "copy+delete").
Out of many file managers I tested (in their default settings):
1) File Commander by Brian Havard: warns when copying a file open for writing (allows to continue), and errors when moving such file (copy is not created too).
2) Free Commander by Marek Jasinski: allows copying such file (without a warning), disallows moving such file (copy is not created too)
3) FAR Manager: allows copying such file (without a warning), disallows moving such file (copy is not created too)
4) Unreal Commander: allows copying such file (without a warning), disallows moving such file (copy is not created too)
5) Total Commander: allows copying such file (without a warning), while moving such file, a copy is created, original not deleted, no warning
6) Windows Explorer: allows copying such file (without a warning), disallows moving such file (copy is not created too)
As I'm not a programmer (except for PHP webpages, which doesn't really count in this case, does it?), I'm unable to code a patch myself. But from what I know the fix is rather simple: shouldn't take much time to implement and shouldn't interfere with current functionality.
Regards for all.
so I've been struggling with this problem with WinSCP. I'm using it to transfer files from a financial application to a banking system and every now and then accountants would complain that some information is missing from those files. It turned out that sometimes transfer would be attempted on a file that is still being generated (it takes on average 3-4 minutes to generate a file) and in such case WinSCP copies a part of the file generated sofar, and doesn't issue any warning (even though I use the -delete switch, meaning that I'm actually moving the file). This is a classical case of data loss.
I asked Martin whether this could be fixed, and his answer was "Maybe if more people ask for it.". So, here I am, asking if you could support me in this case.
Description of an issue:
WinSCP allows to copy a file that is opened for writing. This can be useful at times (for example you can copy an Excel sheet that is open), but it can lead to data loss: imagine a situation where a program fetches data from database and saves it to file. If you copy a file while it's not yet ready, you'll get a copy that is missing some part of data, but since you got no warning, you may think that the file is complete. This is even trickier when a transfer is done automatically, periodically from script (which WinSCP is capable of doing and thus is predestined to support).
Proposed solution:
1) On attempt to copy a file that is opened for writing, issue a warning, with possibility to continue or skip.
2) Point 1 should be configurable – with possibility to disable it, but enabled by default (I don't insist on the last one).
3) On attempt to move a file that is opened for writing, issue a warning, with possibility to continue or skip. If user presses "skip", file shouldn't be copied at all (treat "move" as one logical operation, not two separate "copy+delete").
Out of many file managers I tested (in their default settings):
1) File Commander by Brian Havard: warns when copying a file open for writing (allows to continue), and errors when moving such file (copy is not created too).
2) Free Commander by Marek Jasinski: allows copying such file (without a warning), disallows moving such file (copy is not created too)
3) FAR Manager: allows copying such file (without a warning), disallows moving such file (copy is not created too)
4) Unreal Commander: allows copying such file (without a warning), disallows moving such file (copy is not created too)
5) Total Commander: allows copying such file (without a warning), while moving such file, a copy is created, original not deleted, no warning
6) Windows Explorer: allows copying such file (without a warning), disallows moving such file (copy is not created too)
As I'm not a programmer (except for PHP webpages, which doesn't really count in this case, does it?), I'm unable to code a patch myself. But from what I know the fix is rather simple: shouldn't take much time to implement and shouldn't interfere with current functionality.
Regards for all.
