Will interrupted transfers be retransferred?

Hi Martin,

I have a dodgy Internet connection that sometimes drops in the middle of a transfer. I was hoping you could answer this question: if I retransfer the files that I was originally trying to transfer, but I choose "New and updated file(s) only" to save time, will a file which was interrupted last time be transferred again?

What I mean is, suppose that a file was in the middle of being transferred but was interrupted, so only half of it was copied to the remote machine. WinSCP terminates the session and asks me to log in again. Then suppose I try and retransfer the file but also choose "New and updated file(s) only". Does WinSCP compare the file size and see that the file needs transferring, or does it just compare the time stamps (which would be the same) and decide that the file does /not/ need transferring?

Cheers then,
Anton

Have you read documentation? :-)

Not since that article was written :) Sorry


Has the behaviour described in the article always been present, or is it something which has been added in a recent version?


Also, I'm not convinced that it works:

I was transferring a folder today when the transfer was interrupted by loss of connection. I re-logged in and retransferred the folder, choosing only to copy new and updated files, and the transfer completed successfully. I have just looked in the folder, and I have a file to which something bad has happened:
only a quarter of the file is there, and while every other file in the folder has permissions rw------- (which is the default I always apply to transfers), the corrupt file has permissions rw-r--r--. I have taken a look at yesterday's backup, and the file was transferred correctly yesterday (full size, with correct permissions).

So it looks like something has gone wrong?

Transfer resuming is available since the SFTP protocol was implemented (more than 2 years ago).

Do you use SFTP? With SCP the resuming is not possible.

Yes, it's SFTP that I'm using. And I do remember seeing .filepart files occasionally before, due to interrupted transfers. Probably the situation I described in the post above was an anomaly (= human error ;-)); I'll let you know if it happens again.


Best wishes,
Anton

Hi Martin,

Well, my connection was interrupted again, so this time I took a careful look to see what the situation was. As before, some files were transferred correctly, and one file was only partially transferred. The truncated remote copy had the same name as the original (e.g. "foo.tex"), and did not have a .filepart extension. Also, its permissions were incorrect: rw-r--r-- instead of what it should have been (and what the correctly transferred files were): rw-------.

However, I can confirm that the "New and updated files only" does notice the difference and will retransfer the file that was truncated. So comparison must be based on file size and not just on filename and timestamp.

Best wishes,
Anton

Anton P. wrote:

Well, my connection was interrupted again, so this time I took a careful look to see what the situation was. As before, some files were transferred correctly, and one file was only partially transferred. The truncated remote copy had the same name as the original (e.g. "foo.tex"), and did not have a .filepart extension.

Note that there is default 100 kB limit to enable transfer resuming. If the file was smaller, it won't have the .filepart extension.

Also, its permissions were incorrect: rw-r--r-- instead of what it should have been (and what the correctly transferred files were): rw-------.

Permissions are set only after the transfer finishes.

However, I can confirm that the "New and updated files only" does notice the difference and will retransfer the file that was truncated. So comparison must be based on file size and not just on filename and timestamp.

It is not :-) Only timestamp is used.

martin wrote:

Note that there is default 100 kB limit to enable transfer resuming. If the file was smaller, it won't have the .filepart extension.

Agreed, and I had this 100K limit set. However, I see from the documentation that 'text mode resume' is not supported. I think that this was the problem in my earlier post, as my "foo.tex" file is slightly bigger than 100K---it's a long file!

Is 'text mode resume' unsupported even if you choose "Resume all files (not recommended)" from the resume preferences? Either way, I think that there should be a note on the preferences dialog informing the user of this, and we should make a clarification in the documentation regarding this. Also, could you explain *why* resuming all files is "not recommended"?

It seems to me that the resume functionality is not much use for backup purposes unless it applies to every file, regardless of size or transfer mode. Without this, if there is an interruption you either have to retransfer the whole job (because the interrupted file might have been a small file, or a text-mode file) or you have to manually search through the remote system using the folder compare feature in order to find the interrupted file (which might be buried deep in the tree being transferred.)


Most importantly, something still seems to be going wrong with the current resume functionality. I have done some tests: I have a file called "foo.dvi" which is bigger than 100K and which will be transferred as binary not as text. If I drag that file across and refresh the remote panel while it is transferring, I can see the .filepart extension, and if I disconnect and then retransfer, WinSCP will ask whether I want to resume. Great!

But here's the problem that (as I now realize) has been affecting me: my "usual" backup job involves me dragging a folder across, not individual files, and "foo.dvi" is inside that folder. If I do not refresh or change directory in the remote panel, obviously I cannot see what extension is being put on the "foo.dvi" file. But if I manually disconnect, I reconnect and enter the remote folder and inspect the file, then
1) it does not have a .filepart extension;
2) its timestamp is set to the transfer time not the Windows timestamp;
3) its permissions are incorrect.
(It has the time of the transfer, not the Windows timestamp; presumably the remote timestamp is modified after transfer has completed, at the same time as the permissions?)

I have then tested by actually refreshing the remote panel and entering the folder while the transfer is taking place, and by doing lots of refreshes I can see that none of the files inside that folder is being given the .filepart extension as the transfer progresses, even if they are >100K and are binary. So it seems that the resume mechanism is not working inside folders, only at the "top level" of the transfer.

Best wishes,
Anton

Anton P. wrote:

Is 'text mode resume' unsupported even if you choose "Resume all files (not recommended)" from the resume preferences?

Yes.

Either way, I think that there should be a note on the preferences dialog informing the user of this, and we should make a clarification in the documentation regarding this.

Well, I'm not going to explain this on the dialog. It is too complex. But I can change "All files" to something like "Disregarding size".

Also, could you explain *why* resuming all files is "not recommended"?

Checking for existence of partial file takes time and bandwidth. Doing so for all files, when transferring thousands of small files can degrade performace by tens of percents.

It seems to me that the resume functionality is not much use for backup purposes unless it applies to every file, regardless of size or transfer mode.

Use synchronisation :-)

I have then tested by actually refreshing the remote panel and entering the folder while the transfer is taking place, and by doing lots of refreshes I can see that none of the files inside that folder is being given the .filepart extension as the transfer progresses, even if they are >100K and are binary. So it seems that the resume mechanism is not working inside folders, only at the "top level" of the transfer.

There definitely seems to be a bug. I'll check it. Thanks!

Is that bug already fixed?

martin wrote:

Anton P. wrote:

Is 'text mode resume' unsupported even if you choose "Resume all files (not recommended)" from the resume preferences?

Yes.

Either way, I think that there should be a note on the preferences dialog informing the user of this, and we should make a clarification in the documentation regarding this.

Well, I'm not going to explain this on the dialog. It is too complex. But I can change "All files" to something like "Disregarding size".

Also, could you explain *why* resuming all files is "not recommended"?

Checking for existence of partial file takes time and bandwidth. Doing so for all files, when transferring thousands of small files can degrade performace by tens of percents.

It seems to me that the resume functionality is not much use for backup purposes unless it applies to every file, regardless of size or transfer mode.

Use synchronisation :-)

I have then tested by actually refreshing the remote panel and entering the folder while the transfer is taking place, and by doing lots of refreshes I can see that none of the files inside that folder is being given the .filepart extension as the transfer progresses, even if they are >100K and are binary. So it seems that the resume mechanism is not working inside folders, only at the "top level" of the transfer.

There definitely seems to be a bug. I'll check it. Thanks!

Hi.
I am using command line to synchronise files. I have only been using for a few weeks which have been fine fine but now I have 2 problems which are related.
This is with SFTP, using version 4.3.7.0
1. I get an error message "Bad window size in WINDOW_ADJUST". I do not know why I get this message and cannot find anything to help me explain. This by itself is not a big problem (but it would be nice to understand and fix it) because WINSCP reconnects automatically and tries to continue.
2. This is the bigger problem. If I get the above error, WinSCP reconnects and carries on - BUT it redownloads the entire file into the existing partfile at the end of the data which is already there. Surely, the whole point of resuming is not to replicate the data? So I end up with a file which contains 2 copies of all the data prior to disconnection (in this example below the first 1.5MB or so) and one copy of the rest of the file. I also get no notification that this has happened except in the CMD window. Is there a fix for this or is there something I can change on the command line or in the txt file to make this problem go away without having to use ini file? I have tried to read all the relevant documentation but cannot find answer to this.

The command I use is
winscp.com /open %user%:%pass%@%server% -hostkey=%serverhostkey% /script=Weeklydataall.txt

and the contents of weeklydataall.txt is

option batch on
option confirm off
cd quiz
option include *.TXT*
synchronize local
exit


Thanks very much.

Hi

I have some more information on this but it does not answer why the resume does not happen properly. It appears that the connection is interrupted when 1GB in total has been transferred per session. When 1 GB is reached, the transfer is interrupted, then WinSCP reconnects and starts the interrupted file again copying down into the existing partfile but appends the entire file to the existing partfile as previously described.
The disconnect and reconnect would be fine if the resume worked properly but it does appear that it does not.
Any ideas please?

Anonymous wrote:

I have some more information on this but it does not answer why the resume does not happen properly. It appears that the connection is interrupted when 1GB in total has been transferred per session. When 1 GB is reached, the transfer is interrupted, then WinSCP reconnects and starts the interrupted file again copying down into the existing partfile but appends the entire file to the existing partfile as previously described.
The disconnect and reconnect would be fine if the resume worked properly but it does appear that it does not.
Any ideas please?

Please email me a full log file showing the problem.

> Hi Martin
>
> Ref topic
> https://winscp.net/forum/viewtopic.php?t=2406
>
> It does appear I have 2 problems still -
> 1) Cannot force the script to Abort. This one is set to Abort and also
> reconnecttime off. I tried various options with no luck with any of them.
> 2) Resume logic does not seem to be working properly.
>
> here is a log file showing the problem and also screen print of the file
> after the data has been finished. I highlighted the first field of the
> second attempt. As you can see it is appended to the failed part.
>
> I hope this helps in finding a solution.
> Thanks for all your work.

Email has the attachments.

And just in case anyone else has same problem. -

To get round this problem, the solution I am using now is to use the ini file pre-copied into the same folder as winscp.exe and winscp.com with ResumeReport=2 already set to prevent the partfiles being created. Therefore the connection is still resumed but the file is started again cleanly. It works for us so far in testing. :)