FTPS transfers very slow for many small files :: Support Forum

Sharken
Joined:: 2013-07-15
Posts:: 11

FTPS transfers very slow for many small files

2013-08-08 13:08

Hi,
I am transferring a lot of small files (20.000) using Implicit FTPS on port 990.
Transfer is extremely slow, there is a visible delay between each individual file transfer.

If i compare with FileZilla the difference is massive. Apparently WinSCP is doing something that
is hurting performance when copying lots of small files.

Hopefully this behavior can be improved, as it is unusable for a large amount of small files as it is.
This was tested on version 5.2.2 (Build 3365).

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,432
Location:: Prague, Czechia

Re: FTPS transfers very slow for many small files

2013-08-11

I believe you are a victim of this bug:
https://winscp.net/tracker/1021

It is fixed in 5.1.6 and will be fixed in 5.2.3 beta.

Reply with quote

Sharken

2013-08-12 08:11

Great news, will be testing 5.2.3 beta then

Reply with quote

Sharken
Joined:: 2013-07-15
Posts:: 11

2013-08-12 13:25

Just finished testing 5.2.3 beta and it hasn't improved the situation.

It seems that even the WinSCP GUI is slow when copying many small files. No more than 3-4 files
is copied each second, which is way too slow when you have 20k+ files to copy.

I cant really give any further info, there seems to be some kind of overhead for each file, that
prevents the filecopy processs from running fast :/

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,432
Location:: Prague, Czechia

2013-08-14

Please attach a full log file showing the problem (using the latest version of WinSCP).

To generate log file, enable logging, log in to your server and do the operation and only the operation that causes the error. Submit the log with your post as an attachment. Note that passwords and passphrases not stored in the log. You may want to remove other data you consider sensitive though, such as host names, IP addresses, account names or file names (unless they are relevant to the problem). If you do not want to post the log publicly, you may email it to me. You will find my address (if you log in) in my forum profile. Please include link back to this topic in your email. Also note in this topic that you have emailed the log.

Reply with quote

Sharken

2013-08-14 11:26

Email with logfile have been sent to you now, i hope that it will help you locate the issue.

Reply with quote

Sharken

2013-08-27 12:45

Tracefile with additional debugging info sent.

Reply with quote

tomb
Joined:: 2013-07-05
Posts:: 19

2013-09-03 17:00

I was the first victim of this bug 1021

It was indeed solved in 5.1.6. With this bug solved I transferred 2800 small files in 17 min: about 3 files/sec. This is fast enough for me. With the bug it took several hours.

It seems that WINSCP has quite a lot of overhead for transferring one file. I hope it is identified as a new bug and that the overhead can be reduced.

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,432
Location:: Prague, Czechia

2013-09-17

Sharken wrote:

Just finished testing 5.2.3 beta and it hasn't improved the situation.

It seems that even the WinSCP GUI is slow when copying many small files. No more than 3-4 files
is copied each second, which is way too slow when you have 20k+ files to copy.

I cant really give any further info, there seems to be some kind of overhead for each file, that
prevents the filecopy processs from running fast :/

This bug has been added to the tracker:
https://winscp.net/tracker/1058

Reply with quote

Sharken

2013-09-17 14:39

Can confirm the bug has been fixed, very pleased with how the bug was handled and ultimately fixed.

Thanks !

Reply with quote

tomb

2013-09-18 15:43

So it seems the bug 1058 is solved in 5.1.8. I have the same performance issue in FTP and SFTP with a transfer speed of at most 3-4 files/sec. I wonder if this bug and fix is specifically for FTPS. Does anyone know this? I am waiting for the release of 5.1.8 to test it myself

Reply with quote

Sharken

2013-09-19 07:29

According to https://winscp.net/eng/docs/history_old#5.2.4 the fix will be included in 5.1.8/5.2.5 beta. Note that neither of these has been released yet.

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,432
Location:: Prague, Czechia

2013-09-19

tomb wrote:

So it seems the bug 1058 is solved in 5.1.8. I have the same performance issue in FTP and SFTP with a transfer speed of at most 3-4 files/sec. I wonder if this bug and fix is specifically for FTPS. Does anyone know this? I am waiting for the release of 5.1.8 to test it myself

Please start a new topic and attach a complete session log file showing your problem. Separately for FTP and SFTP. Thanks.

Reply with quote

tomb

2013-09-20 11:53

I just did a short test with SFTP and FTP and found an SFTP speed of 20-25 files/sec for tiny files. That is much faster than FTP and fully adequate for me.

I will start a new topic for the slow FTP.

Reply with quote

FTPS transfers very slow for many small files

FTPS transfers very slow for many small files

Re: FTPS transfers very slow for many small files

Documentation

Support

Associations

Follow Us