Large numbers of directories

Hi,

I've been using WinSCP for about a year and it is GREAT! Thanks for your tremendous contribution.

I am using WinSCP (v. 3.8.2) to keep a backup of my website (on linux) on my home PC (Windows XP) . I have a large number of directories and more than 30,000 files in them, but only a handful change in a given week, and there are perhaps a hundred additions (new files) in a given day.

Even on my 15Mb fiber optic connection where I get a reliable 8Mbit throughput to my server with pretty fast pings, WinSCP is starting to take a long time to investigate all of those directories.

Would it be possible for WinSCP to do a single recursive look into directories on the remote side and process all of that on the local windows side? Seems to me it would be much faster than the back and forth with all of the latency issues.

Perhaps inclusion of an rsync option would help?

I am hoping that something along these lines can be done, because the amount of time to examine the directories is starting to become too much :( :( :( , and I love my WinSCP!!! :mrgreen:

Actually, the number of files is bigger than I rememebred:

Size: 7.80 GB (8,384,999,253 bytes)
Contains: 44,359 Files, 692 Folders

I have registered on the forum. Thanks.

Ron wrote:

Would it be possible for WinSCP to do a single recursive look into directories on the remote side and process all of that on the local windows side? Seems to me it would be much faster than the back and forth with all of the latency issues.

What do you mean by "single recursive look"? :-) What protocol are you using SCP or SFTP? Do you use synchronization feature of WinSCP?

martin wrote:

What do you mean by "single recursive look"? :-) What protocol are you using SCP or SFTP? Do you use synchronization feature of WinSCP?

Yes, I'm using the sync feature and SCP to keep my PC up to date as a backup of my UNIX website.

What I am suggesting is if you could add the -R to your ls command(s) on the UNIX side and then bring all of that data back, all at once, and process it on the PC side. The examination of the directories is taking more than 6 minutes because of the number of directories and the back-and-forth latency of looking at a directory, comparing it to the PC, look at the next directory, compare it to the PC, etc., etc.

If you asked UNIX to do a recursive look bringing all of the directory information back to the PC in one big grab, then examine all of the PC directories, I would guess this would be a much faster approach.

Thanks for considering it.

Sure, it would may make it faster.

However synchronization function in WinSCP is implemented in protocol-independent way. Your suggestion is SCP-specific. Also it would list even directories excluded from synchronization or those not present on local side. This in turn may cause the synchronization to be even slower in some cases.

However I'll consider some improvements based on your suggestions.