The next version will be able to use up to 4 GB of memory:
Issue 2412 – Increase WinSCP memory limit to 4 GB
Still working on 64-bit support, which will allow use of all available memory.

Thanks for your report.
This should be resolved with upgrade to 64-bit:
Issue 618 – 64-bit version of WinSCP

When using the WinSCP 6.5.3 synchronize function on a folder with too many files, the program hangs, then runs out of memory. This happens consistently on this folder

Error message:
Out of memory

Status: not_matched

I know how to reproduce the problem or the problem happens frequently enough. I wish to be contacted by the WinSCP team to help resolving the problem.