Seems like I found WinSCP right on time. Some notes:

1. To new users – You need to select the folders first, then right-click the Size column header, then select Calculate Directory Sizes. If you don't select folders, this function is disabled.
   
2. The calculated size will NOT be saved once you move to a different folder. So if your folders are huge and you spent 10 mins waiting for the calculation, then navigated to a different folder and came back to the first folder, sorry mate, you gotta do it again.
   

To Devs: Not being ungrateful though, this still beats every other workaround I know of. May I suggest for a future upgrade, if you could save the file size into some kinda cache and update it only if the Changed timestamp has changed, or something. Or display a (!) icon to indicate the folder size is outdated, so we can decide whether or not to recalculate or leave it be. Thanks for developing an awesome software and making it free for all to use.

Directory sizes in file panel are available in the latest WinSCP 6.0 beta only:
https://winscp.net/eng/docs/ui_file_panel#directory_sizes
Issue 41 – Show directory size in file panel

I have had a good look through and cannot see a simple solution to this.

I can see a folder list, I cannot see a size. The only setting is to see file size and not folder size. I have to open the properties box to find a folder size, but then I have to keep repeating this for many folders and it is both time consuming and not user friendly.

Is this a feature to come, a bug in the current or my lack of understanding the UI?