Thanks for your research. I'll consider it.

Thanks ! I understand that the problem is mainly because the FTP server's TLS parameters aren't secure enough.

But for your information, FileZilla has no problem connecting this server.
I was curious and with Wireshark, I found that the TLS handshake uses cipher TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA with FileZilla, whereas it uses TLS_DHE_RSA_WITH_AES_256_CBC_SHA with WinSCP.

And indeed, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA needs a smaller DH key and seems a bit better than TLS_DHE_RSA_WITH_AES_256_CBC_SHA. For example in Microsoft documentation about TLS ciphers in Windows (cf. https://learn.microsoft.com/en-us/windows/win32/secauthn/tls-cipher-suites-in-windows-11-v22h2), where TLS_DHE_RSA_WITH_AES_256_CBC_SHA is now even disabled by default.

I can see that you are using OpenSSL DEFAULT ordered ciphers list (where TLS_DHE_RSA_WITH_AES_256_CBC_SHA is before TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA), whereas FileZilla's supported ciphers list explicitely gives priority to TLSv1.3, then ECDHE, then the rest.
I don't know if you care to have the same level of compatibility for old servers than FileZilla, without changing the minimum TLS version in WinSCP. If so, maybe you can call OpenSSL with a param cipher TLSv1.2:ECDHE:ALL:!COMPLEMENTOFDEFAULT ? It will use the same list of DEFAULT ciphers, with a priority for TLSv1.3 ciphers (always) then TLSv1.2, and for older TLS versions, there is a priority on ECDHE ciphers.

Demo:

I assume it's because to enable TLS 1.1, WinSCP needs to lower OpenSSL security level. What has an additional side effect of enabling insecurely low key sizes.
https://docs.openssl.org/master/man3/SSL_CTX_set_security_level/#default-callback-behaviour
I have documented this:
https://winscp.net/eng/docs/ui_login_tls#fn1

Hello,
I have the following error with WinSCP 6.5.1 when connecting to a public Explicit FTPS server (for connexion params : cf. https://dlptest.com/ftp-test/) :

SSL3 alert write: fatal: handshake failure
OpenSSL 0A00018A: dh key too small
Can't establish TLS connection.
Disconnected from server
Connection failed.

Interestingly, I finally managed to make it work with WinSCP GUI by setting minimum TLS version 1.1.
But I think there is a problem in WinSCP, because the server doesn't support TLSv1.1 (indeed, Wireshark says TLSv1.2 is used).