Remember already downloaded files so they are not downloaded again

You might have a server, where some kind of data files are periodically generated. You need to regularly download a new set of files to the local machine, where the files are processed. If you keep local copies of the files, you can use a simple synchronization. But if you cannot keep the local copies, you will have to remember, what files did you download already. And this is, what this script does. An alternative is to remember a timestamp of the last transferred file.

You can install this script as an WinSCP extension by using this page URL in the Add Extension command. The extension will automatically use the current local and remote directories.

To run the script manually or to schedule it, use:

powershell.exe -File C:\path\DownloadNewFiles.ps1 -sessionUrl "sftp://username:password;fingerprint=ssh-rsa-xxxxxxxxxxx...@example.com/" -localPath "C:\local\dest" -remotePath "/remote/source" -listPath "C:\path\downloaded.txt"

# @name         Download &New Files...
# @command      powershell.exe -ExecutionPolicy Bypass -File "%EXTENSION_PATH%" ^
#                   -sessionUrl "!E" -localPath "%LocalPath%" -remotePath "%RemotePath%" ^
#                   -listPath "%ListPath%" -pause -sessionLogPath "%SessionLogPath%"
# @description  Downloads new files and appends their list to a file, ^
#                   so they are not downloaded again the next time
# @version      1
# @homepage     https://winscp.net/eng/docs/library_example_remember_downloaded_files
# @require      WinSCP 5.14
# @option       - -run group "Directories"
# @option         RemotePath -run textbox "Download new files from &remote directory:" "!/"
# @option         LocalPath -run textbox "... to &local directory:" "!\"
# @option       - -run group "Options"
# @option         ListPath -run textbox "Re&member already downloaded files in:" ^
#                     "%LOCALAPPDATA%\WinSCP\DownloadNewFiles\!N.txt"
# @option       - -config group "Logging"
# @option         SessionLogPath -config sessionlogfile
# @optionspage  https://winscp.net/eng/docs/library_example_remember_downloaded_files#options
 
param (
    # Use Generate Session URL function to obtain a value for -sessionUrl parameter.
    $sessionUrl = "sftp://user:mypassword;fingerprint=ssh-rsa-xxxxxxxxxxx...@example.com/",
    [Parameter(Mandatory = $True)]
    $localPath,
    [Parameter(Mandatory = $True)]
    $remotePath,
    [Parameter(Mandatory = $True)]
    $listPath,
    $sessionLogPath = $Null,
    [Switch]
    $pause
)
 
try
{
    # Load WinSCP .NET assembly
    $assemblyPath = if ($env:WINSCP_PATH) { $env:WINSCP_PATH } else { $PSScriptRoot }
    Add-Type -Path (Join-Path $assemblyPath "WinSCPnet.dll")
 
    # Setup session options from URL
    $sessionOptions = New-Object WinSCP.SessionOptions
    $sessionOptions.ParseUrl($sessionUrl)
 
    $session = New-Object WinSCP.Session
    
    try
    {
        $session.SessionLogPath = $sessionLogPath
 
        $listPath = [Environment]::ExpandEnvironmentVariables($listPath)
        $listDir = (Split-Path -Parent $listPath) 
        New-Item -ItemType directory -Path $listDir -Force | Out-Null 
 
        if (Test-Path $listPath)
        {
            Write-Host "Loading list of already downloaded files from $listPath..."
            $downloaded = @(Get-Content $listPath)
        }
        else
        {
            Write-Host "File $listPath with list of already downloaded files doesn't exist yet."
            $downloaded = @()
        }
 
        Write-Host "Connecting..."
        $session.Open($sessionOptions)
 
        Write-Host "Looking for new files..."
        $files =
            $session.EnumerateRemoteFiles(
                $remotePath, "*", [WinSCP.EnumerationOptions]::AllDirectories)
 
        $count = 0
        foreach ($fileInfo in $files)
        {
            $remoteFilePath = $fileInfo.FullName
            if ($downloaded -notcontains $remoteFilePath)
            {
                $remoteFileLen = $fileInfo.Length
                Write-Host `
                    "Found new file $remoteFilePath with size $remoteFileLen, downloading..."
 
                $localFilePath =
                    [WinSCP.RemotePath]::TranslateRemotePathToLocal(
                        $remoteFilePath, $remotePath, $localPath)
                $localFileDir = (Split-Path -Parent $localFilePath) 
                New-Item -ItemType directory -Path $localFileDir -Force | Out-Null 
 
                $source = [WinSCP.RemotePath]::EscapeFileMask($remoteFilePath)
                $session.GetFiles($source, $localFilePath).Check()
 
                Add-Content $listPath $remoteFilePath
 
                $count++
 
                Write-Host "Downloaded."
            }
        }
 
        if ($count -gt 0)
        {
            Write-Host "Done, downloaded $count files."
        }
        else
        {
            Write-Host "Done, no new files found."
        }
    }
    finally
    {
        # Disconnect, clean up
        $session.Dispose()
    }
 
    $result = 0
}
catch
{
    Write-Host "Error: $($_.Exception.Message)"
    $result = 1
}
 
# Pause if -pause switch was used
if ($pause)
{
    Write-Host "Press any key to exit..."
    [System.Console]::ReadKey() | Out-Null
}
 
exit $result