How do I transfer new/modified files only?

Advertisement

When destination directory contains old files

When your destination directory already contains all the previously transferred files, synchronize source directory to destination directory, instead of simple transfer.

Basically use synchronize local instead of get and synchronize remote instead of put.

With WinSCP .NET assembly that means, use Session.SynchronizeDirectories, with argument mode set to SynchronizationMode.Remote or SynchronizationMode.Local instead of Session.GetFiles or Session.PutFiles respectively.

When destination directory does not contain old files

When the old files are removed from the destination directory, you can instead select the new files based on timestamp.

Relative time constraint

For that you can use file mask with time constraint. For example, to transfer only files created/modified since yesterday, use mask *>=1D (means all files modified in the last 24 hours).

In scripting, apply the mask using -filemask=<mask> switch of get or put commands.

For example:

put -filemask="*>=1D" *

In WinSCP .NET assembly, use TransferOptions.FileMask. PowerShell example:

$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.FileMask = "*>=1D"
 
$session.PutFiles("d:\toupload\*", "/home/user/", $False, $transferOptions)

Advertisement

Absolute time constraint

If you need to select file specifically created/modified today (not simply in the last 24 hours), you can use today keyword (or other time specifications).

In scripting:

put -filemask="*>=today" *

In WinSCP .NET assembly:

$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.FileMask = "*>=today"
 
$session.PutFiles("d:\toupload\*", "/home/user/", $False, $transferOptions)

Remembering the last timestamp

In some cases you cannot rely on some artificially determined timestamp. E.g. because the files are being added continuously and your code needs to continuously (or very frequently) fetch them. In such case, you can remember the last run time, or even better the timestamp of the latest file processed in the last run, and use that timestamp as the threshold for the consecutive run.

The following snippet shows, how to implement a continuously running PowerShell script that fetches new files:

$sourcePath = "/remote/path/*"
$destPath = "C:\local\path\"
$lastTimestamp = $Null
 
while ($True)
{
    $transferOptions = New-Object WinSCP.TransferOptions
 
    if ($lastTimestamp -ne $Null) 
    {
        Write-Host "Downloading files modified after $lastTimestamp..."
        $transferOptions.FileMask = ("*>" + $lastTimestamp.ToString("yyyy-MM-dd HH:mm:ss"))
    }
    else
    {
        Write-Host "Downloading all files..."
    }
 
    $transferResult = $session.GetFiles($sourcePath, $destPath, $False, $transferOptions)
    $transferResult.Check()
 
    # Find the latest downloaded file
    $latestTransfer =
        $transferResult.Transfers |
        Sort-Object -Property @{ Expression = { (Get-Item $_.Destination).LastWriteTime } } `
                    -Descending |
        Select-Object -First 1
 
    if ($latestTransfer -eq $Null)
    {
        Write-Host "No files found."
    }
    else
    {
        $lastTimestamp = (Get-Item $latestTransfer.Destination).LastWriteTime
        Write-Host (
            "Downloaded $($transferResult.Transfers.Count) files, " +
            "latest being $($latestTransfer.FileName) with timestamp $lastTimestamp.")
    }
 
    Write-Host "Waiting..."
    Start-Sleep -Seconds 5
}

Advertisement

Though, your code may run in separate scheduled runs, rather than continuously. Then you have to persist the last timestamp between the runs, for example like this:

# Read last timestamp of previous run, if any
$timestampFile = "timestamp.dat"
if (Test-Path $timestampFile)
{
    $lastTimestamp = [DateTime]::ParseExact((Get-Content $timestampFile), 'O', $Null)
}
else
{
    $lastTimestamp = $Null
}
 
# The inner code from loop of the above code snippet
...
 
# Persist timestamp for the next run
Set-Content -Path $timestampFile -Value $lastTimestamp.ToString("O")

Remembering already transferred files

See example Remember already downloaded files so they are not downloaded again.

Last modified: by martin