How do I transfer new/modified files only?
Advertisement
When destination directory contains old files
When your destination directory already contains all the previously transferred files, synchronize source directory to destination directory, instead of simple transfer.
Basically use synchronize local
instead of get
and synchronize remote
instead of put
.
With WinSCP .NET assembly that means, use Session.SynchronizeDirectories
, with argument mode
set to SynchronizationMode.Remote
or SynchronizationMode.Local
instead of Session.GetFiles
or Session.PutFiles
respectively.
When destination directory does not contain old files
When the old files are removed from the destination directory, you can instead select the new files based on timestamp.
Relative time constraint
For that you can use file mask with time constraint. For example, to transfer only files created/modified since yesterday, use mask *>=1D
(means all files modified in the last 24 hours).
In scripting, apply the mask using -filemask=<mask>
switch of get
or put
commands.
For example:
put -filemask="*>=1D" *
In WinSCP .NET assembly, use TransferOptions.FileMask
. PowerShell example:
$transferOptions = New-Object WinSCP.TransferOptions $transferOptions.FileMask = "*>=1D" $session.PutFiles("d:\toupload\*", "/home/user/", $False, $transferOptions)
Advertisement
Absolute time constraint
If you need to select file specifically created/modified today (not simply in the last 24 hours), you can use today
keyword (or other time specifications).
In scripting:
put -filemask="*>=today" *
In WinSCP .NET assembly:
$transferOptions = New-Object WinSCP.TransferOptions $transferOptions.FileMask = "*>=today" $session.PutFiles("d:\toupload\*", "/home/user/", $False, $transferOptions)
Remembering the last timestamp
In some cases you cannot rely on some artificially determined timestamp. E.g. because the files are being added continuously and your code needs to continuously (or very frequently) fetch them. In such case, you can remember the last run time, or even better the timestamp of the latest file processed in the last run, and use that timestamp as the threshold for the consecutive run.
The following snippet shows, how to implement a continuously running PowerShell script that fetches new files:
$sourcePath = "/remote/path/*" $destPath = "C:\local\path\" $lastTimestamp = $Null while ($True) { $transferOptions = New-Object WinSCP.TransferOptions if ($lastTimestamp -ne $Null) { Write-Host "Downloading files modified after $lastTimestamp..." $transferOptions.FileMask = ("*>" + $lastTimestamp.ToString("yyyy-MM-dd HH:mm:ss")) } else { Write-Host "Downloading all files..." } $transferResult = $session.GetFiles($sourcePath, $destPath, $False, $transferOptions) $transferResult.Check() # Find the latest downloaded file $latestTransfer = $transferResult.Transfers | Sort-Object -Property @{ Expression = { (Get-Item $_.Destination).LastWriteTime } } ` -Descending | Select-Object -First 1 if ($latestTransfer -eq $Null) { Write-Host "No files found." } else { $lastTimestamp = (Get-Item $latestTransfer.Destination).LastWriteTime Write-Host ( "Downloaded $($transferResult.Transfers.Count) files, " + "latest being $($latestTransfer.FileName) with timestamp $lastTimestamp.") } Write-Host "Waiting..." Start-Sleep -Seconds 5 }
Advertisement
Though, your code may run in separate scheduled runs, rather than continuously. Then you have to persist the last timestamp between the runs, for example like this:
# Read last timestamp of previous run, if any $timestampFile = "timestamp.dat" if (Test-Path $timestampFile) { $lastTimestamp = [DateTime]::ParseExact((Get-Content $timestampFile), 'O', $Null) } else { $lastTimestamp = $Null } # The inner code from loop of the above code snippet ... # Persist timestamp for the next run Set-Content -Path $timestampFile -Value $lastTimestamp.ToString("O")
Remembering already transferred files
See example Remember already downloaded files so they are not downloaded again.