Forum » .NET assembly / COM library »

Post a reply

Username

Subject

Message body

Font colour:

Font size:

[quote="lintho"]Hello!

After some great help from you Martin last year I managed to get my scripts working in pipelies in devops (Azure).

However, the downloading\uploading part is taking a loong time when im trying to deploy new versions of my websites.

This is my current script – which is downloading files from a folder on production web app, to a staging slot before swapping the slots.
[code]@powershell@
$ftp = '$(hostname)'
$user = '$(produser)'
$pass = '$(prodpass)'
$folder = 'site/wwwroot/wwwroot/images/uploaded/*'
$sitenameFolder =  '$(sitename)'
$target = "C:\"+ $sitenameFolder+"\*"
Write-Host $target

# Load WinSCP .NET assembly
Add-Type -Path  "$(System.DefaultWorkingDirectory)/drop-yaml/drop-yaml/ftp/WinSCPnet.dll"

# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Ftp
    HostName = $ftp
    UserName = $user
    Password = $pass 
}

$session = New-Object WinSCP.Session

try
{
    # Connect
    $session.Open($sessionOptions)

# Download files

$transferResult = $session.GetFiles($folder, $target)
    $transferResult.Check()
 
    # Print results
    foreach ($transfer in $transferResult.Transfers)
    {
        Write-Host "Download of $transfer succeeded"
    }
}
finally
{
    # Disconnect, clean up
    $session.Dispose()
}    
[/code]
and for upload:
[code]@powershell@
$ftp = '$(hostname)'
$user = '$(destinationuser)'
$pass = '$(desinationpass)'
$folder = 'site/wwwroot/wwwroot/images/uploaded/*'

$sitenameFolder =  '$(sitename)'
$fromFolder= "C:\"+ $sitenameFolder+"\*"

Write-Host $fromFolder

Add-Type -Path "$(System.DefaultWorkingDirectory)/drop-yaml/drop-yaml/ftp/WinSCPnet.dll"

$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Ftp
    HostName = $ftp
    UserName = $user
    Password = $pass 
}

$session = New-Object WinSCP.Session

try
{
    # Connect
    $session.Open($sessionOptions)
    $remotePath = $fromFolder
    $transferResult = $session.PutFiles(($fromFolder + "*.*"), ($folder + "*.*"), $False)

foreach ($transfer in $transferResult.Transfers)
    {
        Write-Host "Uploaded of $transfer succeeded"
    }
}
finally
{
    # Disconnect, clean up
    $session.Dispose()
}
[/code]
Is there any way to optimize this?[/quote]

Options

Disable BBCode in this post

Notify me when a reply is posted (registered users only)

Add an Attachment

If you do not want to add an Attachment to your Post, please leave the Fields blank.

Filename

(maximum 10 MB; please compress large files; only common media, archive, text and programming file formats are allowed)

File Comment

Options

Private file (available to author and site moderators only)

Topic review

martin

2021-03-16

lintho wrote:

So, instead of downloading all files, is it possible\faster to check filesum on files or something so I don't have to download all files?

Theoretically yes, but most FTP servers do not support this. If your FTP server don't, you might build a service that provide the checksums and use it on your PowerShell script to choose the files to update. But in that case you might have the service automatically pull the updated files from the source, and you won't need the local intermediate PowerShell/WinSCP script.

martin

2021-03-16

lintho wrote:

But, on the website it its possible to upload files directly to a folder on the site. When i am deploying new versions of my website using CD/CI, i am using 'slots' in azure, so my staging slot is replacing the production website. If i do that without transfering files using winSCP, all files uploaded to the website ends up on the staging slot, and is NOT on production anymore. Therefor i am downloading all files from prod, and uploading them to staging slot, before i replace staging with prod.

I do not know anything about this, so I cannot advise.

lintho

2021-03-16 11:09

So, instead of downloading all files, is it possible\faster to check filesum on files or something so I don't have to download all files?

lintho

2021-03-16 11:06

Hello.
Thanks for a quick reply! I will check it out.

I thought this was clear:

downloading files from a folder on production web app, to a staging slot before swapping the slots.

But, on the website it its possible to upload files directly to a folder on the site. When i am deploying new versions of my website using CD/CI, i am using 'slots' in azure, so my staging slot is replacing the production website. If i do that without transferring files using WinSCP, all files uploaded to the website ends up on the staging slot, and is NOT on production anymore. Therefor i am downloading all files from prod, and uploading them to staging slot, before i replace staging with prod.

martin

Re: Download only 'new' or 'modified' files

2021-03-16

This is quite broad question. And you didn't really explain what are you trying to achieve.
So I can only vaguely suggest that you may parallelize the transfer, if you need it to be completed faster.
We have an example here:
https://winscp.net/eng/docs/library_example_parallel_transfers#powershell

lintho

Download only 'new' or 'modified' files

2021-03-16 09:41

Hello!

After some great help from you Martin last year I managed to get my scripts working in pipelies in devops (Azure).

However, the downloading\uploading part is taking a loong time when im trying to deploy new versions of my websites.

This is my current script – which is downloading files from a folder on production web app, to a staging slot before swapping the slots.

$ftp = '$(hostname)'
$user = '$(produser)'
$pass = '$(prodpass)'
$folder = 'site/wwwroot/wwwroot/images/uploaded/*'
$sitenameFolder =  '$(sitename)'
$target = "C:\"+ $sitenameFolder+"\*"
Write-Host $target
 
# Load WinSCP .NET assembly
Add-Type -Path  "$(System.DefaultWorkingDirectory)/drop-yaml/drop-yaml/ftp/WinSCPnet.dll"
 
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Ftp
    HostName = $ftp
    UserName = $user
    Password = $pass 
}
 
$session = New-Object WinSCP.Session
 
try
{
    # Connect
    $session.Open($sessionOptions)
 
    # Download files
 
    $transferResult = $session.GetFiles($folder, $target)
    $transferResult.Check()
 
    # Print results
    foreach ($transfer in $transferResult.Transfers)
    {
        Write-Host "Download of $transfer succeeded"
    }
}
finally
{
    # Disconnect, clean up
    $session.Dispose()
}

and for upload:

$ftp = '$(hostname)'
$user = '$(destinationuser)'
$pass = '$(desinationpass)'
$folder = 'site/wwwroot/wwwroot/images/uploaded/*'
 
$sitenameFolder =  '$(sitename)'
$fromFolder= "C:\"+ $sitenameFolder+"\*"
 
Write-Host $fromFolder
 
Add-Type -Path "$(System.DefaultWorkingDirectory)/drop-yaml/drop-yaml/ftp/WinSCPnet.dll"
 
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Ftp
    HostName = $ftp
    UserName = $user
    Password = $pass 
}
 
$session = New-Object WinSCP.Session
 
try
{
    # Connect
    $session.Open($sessionOptions)
    $remotePath = $fromFolder
    $transferResult = $session.PutFiles(($fromFolder + "*.*"), ($folder + "*.*"), $False)
 
    foreach ($transfer in $transferResult.Transfers)
    {
        Write-Host "Uploaded of $transfer succeeded"
    }
}
finally
{
    # Disconnect, clean up
    $session.Dispose()
}

Is there any way to optimize this?

Post a reply

Topic review

Re: Download only 'new' or 'modified' files

Download only 'new' or 'modified' files

Documentation

Support

Associations

Follow Us