Interpreting XML Log for Advanced Scripting

To implement such task, you have two options:

• Have several scripts, each implementing distinct steps of overall operation. One or more of these scripts will be generated based outcome of previous script, where the outcome may be determined from XML log generated by previous script. This approach is easier to implement, though it may become inefficient, if there are multiple steps, as each steps require (re)opening session. Most examples in this article use this approach.
• Drive WinSCP from another application, feeding the commands using redirected standard input and continuously reading the XML log to determine the commands outcome. This approach require more advanced programming techniques. See chapter Continuous reading of the XML log file.

Before starting you should:

• Have WinSCP installed;
• Know how to connect to your FTP/SFTP account;
• Know what WinSCP scripting commands to use for your task (e.g. file transfer);
• Understand how WinSCP can be used from .NET application.

You will need to parse directory listing, when you need to:

• check for file existence;
• perform operation for every file in a directory or for subset of the files in a directory.

Supposing the previous script had used ls command to list contents of some directory, XML log file will look like:

<?xml version="1.0" encoding="UTF-8"?>
<session xmlns="http://winscp.net/schema/session/1.0"
         name="martin@example.com" start="2011-03-03T16:21:06.544Z">
  <ls>
    <destination value="/home/martin/public_html" />
    <files>
      <file>
        <filename value="." />
        <type value="d" />
        <modification value="2008-12-22T12:16:23.000Z" />
        <permissions value="rwxr-xr-x" />
      </file>
      <file>
        <filename value=".." />
        <type value="d" />
        <modification value="2008-03-25T08:15:53.000Z" />
        <permissions value="rwxr-xr-x" />
      </file>
      <file>
        <filename value=".htaccess" />
        <type value="-" />
        <size value="107" />
        <modification value="2008-12-02T06:59:58.000Z" />
        <permissions value="rw-r--r--" />
      </file>
      <file>
        <filename value="about.html" />
        <type value="-" />
        <size value="24064" />
        <modification value="2007-10-04T21:43:02.000Z" />
        <permissions value="rw-r--r--" />
      </file>
      <!-- more files -->
    </files>
    <result success="true" />
  </ls>
</session>

Following example C# code generates script (on standard output) that moves all files from the directory listing above, modified the last time in 2007, to a different folder:

using System;
using System.Diagnostics;
using System.Xml;
using System.Xml.XPath;
 
...
 
const string targetDirectory = "/home/martin/backup/";
 
XPathDocument log = new XPathDocument(logPath);
XmlNamespaceManager ns = new XmlNamespaceManager(new NameTable());
ns.AddNamespace("w", "http://winscp.net/schema/session/1.0");
XPathNavigator nav = log.CreateNavigator();
 
string sourceDirectory = nav.SelectSingleNode("//w:ls/w:destination/@value", ns).Value;
if (!sourceDirectory.EndsWith("/"))
{
    sourceDirectory += "/";
}
XPathNodeIterator files = nav.Select("//w:file", ns);
foreach (XPathNavigator file in files)
{
    string modification = file.SelectSingleNode("w:modification/@value", ns).Value;
    string filename = file.SelectSingleNode("w:filename/@value", ns).Value;
    if (modification.StartsWith("2007") && (filename != ".") && (filename != ".."))
    {
        string moveCommand =
            string.Format("mv \"{0}{1}\" \"{2}\"", sourceDirectory, filename, targetDirectory);
        Console.WriteLine(moveCommand);
    }
}

Output of the above console application will be like:

mv "/home/martin/public_html/.htaccess" "/home/martin/backup/"
mv "/home/martin/public_html/about.html" "/home/martin/backup/"

To utilize such console application (giving it name parselisting.exe) you can use following command.

parselisting.exe | winscp.com /command "open sftp://user:password@example.com/"

WinSCP first runs the commands specified on command-line, then it runs the command fed by parselisting.exe.

Alternatively, you can use temporary script file like:

echo open sftp://user:password@example.com/ >> script.txt
parselisting.exe >> script.txt
winscp.com /script=script.txt

If you need to run WinSCP from continuously running application, like GUI application, you cannot do with generating WinSCP script using output redirection. Instead you can run WinSCP directly from your application and replace the Console.WriteLine call with writing to WinSCP input stream. See guide for SFTP transfers in .NET for details and following example for complete implementation.

using System;
using System.Diagnostics;
using System.Xml;
using System.Xml.XPath;
 
...
 
const string logPath = "log.xml";
const string targetDirectory = "/home/martin/backup/";
 
// Run hidden WinSCP process
Process winscp = new Process();
winscp.StartInfo.FileName = "WinSCP.com";
winscp.StartInfo.Arguments = "/xmllog=" + logPath;
winscp.StartInfo.UseShellExecute = false;
winscp.StartInfo.RedirectStandardInput = true;
winscp.StartInfo.RedirectStandardOutput = false;
winscp.StartInfo.CreateNoWindow = true;
winscp.Start();
 
// Feed in the scripting commands
winscp.StandardInput.WriteLine("option batch abort");
winscp.StandardInput.WriteLine("option confirm off");
winscp.StandardInput.WriteLine("open mysession");
winscp.StandardInput.WriteLine("ls");
winscp.StandardInput.Close();
 
// Wait until WinSCP finishes
winscp.WaitForExit();
 
// Parse and interpret the XML log
// (Note that in case of fatal failure the log file may not exist at all)
XPathDocument log = new XPathDocument(logPath);
XmlNamespaceManager ns = new XmlNamespaceManager(new NameTable());
ns.AddNamespace("w", "http://winscp.net/schema/session/1.0");
XPathNavigator nav = log.CreateNavigator();
 
// Avoid overwritting log file by second run
winscp.StartInfo.Arguments = "";
 
// Re-run WinSCP process
winscp.Start();
 
// Feed in the start-up scripting commands
winscp.StandardInput.WriteLine("option batch abort");
winscp.StandardInput.WriteLine("option confirm off");
winscp.StandardInput.WriteLine("open mysession");
 
// Lookup source directory of the listing
string sourceDirectory = nav.SelectSingleNode("//w:ls/w:destination/@value", ns).Value;
if (!sourceDirectory.EndsWith("/"))
{
    sourceDirectory += "/";
}
// For every file in the directory listing with modification time in 2007,
// feed in move command to second instance of WinSCP
XPathNodeIterator files = nav.Select("//w:file", ns);
foreach (XPathNavigator file in files)
{
    string modification = file.SelectSingleNode("w:modification/@value", ns).Value;
    string filename = file.SelectSingleNode("w:filename/@value", ns).Value;
    if (modification.StartsWith("2007") && (filename != ".") && (filename != ".."))
    {
        string moveCommand =
            string.Format("mv \"{0}{1}\" \"{2}\"", sourceDirectory, filename, targetDirectory);
        winscp.StandardInput.WriteLine(moveCommand);
    }
}
 
// Wait until WinSCP finishes
winscp.WaitForExit();

You will need to detect, what files were affected by operation, when you need to:

• log list of affected files;
• move away (back up) source files after successfully transferring them.

Supposing the previous script had used get command to download all files from certain remote directory, XML log file will look like:

<?xml version="1.0" encoding="UTF-8"?>
<session xmlns="http://winscp.net/schema/session/1.0"
         name="martin@example.com" start="2011-03-25T16:11:34.756Z">
  <download>
    <filename value="/home/martin/public_html/.htaccess" />
    <destination value="d:\www\.htaccess" />
    <result success="true" />
  </download>
  <download>
    <filename value="/home/martin/public_html/about.html" />
    <destination value="d:\www\about.html" />
    <result success="true" />
  </download>
  <download>
    <filename value="/home/martin/public_html/index.html" />
    <destination value="d:\www\index.html" />
    <result success="true" />
  </download>
</session>

Following example C# code generates script (on standard output) that moves all successfully transferred remote files, to a different folder:

using System;
using System.Diagnostics;
using System.Xml;
using System.Xml.XPath;
 
...
 
const string targetDirectory = "/home/martin/backup/";
 
XPathDocument log = new XPathDocument(logPath);
XmlNamespaceManager ns = new XmlNamespaceManager(new NameTable());
ns.AddNamespace("w", "http://winscp.net/schema/session/1.0");
XPathNavigator nav = log.CreateNavigator();
 
XPathNodeIterator files = nav.Select("//w:download[w:result/@success='true']", ns);
foreach (XPathNavigator file in files)
{
    string filename = file.SelectSingleNode("w:filename/@value", ns).Value;
    string moveCommand =
        string.Format("mv \"{0}\" \"{1}\"", filename, targetDirectory);
    Console.WriteLine(moveCommand);
}

For use of the generated script, see Parsing Directory Listing chapter.

If you want to avoid running new instance of WinSCP for every conditional step of your task you can:

• Run WinSCP from your application (e.g. .NET one);
• Redirect standard input, so you can feed in your commands;
• Continuously read XML log file to determine the commands outcome;
• Feed in more commands based in outcome of previous ones.

Continuous reading of XML file that is being written to is non-trivial task. You will need to repeatedly parse the file using parser that can handle incomplete (non-well-formed) XML, e.g. stream-based parser (such as XmlReader), until the outcome of your commands appear.

Following simple example is continuously reading the XML log file during lengthy download operation, printing list of files that get transferred.

using System;
using System.Diagnostics;
using System.IO;
using System.Threading;
using System.Xml;
 
...
 
const string logname = "log.xml";
 
// Make sure the log file does not exist, so we do not happen to start reading some old log
File.Delete(logname);
 
Process winscp = new Process();
winscp.StartInfo.FileName = "winscp.com";
winscp.StartInfo.Arguments = "/xmllog=\"" + logname + "\"";
winscp.StartInfo.UseShellExecute = false;
winscp.StartInfo.RedirectStandardInput = true;
winscp.StartInfo.CreateNoWindow = true;
winscp.Start();
 
winscp.StandardInput.WriteLine("option batch abort");
winscp.StandardInput.WriteLine("option confirm off");
winscp.StandardInput.WriteLine("open mysession");
winscp.StandardInput.WriteLine("get *");
winscp.StandardInput.Close();
 
// Wait until the log file gets created or WinSCP terminates (in case of fatal error)
do
{
    if (winscp.HasExited && !File.Exists(logname))
    {
        throw new Exception("WinSCP process terminated without creating a log file");
    }
} while (!File.Exists(logname));
 
const string ns = "http://winscp.net/schema/session/1.0";
 
int position = 0;
bool logClosed;
 
do
{
    FileStream stream;
    try
    {
        // First try to open file without write sharing.
        // This fails, if WinSCP is still writing to the log file.
        // This is done only as a way to detect that log file is not complete yet.
        stream = File.Open(logname, FileMode.Open, FileAccess.Read, FileShare.Read);
        logClosed = true;
    }
    catch (IOException)
    {
        // If log file is still being written to, open it with write sharing
        stream = File.Open(logname, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
        logClosed = false;
    }
 
    XmlReader reader = XmlReader.Create(stream);
 
    int skip = position;
    position = 0;
    bool inDownload = false;
 
    try
    {
        while (reader.Read())
        {
            // Add processing of nodes you actually need here
 
            // Entering <download/> element
            if ((reader.NodeType == XmlNodeType.Element) &&
                (reader.NamespaceURI == ns) &&
                (reader.LocalName == "download") &&
                !reader.IsEmptyElement)
            {
                inDownload = true;
            }
 
            // Leaving <download/> element
            if ((reader.NodeType == XmlNodeType.EndElement) &&
                (reader.NamespaceURI == ns) &&
                (reader.LocalName == "download"))
            {
                inDownload = false;
            }
            
            // Skip all nodes that we have read in previous iteration(s) already
            if (position >= skip)
            {
                // Got <filename/> element inside <download/>
                if (inDownload &&
                    (reader.NodeType == XmlNodeType.Element) &&
                    (reader.NamespaceURI == ns) &&
                    (reader.LocalName == "filename"))
                {
                    // Note that we should check the <result/> element here,
                    // to see if the transfer was successful or not
                    Console.WriteLine(
                        "{0}: {1} transferred", DateTime.Now, reader.GetAttribute("value"));
                }
            }
 
            ++position;
        }
    }
    catch (XmlException)
    {
        // If log was not closed, it is likely the XML is not well-formed
        // (at least top-level <session/> tag is not closed),
        // so we swallow the parsing errors here.
        if (logClosed)
        {
            throw;
        }
    }
 
    if (!logClosed)
    {
        // Wait for a while before retry
        Thread.Sleep(250);
    }
 
} while (!logClosed);