Forum » Support and Bug Reports »

"Endurance" setting does not prevent "atomic write" problem

Advertisement

javasniffer
Joined:
Posts:
6
Location:
St Albans, UK

"Endurance" setting does not prevent "atomic write" problem

Problem: "Endurance" option does not prevent read error during transfer.

Description: Some of our PHP files are large, and take up to 5 seconds to upload to our web server. If someone visits the page during the upload, they retrieve only part of the file, as SFTP is writing it.

I understand that this used to be a known problem, and I see that the "Endurance" feature has now been added so that uploaded files are copied to a temporary file first, and only on completion of the upload are they copied/renamed, thereby reducing the time window of a partial read to almost zero.

However, if I set Endurance ON (all files) and request the file using HTTP during the upload, I still get a partial file read.

Settings: WinSCP 4.1.8, SFTP-3, Commander View (double-click and drag/drop both fail))

Any help would be very welcome.

WinSCP log: 
Copying 1 files/directories to remote directory "/var/www/vhosts/mydomain.com/httpdocs/TEST/"
  PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 01; Resume: Y (102400); CalcS: Yes; Mask: *.*
  TM: M; ClAr: No; CPS: 0; ExclM(No): 
  AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml
File: "C:\www\mydomain\src\tableinfo.php"
Copying "C:\www\mydomain\src\tableinfo.php" to remote directory started.
Ascii transfer mode selected.
Opening remote file.
Type: SSH_FXP_OPEN, Size: 82, Number: 34563
Type: SSH_FXP_STATUS, Size: 24, Number: 34308
Discarding reserved response
Type: SSH_FXP_HANDLE, Size: 13, Number: 34563
Type: SSH_FXP_WRITE, Size: 4071, Number: 35078
10 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.
Type: SSH_FXP_CLOSE, Size: 13, Number: 36868
Type: SSH_FXP_SETSTAT, Size: 78, Number: 34825
Type: SSH_FXP_STATUS, Size: 24, Number: 36102
Type: SSH_FXP_STATUS, Size: 24, Number: 36358
Type: SSH_FXP_STATUS, Size: 24, Number: 36614
Type: SSH_FXP_STATUS, Size: 24, Number: 36868
Status/error code: 0
Type: SSH_FXP_STATUS, Size: 24, Number: 34825
Status/error code: 0
Listing directory "/var/www/vhosts/mydomain.com/httpdocs/TEST".
Type: SSH_FXP_OPENDIR, Size: 52, Number: 37131
Type: SSH_FXP_HANDLE, Size: 13, Number: 37131
Type: SSH_FXP_READDIR, Size: 13, Number: 37388
Type: SSH_FXP_NAME, Size: 12497, Number: 37388
Type: SSH_FXP_READDIR, Size: 13, Number: 37644
Type: SSH_FXP_NAME, Size: 12529, Number: 37644
Type: SSH_FXP_READDIR, Size: 13, Number: 37900
Type: SSH_FXP_NAME, Size: 12542, Number: 37900
Type: SSH_FXP_READDIR, Size: 13, Number: 38156
Type: SSH_FXP_NAME, Size: 12649, Number: 38156
Type: SSH_FXP_READDIR, Size: 13, Number: 38412
Type: SSH_FXP_NAME, Size: 12675, Number: 38412
Type: SSH_FXP_READDIR, Size: 13, Number: 38668
Type: SSH_FXP_NAME, Size: 12479, Number: 38668
Type: SSH_FXP_READDIR, Size: 13, Number: 38924
Type: SSH_FXP_NAME, Size: 12612, Number: 38924
Type: SSH_FXP_READDIR, Size: 13, Number: 39180
Type: SSH_FXP_NAME, Size: 13105, Number: 39180
Type: SSH_FXP_READDIR, Size: 13, Number: 39436
Type: SSH_FXP_NAME, Size: 12754, Number: 39436
Type: SSH_FXP_READDIR, Size: 13, Number: 39692
Type: SSH_FXP_NAME, Size: 12657, Number: 39692
Type: SSH_FXP_READDIR, Size: 13, Number: 39948
Type: SSH_FXP_NAME, Size: 12903, Number: 39948
Type: SSH_FXP_READDIR, Size: 13, Number: 40204
Type: SSH_FXP_NAME, Size: 13375, Number: 40204
Type: SSH_FXP_READDIR, Size: 13, Number: 40460
Type: SSH_FXP_NAME, Size: 6121, Number: 40460
Type: SSH_FXP_READDIR, Size: 13, Number: 40716
Type: SSH_FXP_STATUS, Size: 28, Number: 40716
Status/error code: 1
Type: SSH_FXP_CLOSE, Size: 13, Number: 40964

e.g.

-- time --> 1----------2----------3----------4----------5----------6----------7

1 = HTTP read file hello.php - OK
2 = start transfer of hello.php using SFTP-3 in WinSCP 4.1.8
3 = HTTP read file hello.php - ERROR
4 = HTTP read file hello.php - ERROR
5 = HTTP read file hello.php - ERROR
6 = finish transfer of hello.php using SFTP-3 in WinSCP 4.1.8
7 = HTTP read file hello.php - OK

Reply with quote

Advertisement

martin◆
Site Admin
martin avatar
Joined:
Posts:
41,468
Location:
Prague, Czechia

Re: "Endurance" setting does not prevent "atomic write" problem

This feature does not work for ascii transfers. Force binary transfer if possible. I have updated documentation accordingly.

Reply with quote

javasniffer
Joined:
Posts:
6
Location:
St Albans, UK

Thanks prikryl.

I have tried this, and it reduces the time window, but not to zero.

Is there *any* way to prevent a destination file being read while it is being moved/renamed remotely by SCP?

Reply with quote

martin◆
Site Admin
martin avatar

javasniffer wrote:

Is there *any* way to prevent a destination file being read while it is being moved/renamed remotely by SCP?
What difference that would make? You page still won't work.

Reply with quote

javasniffer
Joined:
Posts:
6
Location:
St Albans, UK

When WinSCP is being used to upload files to a live web site, there must be a mechanism - somewhere - to prevent partially written files from being read by the Web server program.

I was hoping that such a solution might be available using the SCP protocol and/or the WinSCP program - maybe I am wrong.

Perhaps the only way to achieve what I need (i.e. no read errors by the web server) is to implement a locking mechanism myself using a shell script? Currently, we upload files to the (live) web server directory using WinSCP, and this is causing problems.

As one of the main uses of WinSCP is to upload files to a web server, I think this is an important point that should be addressed here.

Does anyone else have any solutions?

Reply with quote

Advertisement

martin◆
Site Admin
martin avatar

Sorry I'm still not sure if I understand. So you would accept that the page does not load while uploading. While it is inacceptable that partial content is read?

Reply with quote

javasniffer
Joined:
Posts:
6
Location:
St Albans, UK

The requirement is that files in the web server's directory must always be read completely: before, during and after upload.

From your response so far, it looks like this problem will have to be solved outside of the scp transfer, e.g. by copying to a temporary directory which is not used by the web server:

a) scp files to a remote 'temp' directory, not used by the web server
b) lock the 'web' directory, the one used by the web server
c) copy the files locally from 'temp' to 'web' directory, using cp or rsync
d) unlock the 'web' directory

The web server would have to be configured to re-try for a period of time when it finds the 'web' directory locked. Does anyone know how to do this with Apache 2.0, or will Apache re-try automatically?

Reply with quote

javasniffer
Joined:
Posts:
6
Location:
St Albans, UK

I have found an answer.

In order to guarantee that a file exists (in its entirety) to any process both before and after the file contents are changed, cp() function will not work, as it is not atomic. Instead, the mv() function must be used. mv() uses rename() which is guaranteed to be atomic.

So, to prevent partial files being served by Apache, WinSCP alone is not enough. The files must be moved/renamed locally using mv().

e.g.

1. using WinSCP, copy fileA.html from the local machine to /path/to/TEST directory on the remote web server machine (where we are not concerned about fileA.html being read during the copying operation)

2. using a local cp() command on the remote machine, create a temporary copy of the file which we will move to the 'live' directory - cp /path/to/TEST/fileA.html /path/to/temp/fileA.html

3. using a local mv() command on the remote machine, atomically overwrite fileA.html in the 'live' web directory - mv /path/to/temp/fileA.html /path/to/live/fileA.html

Your script (or shell) will need write access to both directories /path/to/temp and /path/to/live.

----

Note: there is some disagreement about whether mv() in a given implementation uses the library function rename(), a function that is guaranteed to be atomic. Some implementations may use link() and unlink() which are *not* atomic, and will therefore not prevent partial reading of the web server files. You are advised to check your version of *nix.

Reply with quote

andipeh
Guest

This feature does not work for ascii transfers. Force binary transfer if possible

i have tested this now, i switched everyting to binary and enabled this resume/transfer to temporay file for all files.
but winscp still writes directly into the files, and crashing the scripts :(

please tell me what i can do

Reply with quote

Advertisement

javasniffer

WinSCP cannot be used to transfer files to a live web server, if you cannot tolerate partial file reads.

You must use mv on the server to move a copy of the file into the live web directory.

Reply with quote

martin◆
Site Admin
martin avatar
Joined:
Posts:
41,468
Location:
Prague, Czechia

andipeh wrote:

i have tested this now, i switched everyting to binary and enabled this resume/transfer to temporay file for all files.
but winscp still writes directly into the files, and crashing the scripts :(
Please post a log file again, with the new settings.

Reply with quote

martin◆
Site Admin
martin avatar
Joined:
Posts:
41,468
Location:
Prague, Czechia

javasniffer wrote:

WinSCP cannot be used to transfer files to a live web server, if you cannot tolerate partial file reads.

You must use mv on the server to move a copy of the file into the live web directory.
WinSCP does mv itself, if you "enable transfer to temporary file name".

Reply with quote

andipeh
Guest

thanks for the quick reply.

so here is my log. i have noticed already that the transfer is still done in ascii - so thats probably the reason why it does not create this filepart and do the mv afterwards.

i tried now everything to make the transfer binary but i couldnt. i selected it within the transfer selection on the top menu. then i changed every present and the default transfer type to binary. and finaly i removed .php from the text present. but it still takes this .php from somewhere else and starts a ascii transfer.

i dont know how much debug information you need. just tell me if you need more. 

> 2009-03-20 17:59:06.343 Type: SSH_FXP_REALPATH, Size: 10, Number: 109328
< 2009-03-20 17:59:06.593 Type: SSH_FXP_NAME, Size: 23, Number: 109328
> 2009-03-20 17:59:07.343 [Background 1] Type: SSH_FXP_REALPATH, Size: 10, Number: 109584
< 2009-03-20 17:59:07.609 [Background 1] Type: SSH_FXP_NAME, Size: 23, Number: 109584
> 2009-03-20 17:59:14.359 Type: SSH_FXP_REALPATH, Size: 10, Number: 109840
< 2009-03-20 17:59:14.562 Type: SSH_FXP_NAME, Size: 23, Number: 109840
> 2009-03-20 17:59:15.359 [Background 1] Type: SSH_FXP_REALPATH, Size: 10, Number: 110096
< 2009-03-20 17:59:16.281 [Background 1] Type: SSH_FXP_NAME, Size: 23, Number: 110096
. 2009-03-20 17:59:21.875 [Background 1] Copying 1 files/directories to remote directory "/var/www/skins/unstable/phplib"
. 2009-03-20 17:59:21.875 [Background 1]   PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 01; Resume: N (102400); CalcS: Yes; Mask: 
. 2009-03-20 17:59:21.875 [Background 1]   TM: A; ClAr: No; CPS: 0; ExclM(No): 
. 2009-03-20 17:59:21.875 [Background 1]   AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml
. 2009-03-20 17:59:21.875 [Background 1] File: "C:\DOKUME~1\ROLAND~1.PFA\LOKALE~1\Temp\scp45062\var\www\skins\unstable\phplib\global_config.php"
. 2009-03-20 17:59:21.875 [Background 1] Copying "C:\DOKUME~1\ROLAND~1.PFA\LOKALE~1\Temp\scp45062\var\www\skins\unstable\phplib\global_config.php" to remote directory started.
. 2009-03-20 17:59:21.875 [Background 1] Ascii transfer mode selected.
. 2009-03-20 17:59:21.875 [Background 1] Opening remote file.
> 2009-03-20 17:59:21.875 [Background 1] Type: SSH_FXP_OPEN, Size: 73, Number: 110339
< 2009-03-20 17:59:22.125 [Background 1] Type: SSH_FXP_HANDLE, Size: 13, Number: 110339
> 2009-03-20 17:59:22.125 [Background 1] Type: SSH_FXP_WRITE, Size: 4013, Number: 110854
> 2009-03-20 17:59:22.859 Type: SSH_FXP_REALPATH, Size: 10, Number: 113680
< 2009-03-20 17:59:23.109 Type: SSH_FXP_NAME, Size: 23, Number: 113680
. 2009-03-20 17:59:30.828 [Background 1] 29 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.
> 2009-03-20 17:59:30.828 [Background 1] Type: SSH_FXP_CLOSE, Size: 13, Number: 115204
> 2009-03-20 17:59:30.828 [Background 1] Type: SSH_FXP_SETSTAT, Size: 69, Number: 110601
< 2009-03-20 17:59:31.312 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 114694
> 2009-03-20 17:59:31.359 Type: SSH_FXP_REALPATH, Size: 10, Number: 115472
< 2009-03-20 17:59:31.625 Type: SSH_FXP_NAME, Size: 23, Number: 115472
< 2009-03-20 17:59:32.453 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 114950
< 2009-03-20 17:59:32.453 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 115204
< 2009-03-20 17:59:32.453 [Background 1] Status/error code: 0
< 2009-03-20 17:59:32.453 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 110601
< 2009-03-20 17:59:32.453 [Background 1] Status/error code: 0
. 2009-03-20 17:59:32.453 [Background 1] Listing directory "/var/www/skins/unstable/phplib".

Reply with quote

Advertisement

andipeh
Guest

i now tried the following:
i did not get the transfer option box for background transfers - so i tried to reenable it via preferences > Environment. i checked every confirmation box (exept the last one - continue on error)
also i removed the checkbox from Transfer > Background: Transfer on background by default and no confirmations for background transfers.

did not help - it still does the background transfers on its one - and does it still in ascii instead of binary.


here are some screenshots:

<invalid hyperlink removed by admin> (<invalid hyperlink removed by admin>)

then i tried to drag&drop the files from outside into winscp - and then all works fine. it seems only the background transfers are able to resuming.

<invalid hyperlink removed by admin> (<invalid hyperlink removed by admin>) 

. 2009-03-23 12:19:39.328 Copying 1 files/directories to remote directory "/var/www/skins/unstable/phplib/"
. 2009-03-23 12:19:39.328   PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 01; Resume: Y (102400); CalcS: Yes; Mask: *.*
. 2009-03-23 12:19:39.328   TM: B; ClAr: No; CPS: 0; ExclM(No): 
. 2009-03-23 12:19:39.328   AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml
. 2009-03-23 12:19:39.328 File: "C:\Dokumente und Einstellungen\roland.pfannhauser\Lokale Einstellungen\Temp\global_config.php"
. 2009-03-23 12:19:39.328 Copying "C:\Dokumente und Einstellungen\roland.pfannhauser\Lokale Einstellungen\Temp\global_config.php" to remote directory started.
. 2009-03-23 12:19:39.328 Binary transfer mode selected.
. 2009-03-23 12:19:39.328 Checking existence of file.
> 2009-03-23 12:19:39.328 Type: SSH_FXP_LSTAT, Size: 57, Number: 2231815
< 2009-03-23 12:19:39.421 Type: SSH_FXP_ATTRS, Size: 37, Number: 2231815
. 2009-03-23 12:19:39.421 Checking existence of partially transfered file.
> 2009-03-23 12:19:39.421 Type: SSH_FXP_LSTAT, Size: 66, Number: 2232071
< 2009-03-23 12:19:39.515 Type: SSH_FXP_STATUS, Size: 29, Number: 2232071
< 2009-03-23 12:19:39.515 Status/error code: 2
. 2009-03-23 12:19:39.515 Asking user:
. 2009-03-23 12:19:39.515 Remote file 'global_config.php' already exists. Overwrite?
. 2009-03-23 12:19:39.515  
. 2009-03-23 12:19:39.515 New:         227233 bytes, 23.03.2009 12:19:32
. 2009-03-23 12:19:39.515 Existing:    227227 bytes, 23.03.2009 12:14:56 ()
. 2009-03-23 12:19:44.171 Opening remote file.
> 2009-03-23 12:19:44.171 Type: SSH_FXP_OPEN, Size: 82, Number: 2232323
< 2009-03-23 12:19:44.265 Type: SSH_FXP_HANDLE, Size: 13, Number: 2232323
> 2009-03-23 12:19:44.265 Type: SSH_FXP_WRITE, Size: 4121, Number: 2232838
> 2009-03-23 12:19:46.515 [Background 1] Type: SSH_FXP_REALPATH, Size: 10, Number: 2235152
< 2009-03-23 12:19:46.625 [Background 1] Type: SSH_FXP_NAME, Size: 23, Number: 2235152
. 2009-03-23 12:19:47.250 19 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.
> 2009-03-23 12:19:47.250 Type: SSH_FXP_CLOSE, Size: 13, Number: 2235908
< 2009-03-23 12:19:47.546 Type: SSH_FXP_STATUS, Size: 24, Number: 2235398
< 2009-03-23 12:19:47.765 Type: SSH_FXP_STATUS, Size: 24, Number: 2235654
< 2009-03-23 12:19:47.765 Type: SSH_FXP_STATUS, Size: 24, Number: 2235908
< 2009-03-23 12:19:47.765 Status/error code: 0
> 2009-03-23 12:19:47.765 Type: SSH_FXP_REMOVE, Size: 57, Number: 2236173
< 2009-03-23 12:19:47.843 Type: SSH_FXP_STATUS, Size: 24, Number: 2236173
< 2009-03-23 12:19:47.843 Status/error code: 0
> 2009-03-23 12:19:47.843 Type: SSH_FXP_RENAME, Size: 118, Number: 2236434
< 2009-03-23 12:19:47.937 Type: SSH_FXP_STATUS, Size: 24, Number: 2236434
< 2009-03-23 12:19:47.937 Status/error code: 0
> 2009-03-23 12:19:47.937 Type: SSH_FXP_SETSTAT, Size: 73, Number: 2232585
< 2009-03-23 12:19:48.046 Type: SSH_FXP_STATUS, Size: 24, Number: 2232585
< 2009-03-23 12:19:48.046 Status/error code: 0
. 2009-03-23 12:19:48.078 Listing directory "/var/www/skins/unstable/phplib".

so ether "disabling background transfers" does not work this way or maybe its not meant to work this way? some with resuming/partfile-generating for background transfers.

thanks again for looking into it!
kind regards

Reply with quote

martin◆
Site Admin
martin avatar
Joined:
Posts:
41,468
Location:
Prague, Czechia

Can you send me an email, so I can send you back a debug version of WinSCP to track the problem? Please include link back to this topic in your email. Also note in this topic that you have sent the email. Thanks.

You will find my address (if you log in) in my forum profile.

Reply with quote

Advertisement

Guest

. 2009-03-23 12:19:39.328 Copying "C:\Dokumente und Einstellungen\roland.pfannhauser\Lokale Einstellungen\Temp\global_config.php" to remote directory started.


OK! What the hell! This certain name is my brothers name and it cant be any other. so whats going on with "remote directory..." in connection with his name?
please let me know!

thats what happens if you google family names...

Reply with quote

kyra13
Joined:
Posts:
1
Location:
USA

I tried it as well and something went wrong unfortunately :(
even don't know where to look for decedion...

Reply with quote

Advertisement

martin◆
Site Admin
martin avatar
Joined:
Posts:
41,468
Location:
Prague, Czechia

kyra13 wrote:

I tried it as well and something went wrong unfortunately :(
even don't know where to look for decedion...
What went wrong? How?

Reply with quote

test12567
Joined:
Posts:
1

I use a 5.17.7 client on Windows and can confirm that there is NOT atomic copying php files. Is it supposed to be so or can be overcome somehow via changing config or whatsoever?

Reply with quote

martin◆
Site Admin
martin avatar
Joined:
Posts:
41,468
Location:
Prague, Czechia

test12567 wrote:

I use a 5.17.7 client on Windows and can confirm that there is NOT atomic copying php files. Is it supposed to be so or can be overcome somehow via changing config or whatsoever?
Sorry, but this is quite vague. What protocol are you using? What are your transfer settings? A log file?

Reply with quote

Advertisement

You can post new topics in this forum

Documentation

Support

Associations

Follow Us