move or copy box show no files and gets timeout message

Hey Martin,

just had an idea, i'm moving a 225GB folder on the server, but only the base folder gets displayed in the move box and then the server non-responded time-out shows up after 15 seconds, killing any chance of getting update on progression in the window.

This happens because of using the mv command to move the entire folder, but would it be possible to load the list of files to move and pass them one by one to the mv command so the window with the progress indicator can be updated and thus, helping to eliminate the time-out error ?

i don't have any idea how the move is progressing and it won't stop for another hour, maybe 2.

i really think sending one file at a time through that mv command will at least allow the popup box to be updated for each file rather than just waiting for the mv command to finish the entire base folder.

it would be awesome if you could add that feature

What protocol?

scp

but does it matter? it should work regardless , universally.

Will consider this.
But note that moving file one by one is a different operation than moving whole folder. Timestamps, permissions and ownership of the folder(s) will change.
Also moving files one by one can take a magnitude longer than moving whole folder at once.

hmm, the mv command would change timestamp?

well that's no fix. that shouldn't happen.

but i know what you could do, copy the file with -p, that will preserve permissions/mode/owner then you can delete the original before moving on to the next file.

as for taking a bit longer for the whole procedure, if you mean because of having to populate the file queue, we're used to it on windows, it has to build a list of files to get that progress bar populated (how many files make 100%), you can't know how many files there are until you run the whole tree.

plus if a destination file already exists now you can popup a message asking to confirm. something helpfull like w8.1 overwrite message showing file info for both sides might be helpfull.

and as an aside, when you invoke the cp command, is there a way to wait for it to return something rather than just assume something is wrong?

like in dos batch files when using start you can pass the /wait flag, and the batch pauses until its done.

after all isn't a time out just another way of saying the server is too busy, maybe finishing up the task?

i'd do something different than a popup when the timeout is reached, i have an idea: is there any way to get the pid of the command you've just invoked? if so then look if it's still running. if so then assume all is fine and check every few seconds until it's not in ram anymore, then check destination and do what needs to be done then move on to the next file. that is if you can't pause the procedure while the copy is on going.

food for though.

smitty123 wrote:

as for taking a bit longer for the whole procedure, if you mean because of having to populate the file queue, we're used to it on windows, it has to build a list of files to get that progress bar populated (how many files make 100%), you can't know how many files there are until you run the whole tree.

It's not bit longer, it's magnitudes longer. It's not because of the queue.
It because for each file, you have to issue a separate mv command, wait for the server to respond and repeat zillion times. It's the lag caused by sending the command and waiting for the response (which typically takes longer than the actual move operation) is what makes the difference.

i'd do something different than a popup when the timeout is reached, i have an idea: is there any way to get the pid of the command you've just invoked? if so then look if it's still running. if so then assume all is fine and check every few seconds until it's not in ram anymore, then check destination and do what needs to be done then move on to the next file. that is if you can't pause the procedure while the copy is on going.

That could work with the SCP protocol only, when WinSCP operates over a shell. Not for other protocols. And I'm not going to implement a SCP-only solution. The SCP is dead.

martin wrote:

smitty123 wrote:

as for taking a bit longer for the whole procedure, if you mean because of having to populate the file queue, we're used to it on windows, it has to build a list of files to get that progress bar populated (how many files make 100%), you can't know how many files there are until you run the whole tree.

It's not bit longer, it's magnitudes longer. It's not because of the queue.
It because for each file, you have to issue a separate mv command, wait for the server to respond and repeat zillion times. It's the lag caused by sending the command and waiting for the response (which typically takes longer than the actual move operation) is what makes the difference.

The procedure as it is causes the timeout popup, a popup that closes the session and leaves us in turn unable to keep track of how it is going.

Because of that popup, i had to spend 10 hours testing my backup files.

There are no shortcuts in life. A shortcut that causes a problem is no shortcut at all.

Hello Martin,

in v5.7.5 the move box still isn't doing things right, would you pelase do something about that ?

the way i see it, it's very simple, the problem is the box itself waiting and timeing out because the server doesn't respond in time.

maybe one way to alleviate this problem would be to increase the time out but also, if instead of waiting for the whole command to finish and report back to the box, instead it would work best to do 1 file at a time.

it's a much shorter wait to wait for a file than for a whole folder to finish.

so, with 5 or 6 lines of code, you could change the way this move/copy box operates and in return change the way we appreciate your program.

right now, as it is, each time i have to copy or move folders i'm apprehensive to think this will cause a problem and i'll have to deal with it. either by verifying my files with an md5 or crc check with another program. or jsut "hope for the best". that isn't the best way to do things.

Gather the names of all files and folders in source then create the target/destination folder and copy/move 1 file at a time to the target in same folder. easy peasy.

A simple for next loop and use the cp or mv function where needed from source folder to target. run up the tree and report back after each file to the move box so it doesn't time out.

this way, you'll be able to update the progress bar often and keep things going instead of timing out.

hope this works, i'm very much like to find a solution to this problem.

cheers

Did you consider using SFTP? With SFTP the move works as you want.

martin wrote:

Did you consider using SFTP? With SFTP the move works as you want.

the move command doesn't work with sftp, but for some reason it's in the context menu when trying to move a file. maybe remove the menu entries that aren't allowed by protocol.

scp allows the move, and doesn't bitch about time out if you set the time out to the max.

but no progress bar on big files or folders or time estimate which would really be useful.

selecting a bunch of small files does update the bar. but who cares about small files, its the big gigabyte files that take longer if moving by network.

consider this scenario: 200GB+ (backup img) files needing to be moved from hdd to usb every month.

either plug the usb in a computer or plug it directly to the nas. bypassing network and allowing the computer to do something else.

that's when winscp comes in handy.

sure i could use putty and command line but who wants to do that when you've got a gui.

smitty123 wrote:

the move command doesn't work with sftp, but for some reason it's in the context menu when trying to move a file. maybe remove the menu entries that aren't allowed by protocol.

It does work. What does not work particularly?

it returns error 4 in SFTP for moving files across 2 drives in the same nas box with the same file system. but scp works fine.

Anyway, the whole point is to make the copy/move box behave properly.

what i'm seeing is it just seems like you simply issue the mv or cp command with source and target, and wait for the whole tree to be moved or copied and some timeout to occur.

Do you know how long that could take, to move a 200GB folder ?

i do, hours. You can't check on progress , for hours.

Plus the timeout generated and the way it's handled need to be improved, adding insult to injury, winscp just terminates the session and we're left wondering what is going on.

No progress bar movement, no estimate, you don't even get an "Overwrite file yes/no?" popup if it target file already exists.

So there's no way to keep track of how it's doing, no way to reconnect to that particular session if timeout, so we're left to just wonder.

But let's think of another way...

So instead of just issuing a simple command like "mv source target" to the server and making a cute lil window appear on screen, let's try make it smarter and be more in control.

Once you know the source and target, show the move box on screen,
then in the code have it run up the source tree and build a list of files and folders to move, like Windows does in preparation of the move/copy.

This will give you 2 things: all possible information on what files to copy and add up how many bytes to copy which in turn gives you better control over the box on screen.

With that you can do many things:

display each file name as it is moved
display a progress bar that actually works in real time
display an estimate of how long this will take for the whole job
if the file or folder already exists or some disk error, you have better control of what to do in that case

And last but not least: show a cancel button that will do more than just look good and then get disabled when procedure starts ;)

(click cancel = kill pid on server and clean up the half-copied target file)

another thing, the timeouts are a real p.i.t.a. yunno ?

maybe there's a better way to handle them...

hmm this is interesting for getting information on the file copying progress... https://unix.stackexchange.com/questions/66795/how-to-check-progress-of-running-cp

For the timeouts, just thinking out loud here, let's see...

ok, we both agree timeouts are a pain but what you do with them doesn't have to be.

Do you get a pid of each cp command your program issues ? if so then we're in luck ! (if not there are still ways, check the link above)

That means you can handle it smarter !

Here's a scenario: big file or folder copy starts, wait, wait, and timeout happens.

instead of just popping up a box that says timeout and break connection...

Let's check if the cp command's pid is still in memory on the server (start a 2nd connex in background silently if you need to).

That will tell you 2 things: 1 the server is still alive and 2 the copy/move is still working or not.



When the move is happening and i get a timeout in winscp, i usually fire up Putty or the nas's interface and look at the processes in memory, i can see the cp or mv procress, so i know it's still working.

So why can't winscp do the same ?




As it is programmed right now, it's very unnerving when we get a timeout error.

most of the time its just that 1 connection to that the server that isn't responding to but it can respond to other connections. So it's not the server it's the way the process is programmed by the server devs.

But we the user get kicked out of winscp and we don't know if the move is still happening (yes we can check the server with top, but winscp has a gui and it should handle it better)

So i'd appreciate it, if or when you have some time, give it a shot ?

Let us know, i'd appreciate if you could find a way.

smitty123 wrote:

When the move is happening and i get a timeout in winscp, i usually fire up Putty or the nas's interface and look at the processes in memory, i can see the cp or mv procress, so i know it's still working.

So why can't winscp do the same ?

I'm no *nix expert, but I doubt that's a portable platform-independent and shell-independent way to to this.

i'm no expert neither, i just picked up a few things when the QNAP experts do their thing on my nas at home via teamviewer.

i don't even have a linux PC at home, strictly a dos/windows user (hence using winscp as it's got a gui and handles all that minutiae for the user, i like simple things that i can turn on. Like me! lol)

Just my nas that uses some form of linux, idk it's all bundled.

Anyway, i know the following commands work on my nas: stat, top, pidof, kill, cp,mv, du, find, grep (not pgrep)
the "pidof" command where you get your cp process pid, might be useful later.

And i'm no expert either, but i guess there are other integrated basic linux commands you use in winscp.

That is why i use winscp and that is why you have to stay in control of the situation and not just have winscp sit and wait for the server to finish or get a timeout then cut the connection if something goes wrong.

you could either rewrite the whole copy procedure and do a loop and send each file 1 by 1 from source to target with cp/mv

or you could read these 2 links, paying attention to

messages #6 and #2.1 and #2.2 of https://unix.stackexchange.com/questions/65077/is-it-possible-to-see-cp-speed-and-percent-copied

and messages #16 and #11 and #5 of https://unix.stackexchange.com/questions/66795/how-to-check-progress-of-running-cp

this should give you the tools to get the information needed to display in the copy/move box in winscp, to update on the progress of the move or copy.

this all goes in the move/copy box to show the information to the user.

Estimates: #5 and 16 of link 2 it doesn't have all the math but its basic idea is there.

Use 1 main connection to get status information to display in the box on screen and 1 background connection to do the transfer.


This way your program does the same job but is better informed, it stays updated of what happens and you can give the user pertinent information like estimates of time it will take based on the stats you get from the server with another connection.

Display that in the copy/move window and keep that cancel button active !

if we press cancel, idk if you can send the cancel signal or just do a brute force thing: kill the process with the pid you already know and if it was in the middle of copying a file then delete the file at destination because the file is only half copied.

With your knowledge of the linux commands it'll only take you a couple hours of testing to figure this all out.

Like i said, put the knowledge about what you do know to your service.

the important thing here is what you do with a timeout.
you could either keep waiting or do what you do now, kill the connection and let the user worry about what's going on.

personally, i'd just try to see if the cp is still going and if it is then don't bother the user with the timeout, just wait it out, giving updates as it goes along.