Incomplete files from scheduled ftp transfer.

Hello,

Incomplete files can download because the new file appears in the directory before it is completely saved to disk. Any data in buffers (or still to be generated) never downloads.

We wondered if file locks could resolve this situation? But unless both the FTP server and the application savings files into your monitored directories apply exclusive file locks, there is no guarantee that an incomplete file will not download.

I have verified this by setting a file lock on a file on my server. If I attempt to edit the same file while requesting a lock, my session pauses; if I just start in editing without referencing the lock, I can edit the file notwithstanding a lock. Sadly, that is how file locking operates: any participant that ignores the lock can bypass the synchronization imposed by the locking mechanism.

Onto TurboFTP: I then attempted to download the locked file through TurboFTP to a PC client in my lab through my FTP server. Since the file downloads without incident, I conclude that this FTP server (vsftpd on Linux) ignores file locks.

We could advise you to implement file locking in your applicadtion, but without a FTP server that respects file locks a secure solution can not be achieved. Moreover, you may not have authority to install another FTP server (that may be your ISP's perogative) and unless you have source to your application, implementing file locks may not be possible.

Nevertheless, there may be a satisfactory solution: If you save these files into another directory and then move (or rename) them into your download directory, then only complete files should appear for download. It is vital that you move these files because if you copy them the same phenomenon can occur: namely, the file name enters the directory while the system is copying the file contents. However, if you move (or rename) the file, then its location on disk never changes - only links within directories. Therefore, the download should be complete as soon as a file appears in the download directory.

Please let us know if this information has been helpful.

Alan

Nevertheless, there may be a satisfactory solution: If you save these files into another directory and then move (or rename) them into your download directory, then only complete files should appear for download. It is vital that you move these files because if you copy them the same phenomenon can occur: namely, the file name enters the directory while the system is copying the file contents. However, if you move (or rename) the file, then its location on disk never changes - only links within directories. Therefore, the download should be complete as soon as a file appears in the download directory.

I looked around for an option to rename the files on the way through but was unable to find the option within the schedule task settings.

I was going to do as you say and copy to a 'temp' directory then when completed 'rename' to a new directory - which sounds like it would work.

But not having the option rules it out.

Am i missing something?

>Some files are erroring due to the pdf files being incomplete.

What errors you actually refer to?

Leon

Leon,

I believe this user hopes to use TurboFTP to upload PDF files to their server. He observes incomplete (and consequently unusable) PDF files on the server.

These incomplete files may occur if an upload aborts or requires a restart. And during any upload there is always a time when the PDF file is allegedly available because it appears in the upload directory, but all the buffers have not been transmitted and saved on the server.

I believe an upload should be directed into a hidden directory where files are not published until all the data has been saved. Then, the files can be moved into the published directory without exposing an incomplete file.

I know I can rename files within command line FTP clients. Can I do the same within TurboFTP? I believe I can because I uploaded a file into a directory test1 on my test server with TurboFTP and then moved the file into a directory test2 using TurboFTP. If I select a file in the server pane, the popup menu (use the right mouse button) offers options to "Move to ...." or "Rename ...". The commands logged are those I would type to perform such a task in a command line FTP client.

These commands would appear to be what our user wants for this part of the task. Can he script this through TurboFTP's task Schedule? Otherwise, he would have to manually move files after their upload.

Alan

I need to see the log to know what errors the user is referring to and I need more details that relate to the user description "Some files are erroring due to the pdf files being incomplete. " Does "incomplete" mean the transferred file is corrupted in transit?

One way to prevent corrupted file in upload or download is to enable
Options | Advanced Con't | Check transferred file integrity

When enabled, the program will issue XCRC command after each file transfer to verify if the transferred file matches the original. If not, the file will be retransferred until it passes the CRC check.

Again, please forward us the task log showing the error so that we can better understand the problem.

Thanks.

Leon

Leon, i should probably have made it more clear - the errors are not in turbo ftp but are in our Prepress software.

The software is erroring when it trys to process the pdf files which turbo ftp is copying from the ftp site because they are incomplete.

I am trying to work out a way of ensuring the pdf files are never left incomplete after a sync cycle has passed.

I have the 'Check file integrity already activated'

As the other admin said i really would prefer it if turbo ftp built the file in a temp directory first and then copied it completely over into the main directory. Is there any way to configure this using the scheduled sync?

Hello,

Let's suppose the first download was always a known file (say I_AM_DOWNLOADING.txt). Then, any downloads would follow. And finally, the last file to download would be another special file (say I_AM_DONE.txt). With appropriate choice of filenames or timestamps, I believe this could be accomplished.

Could you script on your system to delay any processing if the first of the special files is in the destination directory, but the final one hasn't arrived yet? After processing, the special files would need to be cleared.

Alan

Unfortunately the files do not come down in any particular order.

As they are made and checked they are sent to our ftp site.

I have come to the conclusion that there should be an option to build and check the file integrity in a temp folder before copying to the destination.

This would be better in a commercial situation like ours.

Hi Kwijibot,

I am suspecting it is caused by another problem. Do you have the file corruption when running multiple upload tasks at the same time? If so upgrade to 5.00.540 and see if it still persists.

Leon

Hi Kwijibot,
I am suspecting it is caused by another problem. Do you have the file corruption when running multiple upload tasks at the same time? If so upgrade to 5.00.540 and see if it still persists.
Leon

Yes it sounds like my problem but i do not upload only download.

I have updated and will see if it is better.

Kwijibot.

Hi,

How does it work out for you?

Leon

Hi,

A fix is on the way you will get rid of the problem with 5.30.

Leon

I've been looking for this information for a long time, and the material in your article has been quite useful to me in my work driving directions

I have been searching to find a comfort or effective procedure to complete this process and I think this is the most suitable way to do it effectively. BAM University

Mmm.. good to be here in your article or post, whatever, I think I should also work hard for my own website like I see some good and updated working in your site. BAM University

This is very useful post for me. This will absolutely going to help me in my project. BAM University

Hey There. I found your blog using msn. This is a very well written article. I’ll be sure to bookmark it and come back to read more of your useful info. Thanks for the post. I’ll definitely return. BAM University

It was a very good post indeed. I thoroughly enjoyed reading it in my lunch time. Will surely come and visit this blog more often. Thanks for sharing. BAM University

I have recently started a blog, the info you provide on this site has helped me greatly. Thanks for all of your time & work BAM University