DeleteFile() fails on Win10 lazarus / 34-Windows ---------------------------------------------------------------------- dbannon · 2020-01-27 12:37 In my app, I move one file over top of another. You know, write out a file in /tmp then, when its all written, use the new file to replace an existing one. But on Windows, I found that a few users have reported this is not working, I do this - Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     {$ifdef WINDOWS}                    // Windows cannot 'move' over existing file.    if not DeleteFileUTF8(FileName) then begin        if not DeleteFile(FileName) then begin                NoteLoc.ErrorStr:='Failed 2nd Delete';                exit(False);      end;    end;   {$endif}    result := RenameFileUTF8(TmpName, FileName); As you see, I have tried both lazfileutils DeleteFileUTF8 and sysutils DeleteFile(), neither will remove this file.  I cannot replicate this behavour but two of my end users can. They confirm that - * they see the error about Failed 2nd Delete * that the file is still there after the exercise * that they can, manually, remove the file themselves. My own Win10 system does not exhibit this problem but others do. Can anyone suggest whats happening here please ? Davo ---------------------------------------------------------------------- marcov · 2020-01-27 12:43 Windows can hold short transient locks. Avirus software can also. If it fails, try to loop and retry a few times with say 10ms or 100ms pauses inbetween. There are several bugreports for fpmake with similar problems If you use Lazarus in default (utf8) mode, then just the deletefile is enough for FPC3.0+.   The -utf8 versions are mostly a remainder of  before 3.0 ---------------------------------------------------------------------- Thaddy · 2020-01-27 14:40 "if not DeleteFileUTF8(FileName)"? On Windows? Windows is an UTF16 OS. To me it looks suspicious. DeleteFileW (Which is supported by FPC in the windows unit) See https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf- fileapi-deletefilew and notice DeleFileTransacted too and this: https://docs.microsoft.com/en-us/windows/win32/fileio/deprecation-of- txf The crux is that an application other than yours or your application itself may still hold a lock on a file. If that is the case you can not delete it unless you explicitly remove the lock first or wait for the other application to finish: that may be in some cases when such an application is shutdown. On your own application, make sure you do not have a dangling file handle, make sure it is nilled first, before re-using that handle. ---------------------------------------------------------------------- ASBzone · 2020-01-27 16:32 Quote from: dbannon on January 27, 2020, 01:37:44 pmIn my app, I move one file over top of another. You know, write out a file in /tmp then, when its all written, use the new file to replace an existing one. But on Windows, I found that a few users have reported this is not working, I do this - ... My own Win10 system does not exhibit this problem but others do. Can anyone suggest whats happening here please ? Davo What are the file names that this is happening with? ---------------------------------------------------------------------- Bart · 2020-01-27 16:37 Quote from: Thaddy on January 27, 2020, 03:40:41 pm"if not DeleteFileUTF8(FileName)"? On Windows? Windows is an UTF16 OS. Why for Pete's sake would that make you suspicious. Did you even bother to look at the implementation of DeleteFileUTF8? In old Lazarus versions it did explicit conversion from UTF8 to WideString, then called the W-API. In trunk it simply calls SysUtils.DeleteFile() (which does the W-API call). If that makes you suspicious, you might need medication. Bart ---------------------------------------------------------------------- balazsszekely · 2020-01-27 16:45 @Davo Perhaps this helps: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  if not DeleteFileUTF8(FileName) then    ShowMessage(SysErrorMessage(GetLastOSError)); For example:  1. If I try to delete "c:\nothing.txt", the OS returns: "Cannot find the file specified."  2. If I try to delete the currently running application(DeleteFileUTF8(Application.ExeName)), the OS returns: "Acces denied", for obvious reasons. ---------------------------------------------------------------------- 440bx · 2020-01-27 17:04 Quote from: dbannon on January 27, 2020, 01:37:44 pm* that they can, manually, remove the file themselves. Can anyone suggest whats happening here please ? GetMem's suggestion above is the appropriate first step.  Figure out what it is that Windows didn't like in the attempt to delete the file. The fact that the users can manually remove the file themselves also opens the possibility Thaddy mentioned that the file might be being read by an antivirus or some other system "protector" getting in the way of getting things done.  As Marcov suggested, it might worth putting the delete calls in a short "auto retry" loop (10 times ?) with a 10ms delay.  If possible, save the "GetLastError" values and incorporate them in the "delete failed" message.  At least that way you will have a better idea of what may be the cause of the problem. HTH. ---------------------------------------------------------------------- marcov · 2020-01-27 17:30 See https://bugs.freepascal.org/view.php?id=21868 ---------------------------------------------------------------------- dbannon · 2020-01-28 06:03 Thanks folks. I have made a test version of the app that captures the SysErrorMsg  and also waits a bit before trying again. Posted that test version and have to wait for the affected users to try it out. Good advice from people who know Windows far better than me ! Davo ---------------------------------------------------------------------- Thaddy · 2020-01-28 07:55 Quote from: Bart on January 27, 2020, 05:37:29 pmIn trunk it simply calls SysUtils.DeleteFile() (which does the W-API call). Well then. That's what I mean. Nice to see that in trunk, because as you state this has not always been the case.  That still leaves the locking and transacted deletes. ---------------------------------------------------------------------- PascalDragon · 2020-01-28 08:21 Quote from: Thaddy on January 28, 2020, 08:55:36 amQuote from: Bart on January 27, 2020, 05:37:29 pmIn trunk it simply calls SysUtils.DeleteFile() (which does the W-API call). Well then. That's what I mean. Nice to see that in trunk, because as you state this has not always been the case.  Even in older versions it called DeleteFileW. Here is the code from Lazarus 1.0. If you follow the initialization for the function variable DeleteFile_ you'll notice that it's initialized to a function that calls DeleteFileW except on older Windows versions (older than Windows 2000). ---------------------------------------------------------------------- Bart · 2020-01-28 09:20 Quote from: PascalDragon on January 28, 2020, 09:21:08 amEven in older versions it called DeleteFileW. As I clearly stated. Thaddy however seems to have a partial blindness for what other people write. Most likely that is not of an ocular origin. Bart ---------------------------------------------------------------------- dbannon · 2020-01-30 23:47 OK, I have finally sorted this out, superficially at least. While I cannot reproduce the problem myself, a cooperative end user has been running tests and returning log file to me. Whats happening I believe is that when you call DeleteFileUTF8() in windows, the file is (almost always?) deleted but sometimes the function returns false.  My guess is that its not having its return value set reliably. There is a path through the function that allows Result to be random.  As the function very rarely fails, few of us check its return value and it has gone unnoticed. My diagnose is based on calling DeleteFileUTF8() and if it returns false, call FileExists() to really see if its gone. If FileExits() retuns false, allow the programe to proceed, hopefully normally. So far, his tests indicatate that is the case. I will, in the near future, try and step though the Windows code (bit out of my comfort zone) and will follow up on what I find. For the record, this end user also has Vera-Crypt running on his windows box and also uses PortableApps.com.  I don't think there is any connection but worth noting..... Davo ---------------------------------------------------------------------- ASBzone · 2020-01-31 01:16 Quote from: dbannon on January 31, 2020, 12:47:31 amOK, I have finally sorted this out, superficially at least. While I cannot reproduce the problem myself, a cooperative end user has been running tests and returning log file to me. Whats happening I believe is that when you call DeleteFileUTF8() in windows, the file is (almost always?) deleted but sometimes the function returns false.  My guess is that its not having its return value set reliably. There is a path through the function that allows Result to be random.  As the function very rarely fails, few of us check its return value and it has gone unnoticed. My diagnose is based on calling DeleteFileUTF8() and if it returns false, call FileExists() to really see if its gone. If FileExits() retuns false, allow the programe to proceed, hopefully normally. So far, his tests indicatate that is the case. I will, in the near future, try and step though the Windows code (bit out of my comfort zone) and will follow up on what I find. For the record, this end user also has Vera-Crypt running on his windows box and also uses PortableApps.com.  I don't think there is any connection but worth noting..... Davo When AntiMalware tools come into play, they can complicate things.  This can cause delete functions to take more than a normal amount of time, because something else is inspecting the file before finally removing it. If you get more reports on the issue, we should be able to troubleshoot it some more. ---------------------------------------------------------------------- dbannon · 2020-01-31 01:52 No ASBzone, I have put quite a lot of code in there  to diagnose whats happening. As soon as I get a false back from DeleteFileUTF8() I read the system error and then check to see if the file is actually there or not.  Fileexists() tells me its not there anymore.   (So does the system error msg) This, on this fellow's machine, is quite reliable. I also have code in there that dumps details of the file, sleeps for a 10mS and tries again, then for 30mS and, finally for 100mS logging each step carefully.  But its never executed because the file really has been deleted. The problem really does appear to be that DeleteFileUTF8() can return false even when it correctly deleted the file. Davo ---------------------------------------------------------------------- MoCityMM · 2020-03-09 14:33 Was looking for something else but, ran into this post. If I am reading this correctly... Seen similar 'symptoms' with other applications on shared folder (network shares) and local folders. Could be ACL (access control list) or UAC (which is a pain), if an application writes the file - it's the owner of the file in some circumstances (especially if there is a system account associated with the application) and the user may not have access to delete the file. I would check to see what 'group' the user belongs to: 'Users', 'Power Users', etc... If they are part of the Administrators group and it still happens, run iCACLS on file and see if you can delete the file then. Try the link below, it's worth a shot: https://theitbros.com/using-icacls-to-list-folder-permissions-and- manage-files/ -Mo ---------------------------------------------------------------------- dbannon · 2020-03-09 22:38 Because I cannot, myself replicate this problem, its quite hard to work on. I have had two users affected, the first reported the issue, I assume he has stopped using my app and does not respond.  The second person was a lot more helpful, but is now convinced that the problem relates to his use of drive encryption. Between us, we spent a lot of time on this problem, I have to assume he, too, has got to the point where he does not want to put in any more. I have much of my diagnostic code still in there, if it happens to someone else then I can track it much quicker but I suspect it is no longer something that is going to trouble us. So, thanks for your input MoCityMM but its looking like a cold case. Davo ---------------------------------------------------------------------- avra · 2020-03-10 08:20 Quote from: dbannon on January 31, 2020, 02:52:49 amThe problem really does appear to be that DeleteFileUTF8() can return false even when it correctly deleted the file.So at the end this could be fixed with just additional check for FileExists()? ---------------------------------------------------------------------- dbannon · 2020-03-10 22:18 Yes avra, thats what I have done in my code (for windows only). So far, with this users experience, we see only the first log error. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     if not DeleteFile(FullFileName) then begin        ErrorMsg := SysErrorMessage(GetLastOSError);        Debugln('Failed using DeleteFile - file name is :' + FullFilename);        Debugln('OS Error Msg : ' + ErrorMsg);        if not FileExistsUTF8(FullFileName) then          debugln('But, FileExists says its gone, proceed !')        else begin                debugln('I can confirm its still there .');                Debugln('Trying a little sleep...');                sleep(10);                if not DeleteFileUTF8(FullFileName) then begin                     if not FileExistsUTF8(FullFileName) then                   debugln('DeleteFileUTF8 says it failed but FileExists says its gone, proceed !')                else exit(false);          end;        end;    end; I'd be a lot happier if I could see it myself ... Davo ---------------------------------------------------------------------- winni · 2020-03-10 22:50 Hello! What says the OS Error Msg?? Before  that I would insert: Io := IoResult; and then write it with DebugLn Might be at the edge of "Off Topic" but Reminds me to Win 3.1 and our first Netware Server. ( I know, grandpa talks about the war .....) You thought all disk operations were executed, but they were not: It gave the new IOerror 32, but no software new about it. The error said something about "File Sharing Violation". You needed some new addon for the Netware Driver. Maybe it's something similar. Winni ---------------------------------------------------------------------- dbannon · 2020-03-11 01:04 Quote from: winni on March 10, 2020, 11:50:36 pmHello! What says the OS Error Msg?? Before  that I would insert: Io := IoResult; and then write it with DebugLn .... Winni, I do grab SysErrorMessage(GetLastOSError) and drop that out in the debug line. From memory (at end users request, I have closed the issue ticket) it reported that the file could not be found to delete. But, as noted, it appeared it does try to delete it and successes. Davo ---------------------------------------------------------------------- jmnj · 2020-06-17 15:33 This same issue drove me NUTS for some time.  Here's a snippet what I use to reliably delete files in Windows 10... // Requires Units: LazFileUtils, LAZUTF8 FileName:='String with full path of file to delete.'; DeleteFileUTF8(UTF8ToWinCP(FileName)); Hope this helps! ---------------------------------------------------------------------- Remy Lebeau · 2020-06-17 19:59 Quote from: jmnj on June 17, 2020, 05:33:31 pmHere's a snippet what I use to reliably delete files in Windows 10... One, the FileExists() check is unnecessary, as DeleteFileW() will do the same check and set GetLastError() accordingly. Two, why would you pass a CP-encoded string to a UTF-8 function? ---------------------------------------------------------------------- jmnj · 2020-06-18 13:05 Hi Remy, Thanks for your comments.  I've removed the FileExists().  Explanations below: Honestly, I haven't delved into why this works.  I did initially pass UTF8 without success.  Tried racking my head on this for weeks before I was willing to try "anything". For some unknown reason this does work on Windows 10, post update KB4556799 using Lazarus 2.0.8 r62944.  I just don't know exactly -why- it does.  I would love to know why this works and I'd love to know if the "normal" way of deleting files gets a permanent fix so that I'm not resorting to a clear hack.  I'm kinda between a rock and a hard place using this hack on a critical project. ---------------------------------------------------------------------- dbannon · 2020-06-20 00:48 Quote from: Remy Lebeau on June 17, 2020, 09:59:04 pm.... One, the FileExists() check is unnecessary, as DeleteFileW() will do the same check and set GetLastError() accordingly. ..... Just for the record, several variations that I sent the end user apparently received no error message from DeleteFile() and DeleteFileUTF8() but a subsequent call to FileExists() revealed that the file was, in fact still present.  And, on one occasion, I saw a log that indicated, perhaps, the file was deleted but DeleteFile() said it failed !   However, as I later found out, the user was using a disk encryption system and a portable application handler to run it, collectively, we think that was somehow related to the issue. An update to VeraCrypt seems to have helped. I could not duplicate the end user's problem myself so was dependent on sending him/her a test binary. Fortunately, this was a very cooperative user. One other person did have what might have been a similar problem but with hind sight I don't think that is the case.  Early stages of this problem was complicated by the fact that Linux/Mac are quite happy to overwrite an existing file with RenameFileUTF8() but Windows is not.  But I recently discovered that Linux is also upset about using RenameFileUTF8() on an existing file if that existing file is on an SMB share. And smb is, of course a Windows protocol ...... I am convinced that this is one very specific, isolated case.  I have left the obsessively careful code in there for Windows in my app, it does no harm and may, perhaps do some good ? Thanks for input folks ! Davo ---------------------------------------------------------------------- Remy Lebeau · 2020-06-20 22:08 Quote from: dbannon on June 20, 2020, 02:48:37 amJust for the record, several variations that I sent the end user apparently received no error message from DeleteFile() and DeleteFileUTF8() but a subsequent call to FileExists() revealed that the file was, in fact still present. That is certainly a possibility, if the file is in use at the time it is "deleted".  If the file is open, it will be marked for deletion, and won't actually be physically deleted until all open handles to the file are closed. Quote from: dbannon on June 20, 2020, 02:48:37 amAnd, on one occasion, I saw a log that indicated, perhaps, the file was deleted but DeleteFile() said it failed ! That, I don't see being possible, unless something else deleted the file after DeleteFile() was called.  Unless the log was wrong/misleading, for instance if the log showed the file was *about to be* deleted, and then afterwards showed the *actual delete* failing.