[SOLVED] How to cleanup ReadDirectoryChangesW operations properly?
lazarus / 34-Windows

----------------------------------------------------------------------
d7_2_laz · 2021-05-20 14:08

At the end of my migration of an old Delphi 7 app to Lazarus Windows
x64 i detected a remaining flaw with ReadDirectoryChangesW (in
combination with GetOverlappedResult).

The scenario is that i have a old grown dirwatcher component for a
file listview's folder elements.
The folder is same as the related folder in a directory tree.

The component, somehow adapted to Lazarus, works fine in nearly all
use cases, except one.
The problamatic case is when i rename a folder in the tree (with
DO_RENAME in SHFileOperationW) and _afterwards_ the parent folder of
that already renamed child folder,
then the file operationi failds with error ui "already in use".

Before those rename actions the filewatcher thread is terminated  and
a CancelIO done, somehting like "CancelWait" in related articles.

The problem appears to me to be that the ReadDirectoryChangesW resp.
GetOverlappedResult is not cleaned up properly resp. is sticky on a
folder once having renamed it so that i cannot continue on a parent
folder. of it
Tried a lot of modification on the stop thread / stop
ReadDirectoryChangesW, but keep to be stuck.

Yes, it's hard to say something about that without code, but it would
be very hard to condense the complex thing into a little test case.
Maybe a theoretical thought might help, convering "how to finish the
IOs of a ReadDirectoryChangesW properly".
Somebody encountered such?

----------------------------------------------------------------------
ASerge · 2021-05-20 18:29

Are you using CancelIO or CancelIoEx? Do you close the directory
handle when the thread terminates?

----------------------------------------------------------------------
d7_2_laz · 2021-05-20 18:54

CancelIO  (if handle <> 0 and <> INVALID_HANDLE_VALUE), then
CloseHandle

It's worthy to try CancelOEx for this special case?

----------------------------------------------------------------------
ASerge · 2021-05-20 20:07

Quote from: d7_2_laz on May 20, 2021, 08:54:50 pmIt's worthy to try
CancelOEx for this special case?
From docs: "The CancelIoEx function allows you to cancel requests in
threads other than the calling thread." You didn't say where the
cancellation was.

----------------------------------------------------------------------
d7_2_laz · 2021-05-20 20:56

Yes really,  that (in threads other than the calling thread) sounded
promising, but i didn't notice a difference.

I tried to test the cancellation from various places. For instance
from inside the control thread execute of later after having called
the thread terminate.

Maybe it is better for speaking to print an extract from the coding
here.  The control thread execute:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
CtrlThread.Execute;var  pBuffer :   Pointer;  // Buffer for return
values of ReadDirectoryChangesW  dwBufLen :  DWORD;   dwRead :  
 DWORD;   PInfo :     PFILE_NOTIFY_INFORMATION;    dwNextOfs : DWORD;
 dwFnLen :   DWORD;   Overlap :   TOverlapped;   WaitResult: DWORD;  
EventArray: Array[0..2] of THandle; // Array der Handles für
WaitForMultipleObjects  pFileNameForMessage: PWideChar; 
lasterrcode:Cardinal; ErrorMessage: string;  begin  try   // try ..
except    Cancel_IO;    if not DirectoryExists(FsDirPath) then begin 
    Terminate;       exit;    end;     FhFile:=
CreateFileW(PWideChar(UTF8Decode(FsDirPath)),                       
FILE_LIST_DIRECTORY or GENERIC_READ,                       
FILE_SHARE_READ or FILE_SHARE_WRITE or FILE_SHARE_DELETE,            
          nil,                       
OPEN_EXISTING,FILE_FLAG_BACKUP_SEMANTICS or FILE_FLAG_OVERLAPPED,    
                  0);    if (FhFile = INVALID_HANDLE_VALUE) or (FhFile
= 0) then begin        Terminate;        exit;    end;    
FileEvent:=CreateEvent(nil,FALSE,FALSE,nil);   
Overlap.hEvent:=FileEvent;   
TermEvent:=TEvent.Create(nil,FALSE,FALSE,TermEvName);   
SuspEvent:=TEvent.Create(nil,FALSE,FALSE,SuspEvName);   
EventArray[0]:=FileEvent;    EventArray[1]:=TermEvent.Handle;   
EventArray[2]:=SuspEvent.Handle;}    EventArray[0] := FileEvent;    
EventArray[1] := plocaleventrec(TermEvent.Handle)^.Fhandle;   
EventArray[2] := plocaleventrec(SuspEvent.Handle)^.Fhandle;    
dwBufLen := 65535;      // Dirwatch:  BufferSize (=32) *
SizeOf(TFILE_NOTIFY_INFORMATION);    pBuffer := AllocMem(dwBufLen);  
try      while not terminated do begin        dwRead := 0;          if
ReadDirectoryChangesW(FhFile, pBuffer, dwBufLen,                     
          prWatchSubTree,    //  true or false didn't make a
difference here                                 FFilter,      
 //FILE_NOTIFY_CHANGE_FILE_NAME or FILE_NOTIFY_CHANGE_DIR_NAME,      
                         @dwRead, @Overlap, NIL) then        begin   
     WaitResult := WaitForMultipleObjects(3, @EventArray, FALSE,
infinite);          case WaitResult of            WaitDir:           
   begin                 // WAIT_OBJECT_0                  if
GetOverlappedResult(FhFile, Overlap, dwRead, False) then begin       
            PInfo:= pBuffer;                     if dwRead = 0 then
begin   // check overflow                        ErrorMessage :=
SysErrorMessage(ERROR_NOTIFY_ENUM_DIR);                       
SignalError(ErrorMessage, ERROR_NOTIFY_ENUM_DIR);                   
 end;                     repeat                       dwNextOfs :=
PInfo.dwNextEntryOffset;                       FAction :=
PInfo.dwAction;                       dwFnLen :=
PInfo.dwFileNameLength;                       FsFileName :=
WideCharLenToString(@PInfo.dwFileName,dwFnLen div 2);                
      GetMem(pFileNameForMessage, dwFnLen + SizeOf(WideChar));       
              Move(PInfo.dwFileName, Pointer(pFileNameForMessage)^,
dwFnLen);                       PWord(Cardinal(pFileNameForMessage) +
dwFnLen)^ := 0;                        PostMessage(FWndHandle,
WM_DIRWATCH_NOTIFY, FAction, LParam(pFileNameForMessage));           
           pChar(PInfo) := pChar(PInfo)+dwNextOfs;                   
 until dwNextOfs=0;                  end else begin                  
  lasterrcode := GetLastError;                      ErrorMessage :=
SysErrorMessage(lasterrcode);                     
SignalError(ErrorMessage, lasterrcode);                      break;  
              end;  // GetOverlappedResult                end;       
   WaitTerm:                      // WAIT_OBJECT_0+1               
 // dont call Terminate  again here              ;           
WaitSusp: Suspend;             // WAIT_OBJECT_0+2            else    
          break;          end;    // case WaitResult
ReadDirectoryChangesW        end else begin           ErrorMessage :=
SysErrorMessage(GetLastError);           SignalError(ErrorMessage);  
    end;      end;    finally          FreeMem(pBuffer,dwBufLen);    
if FhFile <> 0 then         if FhFile <> INVALID_HANDLE_VALUE then
begin            CancelIOEx(FhFile, 0);   // or nil, no change       
   CloseHandle(FhFile);            FhFile := 0;         end;     
SendMessage(FWndHandle, WM_DIRWATCH_THREADTERMINATED, 0, 0);      with
TFldrControl(Owner) do           ThrdTerminated := True;    end;   //
try   except     on E :Exception do begin        ErrorMessage :=
E.Message;        SignalError(ErrorMessage);     end;  end; end;

----------------------------------------------------------------------
ASerge · 2021-05-21 15:51

Let's go back to the beginning to better understand.
Let the directory tree look like this: Level_0_A\Level_1_A\Level_2_A\.
You start monitoring Level_0_A from subtrees (by starting a thread).
At some point, you stop monitoring (via stopping the thread).
Then rename Level_2_A to Level_2_B (for example) - there are no
errors.
Then rename Level_1_A to Level_1_B - an error occurs.
Do I understand correctly?

----------------------------------------------------------------------
d7_2_laz · 2021-05-21 17:46

Hm, maybe it's easier with a small picture -> attached pic1.
And for simpliness, for the moment forget all monitoring of the tree
itself
and think in terms of: monitoring of one single dedicated folder,
which's filelist does appear on the right side.

(i need the ReadDirectoryChangesW only for the monitoring of a single
folder. That needs much more details information
about changes, ie. file size, date, attribues etc., which would not be
needed and overwhelming for a directory tree monitoring at all).

So, clicking onto child node "Level2_A", monitoring will start for
folder Level2_A, who's details are needed for the file list
informations.

Now: edit caption on the tree node "Level2_A" and rename it to
"Level2_A_test".
The thread will be stopped, waiting until terminated, CancelIO and
CloseHandle.
Then, after a certain security delay (avoid timing issues),
SHFileOperationW is called for to rename the folder.
If successful, the changed folder name will be passed to the file list
and the monitoring will restart for the new folder name
"Level2_A_test".

Now, click on parent node: Level1_A.
Monitoring is stopped and restarted --> now for the Level1_A folder,
and it's details will be displayed in the file list.

Now, When editing Level1_A eg. to "Level1_A_test". SHFileOperationW
will fail with "already in use". That was not the case with Delphi 7
(resp win x32?).

The other way round vice versa:
Rename the parent folder Level1_A to Level1_A_test first, and
afterwards it's child node (sub folder) Level2_A to Level2_A_test:
this does NOT lead to locking  issues.

In short: it appears to me that after stopping thread/monitoring for a
sub filder, it will be still locked by ReadDirectoryChangesW,
forbidding to rename it's parent.
What is needed to achieve is: after monitoring of a folder, get rid of
the locking by the monitoring mechanism. Unfortunately i got no clue
why this locking happens. So any idea about possible causes is very
appreciated.

What i even tried;  directly after a folder rename (with
SHFileOperationW):  get the its handle using CreateFileW and do a
CancelIOEx(FFileHandle, 0)  //  (0 (or nil?) for to stop _all_ iI/O
requests onto this handle). No difference.

----------------------------------------------------------------------
ASerge · 2021-05-22 10:17

There may be a problem in the SHFileOperationW function. Try renaming
the directory using the MoveFile function.

----------------------------------------------------------------------
d7_2_laz · 2021-05-23 12:02

Hm, wouldn't have suspected that side, but, after test, with MoveFile
basically it did behave the same (already in use within the scenario
described).
So i did spend some time to prepare a minimalistic test case, and at
least i can say it did reproduce the issue indeed.
Very simple Gui, 2 entry fields representing a parent and a sub folder
for to paste in,
and, if valid, one can select parent or child by button(analogous to
treenode click)  and then trigger a simple rename action on it.
Instrutions are in a memo field.  -> project attached

When preparing that code, i came more and more close to the impression
that the issue might be due to an inproper thread termination so that
the ReadDirectoryChangesW stays sticky on an old folder and hinders to
do a rename of the parent after having renamed a child.
Maybe Lazarus or x64 may require a slightly different coding here. But
where ..

----------------------------------------------------------------------
ASerge · 2021-05-23 18:13

Quote from: d7_2_laz on May 23, 2021, 02:02:18 pmAfter changing in
Fldrwatch_buffered.pas
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i    
EventArray[1] := plocaleventrec(TermEvent.Handle)^.Fhandle;   
EventArray[2] := plocaleventrec(SuspEvent.Handle)^.Fhandle;to
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  
EventArray[1] := THandle(TermEvent.Handle);  EventArray[2] :=
THandle(SuspEvent.Handle);everything works without errors.

----------------------------------------------------------------------
d7_2_laz · 2021-05-23 19:50

Yes ASerge, that brings the rename capability back (and proves it's
not an issue withReadDirectoryChangesW).

Unfortunately now it does hinder the monitoring completely, as the
WaitResult of WaitForMultipleObjects now will bring an WAIT_FAILED 
(=4294967295), where GetLastError is 6 = ERROR_INVALID_HANDLE.

Curious; exactly that had been my problem in https://forum.lazarus.fre
epascal.org/index.php/topic,53887.msg399495.html#msg399495
which leaded to the cast to be replaced.

You can verify that with the testcase by: selecting one directory,  do
a file change in this directoy ie. copy a file herein, and inspect the
WaitResult of WaitForMultipleObjects.

----------------------------------------------------------------------
d7_2_laz · 2021-05-24 12:10

I've the following picture:

// 27.03.2021: https://forum.lazarus.freepascal.org/index.php/topic,53
887.msg399495.html#msg399495
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
EventArray[1] :=
plocaleventrec(TermEvent.Handle)^.Fhandle;EventArray[2] :=
plocaleventrec(SuspEvent.Handle)^.Fhandle;
Remaining problem: ReadDirectoryChangesW does work fine, but it keeps
locking the folder ... because the thread does not terminate
correctly?

//  23.05.2021 this thread 
 https://forum.lazarus.freepascal.org/index.php/topic,54704.0.html
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
EventArray[1] := THandle(TermEvent.Handle);EventArray[2] :=
THandle(SuspEvent.Handle);
Problem: WaitForMultipleObjects fires WAIT_FAILED  (=4294967295),
where GetLastError is 6 = ERROR_INVALID_HANDLE
Monitoring does not happen at all,  and so the folder will not be
locked of course

I extended and corrected the test case a bit so that it shows up a
minimal monitor reporting for to prove that this really keeps to work,
and throws a message if the nandle for the terminate event is invalid.

Project attached.

----------------------------------------------------------------------
d7_2_laz · 2021-05-24 19:33

Finally i found it:  one line to change:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  //
PulseEvent(StopEvent.Handle);   //  With Delphi
7PulseEvent(THandle(StopEvent.Handle));  // Previous attempt with
Lazarus // ---- replace by:
 (SOLUTION): PulseEvent(plocaleventrec(StopEvent.Handle)^.Fhandle );
Thread's termination will be fired,  "while not terminated" loop can
be left, folder will be freed and parent folder can be renamed.
For everybody who is interested the corrected test case is attached.

----------------------------------------------------------------------
ASerge · 2021-05-26 16:10

Quote from: d7_2_laz on May 24, 2021, 09:33:54 pmCode: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i // ----
replace by:
 (SOLUTION): PulseEvent(plocaleventrec(StopEvent.Handle)^.Fhandle );It
is a pity that TEvent.Handle is not compatible with Delphi. This leads
to such errors.

About the mistakes. In the above code, I didn't see where dwBufLen is
initialized. Apparently the random value from the stack turns out to
be quite large.

In addition to the differences between Lazarus and Delphi, I will give
an example (I did it quickly) where such monitoring can be carried out
without additional threads in Lazarus.

----------------------------------------------------------------------
d7_2_laz · 2021-05-26 19:50

Hello ASerge, thanks again for your attention!

About the dwBufLen initialization, unfortunately it had become a
victim of editorial cuts from obsolete stuff in the first zip file.
I noticed that soon (ouch) and upfrom test_ReadDirectoryChangesW_2.zip
it was in again
 dwBufLen := 65535;
apparently close to yours: CBufferSize = 64 * 1024;

The intention of my own updates of the source i once found and adapted
many years ago simply had been to report file object changes in a
bulk. For to avoid frequent individual screen updates by the using
application (think of e.g. inserts or deletions of dozends of objects
at once).
I'm happy it does work again smoothly, but without the chance to
communicate within this great forum i wouldn't have had a chance.
It's really amazing that an old source basically kept working when
doing the migration to Lazarus x64, after changing a very few couple
of things.

I had not the possiblity to take a closer look at your example yet,
but will try it the next days
and rethink which use case scenario exactly might have had motivated
to use threads for that and if i could avoid them.

----------------------------------------------------------------------
d7_2_laz · 2021-05-27 11:06

Hello ASerge, i did try your example with great interest (such
examples are so helpful for users!) and try to summarize my picture
what's equal and what's different.
I think the core functionality of monitoring is similar, but simply
the method of push back of change info differs a bit.

Your's passes the change info, after GetOverlappedResult, directly
back to the Gui (UpdateLog) which is fully appropriate and sufficient
for most use cases. Such examples are a valuable source to learn!
Mine is, additionally, using WaitForMultipleObjects and, after the
GetOverlappedResult, it passes back the info to the Gui in a deferred
way.
By:  PostMessage to the parent control (TFldrControl), which stores
the data (BufferTheDataViaNotify) in a StringList (where the strings
do contain the names, and the objects keep an info about the change
type, e.g. "rename"). Then, controlled by a timer, those data are
pushed back to the Gui.
Honestly, in a retrospective i do not remember why a thread should be
needed for that (should be proven by practice).
If i could do it from the scratch, i would try to use your example as
base and extend it a bit for to use buffering (then the using app can
decide how to process such changes ...  for instance rebuild a list
completely or maintain items individually), but without
threads.Shouldn't be a big thing-

----------------------------------------------------------------------
ASerge · 2021-05-27 13:23

The method without threads also has its drawbacks. The message queue
of the main thread is used to wait for the event. But if you call up
some system dialog box, such as opening a file, then it will have its
own queue. This means that the event will not be available until the
dialog closes.

----------------------------------------------------------------------
d7_2_laz · 2021-05-27 17:40

... so if one relies on that the monitor is independent from possible
blocking effects from user interactions,
then the thread approach might be a more desired option. The specific
use case each needs to be considered, i think.
Thanks a lot for all your feedbacks ASerge!

----------------------------------------------------------------------
helmutboeckmann · 2025-03-13 14:04

Quote from: ASerge on May 26, 2021, 06:10:52 pmQuote from: d7_2_laz on
May 24, 2021, 09:33:54 pmCode: Pascal  [Select][+][-]window.onload =
function(){var x1 = document.getElementById("main_content_section");
if (x1) { var x = document.getElementsByClassName("geshi");for (var i
= 0; i // ---- replace by:
 (SOLUTION): PulseEvent(plocaleventrec(StopEvent.Handle)^.Fhandle );It
is a pity that TEvent.Handle is not compatible with Delphi. This leads
to such errors.

About the mistakes. In the above code, I didn't see where dwBufLen is
initialized. Apparently the random value from the stack turns out to
be quite large.

In addition to the differences between Lazarus and Delphi, I will give
an example (I did it quickly) where such monitoring can be carried out
without additional threads in Lazarus.


How can I also monitor the subfolders using the sample program
ReadDirChanges.zip?
Thank you.
Regards, Meridian