[SOLVED] TextFile Type and EOF getting tricked by non-printable chars lazarus / 21-General ---------------------------------------------------------------------- Gizmo · 2021-11-18 15:18 Some users have reported an issue to me about a program that simply reads very large text files, line by line, and does stuff with the data on each line. The problem is that the source data that it is designed to read seems to have changed slightly since my last version of the program in that some of the values in it are not always typical ASCII chars. But each line still ends with a Line Ending (0x0D0A). My program is set to expect files of type "TextFile" as input files. So EOF and ReadLn usually reads the lines of the text file, line by line. But, it seems these invalid chars are fooling EOF and making it think it has reached the end of the file, when it hasn't. The source files have hundreds and thousands of lines, and it is stopping on a line that contains invalid chars. For example like the one attached, which shows chars 0x18, 0x0B, 0x1A in the "filename" section. All three chars are not part of the usual A-Z or 0-9 character set. So, how can I fix this? How can I tell EOF to "carry on"? Redcued form of the code is below to avoid pasting the whole functions. But if you want to see the full code, it is here (https://github.com/tedsmith/NSRL- Stripper/blob/master/unsrlstripper.pas#L115) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  FileIn : Textfile;  FileOut : Textfile;  begin  try    AssignFile(FileIn, OpenDialog1.FileName);    reset(FileIn);  finally    //   end;   ...   while not EOF(FileIn) do   // this stops being executed AFTER reading a line with non-valid text chars in it? Not sure why  begin         ...  end; ---------------------------------------------------------------------- marcov · 2021-11-18 15:29 Hex 1A (CTRL-Z) is dos EOF char, but it's behaviour is controlled by boolean CtrlZMarksEOF. It seems that this is a typed constant so it should be possible to modify it with the right $J state ? ---------------------------------------------------------------------- Gizmo · 2021-11-18 15:52 QuoteHex 1A (CTRL-Z) is dos EOF char - Ah, well that explains a lot already! Quotemodify it with the right $J state - you've lost me there a bit. I know its a compiler directive, but not sure how to implement what you mean for this situation? ---------------------------------------------------------------------- howardpc · 2021-11-18 16:30 The default is to allow assignments to typed constants {$J+}. But in case not just doCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$J+}CtrlZMarksEOF := False; Quite likely CtrlZMarksEOF is False anyway, but I think the default setting is platform dependent. A self-explanatory, but more verbose, alternative to {$J } is {$WriteableConst On} (or {$WriteableConst Off}). ---------------------------------------------------------------------- MarkMLl · 2021-11-18 17:44 Quote from: marcov on November 18, 2021, 04:29:34 pmHex 1A (CTRL-Z) is dos EOF char, but it's behaviour is controlled by boolean CtrlZMarksEOF. I'd have said that even in the DOS era relying on ^Z was pretty unusual. It was a CP/M thing (hence also CCP/M, CDOS, MDOS and so on) since the original CP/M filesystem had a filesize granularity of 128 bytes (i.e. it didn't store an exact length). There are still rare situations where a (text) file can end up being padded with ^Z, e.g. if it's been through an X-Modem transfer. Taking that into account, if a program is EOFing on ^Z where that isn't actually padding to the physical end of the file it's a bug. In something :-) MarkMLl ---------------------------------------------------------------------- Gizmo · 2021-11-18 17:47 Thanks guys. Howard - yes, adding {$J+} at the top and CtrlZMarksEOF := False; in my function has now worked a treat! No more issue. The cause is an interesting one, and I have reported it to the people who prepare the source data, in case it is non-intentional. In fairness, the data sets they are preparing are tens of millions in volume. Nevertheless, I now don't need to worry about it either way. Thanks guys.