"Branding" a file with a standard hash algorithm lazarus / 21-General ---------------------------------------------------------------------- MarkMLl · 2024-10-30 11:05 Are there any inbuilt (RTL or FCL) functions which allow one of the standard hash algorithms to be applied on-the-fly to lines as they're read from or written to a text file? I'm assuming this would be OS- specific on account of the different styles of EOL marker. Application is automatically generating .pas wrapper files from a .inc, in order to counter annoying people who think that they're at liberty to edit a file with a prominent DO NOT EDIT THIS MACHINE GENERATED FILE warning it would be useful to be able to append a hash of the .inc and the .pas when generated. MarkMLl ---------------------------------------------------------------------- cdbc · 2024-10-30 11:17 Hi Hmmm.... I use this hash function, to compare archived and active files... Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function ElfHash(const aValue: string;const aTableSize: cardinal): cardinal; // 32 bit same as pjwvar  i, x: cardinal;begin  Result:= 0;  for i:= 1 to Length(aValue) do begin    Result:= (Result shl 4) + Ord(aValue[i]);  x:= Result and $F0000000;    if (x <> 0) then Result:= Result xor (x shr 24);    Result:= Result and (not x);  end;  Result:= Result mod aTableSize;end; { ElfHash }I just set the 'aTableSize' to e.g.: 193713 Dunno if that helps  Regards Benny ---------------------------------------------------------------------- MarkMLl · 2024-10-30 11:41 Quote from: cdbc on October 30, 2024, 12:17:31 pmHmmm.... I use this hash function, to compare archived and active files... Fairly easy to do that sort of thing, and I've got a fair amount of code under the overall aegis of the project in the context of which I was discussing 128-bit integers a couple of weeks ago https://forum.la zarus.freepascal.org/index.php/topic,68535.msg533794.html#msg533794 However what I'd like to be able to do here is use one of the standard hashes, so with reference to e.g. https://github.com/MarkMLl/asound if Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i $ sha1sum AsoundPcmDefs.inc415460cf18ce7db958c83feefc22c3439c1331e1  AsoundPcmDefs.inc $ sha1sum asoundpcm_dynamic.pas1f05e504ba7a55f47624a5204ab5f8299a42e28e  asoundpcm_dynamic.pas  I can append a line to the end of asoundpcm_dynamic.pas: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ...end.// SHA1 415460cf18ce7db958c83feefc22c3439c1331e1 1f05e504ba7a55f47624a5204ab5f8299a42e28e LF {}  At that point it's not only possible to demonstrate that the .pas matches the .inc, but by doing something like Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i $ head --lines=-1< asoundpcm_dynamic.pas | sha1sum1f05e504ba7a55f47624a5204ab5f8299a42e28e  -  it will be possible to demonstrate that the .pas file hasn't been "accidentally" fiddled with. Of course, anybody with a clue would know how to edit in an amended checksum. But in principle at least the number of people with a clue who would try to edit a file marked DO NOT EDIT THIS MACHINE GENERATED FILE is fairly small :-) MarkMLl ---------------------------------------------------------------------- cdbc · 2024-10-30 11:54 Hi Mark I did wonder what's wrong with md5sum / sha1sum... Regards Benny ---------------------------------------------------------------------- MarkMLl · 2024-10-30 11:59 Quote from: cdbc on October 30, 2024, 12:54:13 pmHi Mark I did wonder what's wrong with md5sum / sha1sum... Regards Benny Documented where? MarkMLl ---------------------------------------------------------------------- TRon · 2024-10-30 12:13 Quote from: MarkMLl on October 30, 2024, 12:59:43 pmDocumented where? https://www.freepascal.org/daily/packages/hash/index.html ---------------------------------------------------------------------- MarkMLl · 2024-10-30 12:34 Quote from: TRon on October 30, 2024, 01:13:48 pmQuote from: MarkMLl on October 30, 2024, 12:59:43 pmDocumented where? https://www.freepascal.org/daily/packages/hash/index.html Damn. I should have known better than to trust the documentation... I've got to do something about this. I can see it's been there since at least 2.2.2... initially MD5 and presumably has gained more in the intervening years. MarkMLl ---------------------------------------------------------------------- TRon · 2024-10-30 12:52 Quote from: MarkMLl on October 30, 2024, 01:34:23 pmI can see it's been there since at least 2.2.2... initially MD5 and presumably has gained more in the intervening years. Yes, it has been around for quite a while now. Always nice to have them in the FCL instead of the need to rely on some 3th party crypto/hashing lib (though I always considered Wolfgang's libraries very pleasant (sadly he passed away) ---------------------------------------------------------------------- MarkMLl · 2024-10-30 13:06 Quote from: TRon on October 30, 2024, 01:52:47 pmYes, it has been around for quite a while now. Always nice to have them in the FCL instead of the need to rely on some 3th party crypto/hashing lib (though I always considered Wolfgang's libraries very pleasant (sadly he passed away) Particularly since this is something that I might offer as a tool to the project rather than sticking it on my Github page: it would be highly undesirable for a standard tool to have an external dependency... or for that matter to be too specific about which version of FPC it required. Looking at local sources: I can't see hash anywhere in fpcdoc, certainly up to 3.2.2. Does that mean that it's only been added to the list of .chm files etc. recently, and where can I find the most recent working copies? The reason I'm wondering is that /if/ I spend any time working on the permuted index I mentioned a few months ago https://forum.lazarus.free pascal.org/index.php/topic,67879.msg523666.html#msg523666 I suppose it would be fair to try to tack on a list of packages/libraries which are "believed to exist although documentation might be incomplete". MarkMLl ---------------------------------------------------------------------- Thaddy · 2024-10-30 13:26 The hash package is indeed fairly recent but that doesn't mean fpc was without hashes before that, only not in a dedicated package. units crc and md5 for example are very old and several other hashes are scattered all over the sourcecode like the sha family. Both crc and md5 are well documented units. As is generics.collections, which contains a fancy collection (sic)  of hashes. I don't know if it is documented in the documentation in trunk. Time to build that for a change, but is is so tedious and builds only on linux. And the hash package on its own is not a compendium of hashes, alas. ---------------------------------------------------------------------- MarkMLl · 2024-10-30 13:36 Quote from: Thaddy on October 30, 2024, 02:26:13 pmThe hash package is indeed fairly recent but that doesn't mean fpc was without hashes before that, only not in a dedicated package. units crc and md5 for example are very old and several other hashes are scattered all over the sourcecode like the sha family. Both crc and md5 are well documented units. As is generics.collections, which contains a fancy collection (sic)  of hashes. I don't know if it is documented in the documentation in trunk. Time to build that for a change, but is is so tedious and builds only on linux. The hash package has been there AS I SAID since at least 2.2.2. Don't know about the others: they're at neither https://www.freepascal.org/docs-html/current/rtl/index.html nor https://www.freepascal.org/docs-html/current/fcl/index.html... and I /did/ start there before starting this thread. From the POV of somebody approaching FPC wondering if it will do something he needs, the capability isn't there. MarkMLl ---------------------------------------------------------------------- Thaddy · 2024-10-30 13:42 Well what you need is a crc hash, because that can be error correcting to a certain extend and crc32 is afaik documented and implemented  in at least 3 places. You can store the crc's of the strings in the inc w/o lineendings in a separate file if that is what you mean? Side note: there once was an extremely powerful set of components and tools called "dream '. I had that once, but all I have now is the license... :'( anybody knows what happened to it? Last known location Austalia, but written by Rusians. ) that contained sourcecode that does exactly what you want and more: I used it to create diffs between licensed software and demo software of the same. The patch could be applied to the demo to have the licensed version of it. I know, mea culpa. Youth sin. ---------------------------------------------------------------------- MarkMLl · 2024-10-30 13:54 Quote from: Thaddy on October 30, 2024, 02:42:22 pmWell what you need is a crc hash, because that can be error correcting to a certain extend and crc32 iss documented and implemented  in at least 3 places. You can store the crc's of the strings in the inc w/o lineendings in a separate file if that is what you mean? I DO NOT need a CRC32. I need one of the standard checksums that people use for this specific application, i.e. compatible with cksum, md5sum, or whichever variant of SHA is hash de jour. Not only have I described what I need in sufficient detail that everybody else has got the idea, but CRCs aren't even particularly good at detecting changed text: they were specifically designed to detect the sort of short-run impulsive noise found on telecoms circuits, hence the involvement of CCITT et al. MarkMLl ---------------------------------------------------------------------- TRon · 2024-10-30 14:22 Quote from: MarkMLl on October 30, 2024, 02:06:23 pmLooking at local sources: I can't see hash anywhere in fpcdoc, certainly up to 3.2.2. Does that mean that it's only been added to the list of .chm files etc. recently, and where can I find the most recent working copies? I have been reading the doc since it was present in daily docs (never seen it in mainstream docs). Up to date documentation (trunk) is situated here although I have to admit not being able to locate any reference to package hash or one of the specific units. I am also a bit lost there. Anyone who has a clue ? QuoteThe reason I'm wondering is that /if/ I spend any time working on the permuted index I mentioned a few months ago https://forum.lazarus. freepascal.org/index.php/topic,67879.msg523666.html#msg523666 I suppose it would be fair to try to tack on a list of packages/libraries which are "believed to exist although documentation might be incomplete". Well, the code at least does exist ---------------------------------------------------------------------- Thaddy · 2024-10-30 15:18 As does the doc: https://www.freepascal.org/daily/packages/hash/md5/index-5.html ---------------------------------------------------------------------- MarkMLl · 2024-10-30 15:34 OK, but we're back to the problem that if we start at https://www.freepascal.org/daily/doc/fpctoc.html the hash unit isn't indexed, and that's presumably what's going to become 3.4. Now /if/ I have time to work on the permuted index issue I should be able to follow Marco's advice and scavenge all .chm files... once I get them (somebody's already pointed me at a binary archive). But it would still be useful to know what files in the source tree contribute to e.g. hash.chm: not so much because I relish trying to work from (La)tex source (although I've done weirder things wearing my tech writer's hat) but because it would be useful to be able to look at that and divine what's missing from the user-facing website. MarkMLl ---------------------------------------------------------------------- TRon · 2024-10-30 16:43 Quote from: Thaddy on October 30, 2024, 04:18:18 pmAs does the doc: I was asking about the source of the documentation for package hash (not the generated docs). I seem unable to find them in the source-tree. ---------------------------------------------------------------------- MarkMLl · 2024-10-30 17:22 Quote from: TRon on October 30, 2024, 05:43:03 pmI was asking about the source of the documentation for package hash (not the generated docs). I seem unable to find them in the source-tree. So we can't find the source, and it's not indexed. I wonder whether those observations are related? MarkMLl ---------------------------------------------------------------------- TRon · 2024-10-30 18:40 @MarkMLI: Might also be because I'm too daft to be able to find it atm Having said that, it seems there is some action in/for that package and I also noticed a fcl-hash package directory so maybe someone is in the process of putting some things together and/or reorganize things as we speak (just speculation, I haven't kept a close eye on it). ---------------------------------------------------------------------- dsiders · 2024-10-30 18:48 Quote from: TRon on October 30, 2024, 05:43:03 pmQuote from: Thaddy on October 30, 2024, 04:18:18 pmAs does the doc: I was asking about the source of the documentation for package hash (not the generated docs). I seem unable to find them in the source-tree. There are no FPDoc files under version control for the fcl-hash package. If there were content files, you would see: Code: Bash  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i fpasn.xmlfp ecc.xmlfpecdsa.xmlfphashutils.xmlfppem.xmlfprsa.xmlfpsha256.xmlfpsha51 2.xmlfptlsbigint.xmlonetimepass.xml  They don't exist. All of the generated docs for these units in the daily builds are entirely empty skeletons generated by fpdoc for the source files. If someone wants to "do something about it", just write the docs and submit them as a patch. ---------------------------------------------------------------------- TRon · 2024-10-30 18:52 Splendid and thank you dsiders. That clears things up. I had the notion that in order to create even a skeleton doc that a minimal xml /must/ be present. Seems I got that wrong. Thank you for letting us know 👍 ---------------------------------------------------------------------- dsiders · 2024-10-30 18:58 Quote from: TRon on October 30, 2024, 07:52:53 pmSplendid and thank you dsiders. That clears things up. I had the notion that in order to create even a skeleton doc that a minimal xml /must/ be present. Seems I got that wrong. Thank you for letting us know 👍 This is nothing new. Look for fcl-image docs, and many others. They don't include anything in product release docs that aren't "completely" documented. If you document one unit, but not others in the same package, they generally don't include it in the *-project.xml files for fpdoc. Just my observation. ---------------------------------------------------------------------- TRon · 2024-10-30 19:26 @dsiders: You are far more versed in the topic of documentation than I probably ever will (I'm just dipping my toes, thus still al lot to learn). Any hint is considered a major leap forward ---------------------------------------------------------------------- MarkMLl · 2024-10-30 19:42 Quote from: TRon on October 30, 2024, 08:26:35 pm@dsiders: You are far more versed in the topic of documentation than I probably ever will (I'm just dipping my toes, thus still al lot to learn). Any hint is considered a major leap forward  I'd definitely second that. But riding my usual hobbyhorse: what's the minimum that needs to be done to get the unit into the TOC? MarkMLl ---------------------------------------------------------------------- dsiders · 2024-10-30 20:07 Quote from: MarkMLl on October 30, 2024, 08:42:49 pmQuote from: TRon on October 30, 2024, 08:26:35 pm@dsiders: You are far more versed in the topic of documentation than I probably ever will (I'm just dipping my toes, thus still al lot to learn). Any hint is considered a major leap forward  I'd definitely second that. But riding my usual hobbyhorse: what's the minimum that needs to be done to get the unit into the TOC? MarkMLl My opinion? If you want a unit included then the package needs to be documented. Technically, the package has to be added to the fcl- project.xml for fpdoc. Obviously, the people who generate the docs will make the final decision.