Subj : Squish dupe detection. To : Dieter Ringhofer From : mark lewis Date : Fri Nov 29 2002 02:21 pm DR>>> Gigo is known to be a dupe catapult since many years. ml>> what's wrong with the KEEP_THREADS option? DR> Sorry, I'm not familiar with Gigo and don't know what this DR> features offers for this. I have put it into trash after DR> having a look at it many years ago due to it's msgid handling. i don't know what version you looked at but there were specific changes made to MSGID stuff in GIGO... FSC-0035 is supported... SAFE_THREADS maintains a local database of MSGID, References and Newsgroups instead of trying to store all this in the FTN control lines... it allows for safer message threading and dupe checking at the cost of ~150 bytes per message... there is a maint program to be run against the database periodically... the May 24, 1994 version contained specific work done for squish problems... [quote] MSGID and Message-ID stuff worked on, trying to remove an incompatability with Squish 1.10's dupe checker. Also trying to get messages from internet->fidonet->internet to re-use the original Message-ID: field, so that dupe checkers on the net will catch it (Apparently, at least one country has a problem with dupe loops between the gates and the net). MSGID's when using the NIKE option are now "midxxxxxxxxxxxxx" instead of "" .. The brackets were causing the new Squish 1.10 to think everything was a dupe. Great, eh? [/quote] and from OPTIONS.CFG... [quote] ; Message threading! Here's a nasty option. Turn this on ; to implement some really naaaaassssytty looking MSGID kludge ; lines. ; When people reply to these messages, most BBS packages will ; create a "REPLY" kludge line. When GIGO gets a REPLY ; that it recognize, it will generate the "References:" line ; (the internet equivalent). (This _was_ "NIKE" during some ; beta versions) KEEP_THREADS ; See SAFE_THREADS for the suggested alternative ; A better option is the SAFE_THREADS option. ; Benefits: ; MSGID's look nice and clean ; Newsgroups: and Followup-To: lines honored (Even multi-group ones!) ; References: properly generated ; Disadvantages: ; Big database :-) I use a 20 meg database for 2 weeks of info. ; Slower on outgoing (back to usenet) messages; a good 1 second delay. ; ; Try it; I think you will like it. Be sure to run the CLEANID ; program to maintain the database size! It can run as part of your ; normal processing, since it has options to only act when the database ; gets "too big" by your definations. ;SAFE_THREADS [/quote] DR> The problem with Gigo is it's way to deal with msgids. Take DR> following scenario of dealing with any posting: DR> FTN1 - Gigo - RFC - Gigo - FTN2 DR> Means: DR> At system FTN1 somebody posts something. You have a Gigo based DR> gateway to RFC (news) there as well. Another system picks up DR> this posting from RFC and uses Gigo to gate it to FTN. DR> When having a close look at this posting you'll encounter DR> THREE different msgids in dependancy on where you pick it: DR> 1. the real one DR> 2. the one generated by Gigo at first gate DR> 3. the one generated by Gigo at second gate DR> Now you can link systems FTN1 and FTN2 (they feed each other DR> directly or via many other systems). What happens? a dupe loop that would happen just like it would for echomail... DR> FTN1, where posting originates, gets the very same posting DR> from FTN2 and is unable to identify it to be a dupe. Gigo DR> performs gating again, ... . DR> You will have a mail running in a never ending loop! until someone notices it and breaks the loop just like already has to be done with some echomail loops... DR> It doesn't matter where posting originates in reality (RFC, DR> FTN, ...). Effect is always the same: DR> Gigo produces "new" postings. DR>>> Therefore it's recommended to get rid of it or to use it only DR>>> when you are not feeding any other system. ... ML>>>> I'll try some batch file magic on the incoming ... DR>>> I don't believe you will be very successfull. Instead of ... ml>> the sources for GIGO are available and this stuff could be fixed... ml>> hummm... maybe i'll approach the author and see about him putting it on ml>> sourceforge for group development and refinement... DR> Several years ago author of Gigo has been asked to fix this DR> behaviour and he refused to do it. i remember something about some of this but i never got signed up to the support areas before jason left fidonet and other things that took him away from us... however, i do specifically remember something about some "formula" being used that would create the same msgid on all systems pulling the same message... however i don't recall what ever happened with that... no matter... if GIGO is put on sourceforge someone can still fix this problem if it is still a problem... in any case, the sources to GIGO are not the same as the last binary version released due to drive crash and bad backups... so some work is needed anyway to bring it back up to the level it currently is at... DR> Over here in Germany you'll encounter many FTN-RFC gateways DR> but, AFAIK most of them use Fidogate or parts of it at least. DR> Any gate will be closed very fast when first dupe is DR> encountered. the same happens (or should) when any echomail dupe loop is discovered... however, we must also note that some dupe loops are by design... consider a fully connected backbone network... hubA=========hubB | \ / | | \ / | | \ / | | \ / | | \ / | | X | | / \ | | / \ | | / \ | | / \ | | / \ | hubC=========hubD dupes are a way of life and required to ensure proper message propogation... however, those dupes should not be sent to systems linked off each hub... DR> Reason for high grade of sensibility about this is our cost DR> structure when it comes to telephon: There's no free ride DR> normally (only with one company and there only on sunday when DR> you sign a specific contract). Most often it's cheaper to do DR> long distance calls instead of doing local calls. Therefore we DR> don't have a defined routing structure except of several DR> systems being the "backbone", region and maybe zone gates. It DR> looks like a chaotical knot. This chaotical system works fine DR> since years and: Nobody can destroy it (we had such an DR> exercise in 1993). i heard of that incident... glad the system is robust enough to handle such... )\/(ark * Origin: (1:3634/12) .