Turns out floodgap was running their indexer at the same time I was
running this version of gspider.  They found a lot more sites and
selectors, traversing only type 1 dirs just like me.  I suspect the
html-ification lynx does is my problem.  Perhaps there is more than
one URL on a line sometimes?

Also, the floodgap indexer obeys robots.txt and never hits the same
site more than a couple times a minute - much better than mine.

