----------------------------------------
Ideas for an aggregator
February 06th, 2018
----------------------------------------
I want to build a new gopher aggregator, specifically for phlogs,
and I want it to work in a very particular way. I'm not sure this
is going to be a general solve for everyone, but if I can do it
right it will work for my personal use.
Version 1 Plan:
- It should know that a phlog has been updated, not a whole
gopherhole.
- I should provide it an array of gophermaps to keep track of.
- The gophermap will be hashed each day--store the hashes in
a file for simplicity, or a sqlite db-- and if the hash
changed from the previous day, that link is added to the
daily list.
- If the link doesn't resolve, ignore. It might be temporary.
- It should organize phlog updates by date so I can go back if
I missed a couple entries (like Bongusta)
- It should auto-update daily
- Output should be simple:
- Master gophermap is a symlink to the current month.
- Each gophermap is a year-month for easy archival
- Viewing the directory will list the archive and current
- Linking to current will include a link to the archive
(directory view)
Version 2 Plan:
- The link added to the aggregated list should point to the new
entry, not the main gophermap.
- Potentially do a diff on the gophermap if the hash has
changed and try to identify new links?
- Grab any new links and drop them in verbatim, potentially
- Relative links might need to be rewritten
- Problems with changed links?
- Would this be instead of linking to the map, or in addition
(sub links under a heading?)
- Allow a way for people to submit their phlog to the list and
make it a public system instead of a private aggregator
My plan was to implement this in Perl as a way to learn Perl.
I may still go ahead with that, but it might be easier for me to
build with my current knowledge in python. Regardless, if anyone
reading this wants to take a stab at it, go right ahead. I'd
rather have the tool than be the one to write it.