Post B5mqqVTxipCLJ8t2Q4 by dunkelstern@corteximplant.com
 (DIR) More posts by dunkelstern@corteximplant.com
 (DIR) Post #B5mhkvPBWrS4Nhupo8 by dunkelstern@corteximplant.com
       0 likes, 0 repeats
       
       And if I am bored: I already collected 3.2 million external links to crawl from about 300k Wikipedia articles.Sounds like i really have to implement that web-crawler now.#diySearchEngine
       
 (DIR) Post #B5mhkvkSFmSvRfroxc by daniel@social.telemetrydeck.com
       0 likes, 0 repeats
       
       @dunkelstern I have a book recommendation that might be interesting to you, about how to build search indexes: https://www.researchgate.net/publication/2352627_Modern_Information_Retrieval
       
 (DIR) Post #B5mqdjUyIZPf0vGf2G by dunkelstern@corteximplant.com
       0 likes, 0 repeats
       
       @daniel thanks for the recommendationAs i guess from the preview PDF that’s most about handling the very low level tasks of creating the indexes itself, which is nice to know mind you, but a solved problem with modern tooling (Postgres already can do text vector search with stemming and normalization and HNSW indexing for embedding vector search) and i want to avoid bloom filters and such stuff to represent the search result because i want the users to understand how the results are built. If you throw too much advanced math on that you will get an opaque list of stuff where probably most results you need are in there but you cannot understand why (like google or bing, the first 5 organic results that are not promoted or sponsored stuff are clearly what you wanted, the next 10 are okay, you can probably guess why they rank high and then comes a list of stuff that’s there because of some algorithm that is in eternal churn of relevance vs. SEO)I want to present an interface to the user where they decide how to drill down into a result area and understand what is happening. The goal is more the discovery mode you had with the old web: surfing and letting it take you to nooks you didn’t know existed or researching and being able to trace your steps through the topics you followed up while searching.Nevertheless, i am getting that book 🙂 basics are fun too
       
 (DIR) Post #B5mqlNCb50MEBCJgCe by daniel@social.telemetrydeck.com
       0 likes, 0 repeats
       
       @dunkelstern oh yes I should have said so more clearly – this is just super interesting but might not have practical applications for you right now. I read this when I worked for the cliqz search engine and it's just nicely written and super interesting
       
 (DIR) Post #B5mqqVTxipCLJ8t2Q4 by dunkelstern@corteximplant.com
       0 likes, 0 repeats
       
       @daniel no harm done, i interpreted the recommendation exactly that way 🙂