I've created an open-source search platform - Swirl. You can search
and use AI to find the correct information quickly and easily.
reddit / selfhosted

----------------------------------------------------------------------
search_guy · 2023-09-27 12:04

Our data is spread across multiple sources like SQL databases, Emails,
Notes, Words, Office Applications, etc. And when we're searching for
something, we typically don't know where.

So, me and my friends came up with the id ea of a search platform,
Swirl. That allows us to search multiple sources to get the
information we want. And we made it open source to help people who
want to add search capabilities. We're expanding it with Large
Language Models to generate the answers based on the queries being
asked for.

It's all open source and available on GitHub:
[https://github.com/swirlai/swirl-
search](https://github.com/swirlai/swirl-search)

----------------------------------------------------------------------
Normal_Inevitable880 · 2023-09-27 12:29

Wow, just checked. That's a nice idea. To add multiple search
providers to a single search. Can you elaborate on how are you guys
doing it?

----------------------------------------------------------------------
search_guy · 2023-09-27 13:38

Right on, thanks. Swirl's approach is to (a) adapt the query for each
configured source (aka SearchProvider), (b) unify the results using
jsonpath mappings, (c) re-rank the results using LLM, (d) return the
ranked results... with no code required to configure querying sources
that don't have a new transport. For example youc an get a long way
with the requests connector, almost anything that takes a URL and
returns XML or JSON can be queried with it.

----------------------------------------------------------------------
Normal_Inevitable880 · 2023-09-27 13:58

Yes, looks like a unique approach.

----------------------------------------------------------------------
CheckIcy7072 · 2023-09-27 14:03

Since you've mentioned LLMs, can you do RAG (Retrieval Augmented
Generation) on it?

----------------------------------------------------------------------
search_guy · 2023-09-27 16:03

Absolutely - fetch the top N results from your repository, compose a
prompt, then return insight + list of results.

----------------------------------------------------------------------
CheckIcy7072 · 2023-09-27 17:04

cool

----------------------------------------------------------------------
Butthurtz23 · 2023-09-28 00:44

Nice! I just ran this on Docker and I'm impressed. However, I  would
like to see persistent storage as an option for dockerized swirl.

----------------------------------------------------------------------
search_guy · 2023-09-28 13:51

>https://www.reddit.com/r/selfhosted/comments/16tiqga/ive\_created\_an
\_opensource\_search\_platform\_swirl/

Glad you like it! Storage is configured in the \`.env\` and
\`swirl\_server/settings.py\`, you can  modify it to use PostgreSQL
etc. Some info here: https://github.com/swirlai/swirl-
search/wiki/3.-Admin-Guide#configuring-django

----------------------------------------------------------------------
Butthurtz23 · 2023-09-28 23:30

I'm referring to the volume, see below for example:

\`\`\`
  swirl:
image: swirlai/swirl-search:latest
container\_name: swirl
ports:
\- "8000:8000"
volumes:
\- /local/path:/app/swirl\_server/
command: >
sh -c 'rm -fr ./.swirl && python swirl.py setup &&
/usr/bin/jq ".default" ./config-swirl-demo.db.json | sed -e
"s//$MSAL\_APP\_ID/" \\
\-e "s//$MSAL\_TENANT\_ID/" \\
\-e "s//$MSAL\_CB\_PORT/" \\
\-e "s//$MSAL\_HOST/" > static/api/config/default &&
python swirl.py start celery-worker celery-beats &&
daphne -b 0.0.0.0 -p 8000 swirl\_server.asgi:application'
\`\`\`

This way nothing get wiped out if I updated to the latest image