Introducing SuperDuperDB: An Open source framework that brings AI directly into your database 🔮 reddit / selfhosted ---------------------------------------------------------------------- Sevyten · 2023-11-20 07:53 **🔮 SuperDuperDB is a general-purpose AI development and deployment framework for integrating any AI models and APIs directly with your existing data infrastructure, including real-time inference, model training and vector search. SuperDuperDB is not another database, it "super-dupers" your existing preferred database.** SuperDuperDB eliminates the need for complex MLOps pipelines and specialized vector databases, enabling you to build end-to-end AI applications only using Python! * Generative AI & LLM-Chat * Vector Search * Standard Machine Learning Use-Cases (Classification, Segmentation, Recommendation etc.) * Highly custom AI use-cases involving ultra specialized models #### Key Features: * [**Deploy AI Models and APIs directly on your database**](https://docs.superduperdb.com/docs/docs/apply_models)**:** Integrate all your AI models and APIs directly with your database with a simple Python command in a single scalable deployment that is easy- to-manage, without the need for additional pre-processing steps, ETL or boilerplate code. * [**Streaming Infe**](https://docs.superduperdb.com/docs/docs/daemoni zing_models_with_listeners) No need for duplicating and migrating your data to additional specialized vector databases - turn your existing battle-tested datastore into a fully-fledged multi-modal vector-search database, including easy generation of vector embeddings and vector indexes of your data with preferred models and APIs.Highly custom AI use cases and workflows with specialized models. * [**Scalable Model Training**](https://docs.superduperdb.com/docs/doc s/training_models)**:** Train AI models on large, diverse datasets simply by query-defining your training data, while ensuring optimal performance and ease of management, utilizing in-build computational optimizations. * [**Model Chaining**](https://docs.superduperdb.com/docs/docs/linking _interdependent_models): Easily setup complex workflows by connecting models and APIs to work together in an interdependent and sequential manner. * [**Difficult Data-Types**](https://docs.superduperdb.com/docs/docs/e ncoding_special_data_types): Work directly with images, video, audio in your datastore, and any type which can be encoded as bytes in Python. * [**Feature Store**](https://docs.superduperdb.com/docs/docs/mongodb_ query_API#inserts)**:** Turn your database into a centralized repository for storing and managing inputs and outputs of AI models of arbitrary data-types, making them available in a structured format and known environment. * [**Vector Search**](https://docs.superduperdb.com/docs/docs/vector_search)**:** No need for duplicating and migrating your data to additional specialized vector databases - turn your existing battle-tested datastore into a fully-fledged multi-modal vector-search database, including easy generation of vector embeddings and vector indexes of your data with preferred models and APIs.Highly custom AI use-cases and workflows with specialized models. ​ https://preview.redd.it/zygfqdbmkg1c1.png?width=827&format=png&auto=we bp&s=950c5d89de4931329d2123c67de912af1ed90a3e ⭐️⭐️ ⭐️ Please leave us a star at [https://github.com/SuperDuperDB/su perduperdb](https://github.com/SuperDuperDB/superduperdb) ⭐️ ⭐️ ⭐️ ---------------------------------------------------------------------- RovingShroom · 2023-11-21 00:22 This seems pretty awesome! Most AI projects just seem like hype without any substance, but this seems really useful. I'll send this to a friend. My main concern with plugging it directly into a database would be security. What kind of precautions do you have? Also, do I have to give it access to my whole db or can I prevent access to certain tables like a users table? ---------------------------------------------------------------------- Educational_Set9947 · 2023-11-21 07:35 There is support for mapping outputs to dedicated model-output tables (non-production tables). We are working on also working on mapping outputs to other database/ data deployments. ---------------------------------------------------------------------- fazlulkarimweb · 2023-11-21 07:44 We're an open-source project, so our code is available for everyone to see and contribute to. If you're interested in learning more about how we handle database connections, you can check out our codebase on [Github](https://github.com/SuperDuperDB/superduperdb). As an extra layer of security, I'd recommend whitelisting your IP address in the database. This will help prevent unauthorized access to your data even if your password or URI is compromised. Thanks, and let us know if you have any other questions! ---------------------------------------------------------------------- Educational_Set9947 · 2023-11-21 08:49 That allows you to restrict access to production databases to be read- only. Then the written outputs can go into less critical systems. ---------------------------------------------------------------------- fnikolaidis · 2023-11-21 11:41 Thanks for your interest u/RovingShroom! Following the principle of least privilege, we require read-only access to the tables you want to "index". But the index must also be stored somewhere. One option is maintaining your database as read-only and storing the index externally, such as on your filesystem. Alternatively, you can establish a new table dedicated to housing the index. In this case, the requisite step would be granting us write access to that specific table (e.g `superduper_index`). For optimal security, consider creating a new `superduper` user, granting read-only access to your data tables, and read-write access to `superduper_index`. ---------------------------------------------------------------------- RovingShroom · 2023-11-21 22:32 So your \`superduper\` user could be denied read access on tables you don't need. Makes sense.