https://github.com/MinishLab/vicinity Skip to content Navigation Menu Toggle navigation Sign in * Product + GitHub Copilot Write better code with AI + Security Find and fix vulnerabilities + Actions Automate any workflow + Codespaces Instant dev environments + Issues Plan and track work + Code Review Manage code changes + Discussions Collaborate outside of code + Code Search Find more, search less Explore + All features + Documentation + GitHub Skills + Blog * Solutions By company size + Enterprises + Small and medium teams + Startups By use case + DevSecOps + DevOps + CI/CD + View all use cases By industry + Healthcare + Financial services + Manufacturing + Government + View all industries View all solutions * Resources Topics + AI + DevOps + Security + Software Development + View all Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Reseting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} MinishLab / vicinity Public generated from MinishLab/watertemplate * Notifications You must be signed in to change notification settings * Fork 0 * Star 45 Lightweight Nearest Neighbors with Flexible Backends minishlab.github.io/ License MIT license 45 stars 0 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 1 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights MinishLab/vicinity main BranchesTags [ ] Go to file Code Folders and files Last commit Last Name Name message commit date Latest commit History 34 Commits .github/workflows .github/workflows tests tests vicinity vicinity .gitignore .gitignore .pre-commit-config.yaml .pre-commit-config.yaml LICENSE LICENSE Makefile Makefile README.md README.md pyproject.toml pyproject.toml uv.lock uv.lock View all files Repository files navigation * README * MIT license Vicinity: Lightweight Nearest Neighbors Package version Supported Python versions Downloads Codecov License - MIT Quickstart * Main Features * Supported Backends * Installation Vicinity is a light-weight, low-dependency vector store. It provides a simple and intuitive interface for nearest neighbor search, with support for different backends and evaluation. There are many nearest neighbors packages and methods out there. However, we found it difficult to compare them. Every package has its own interface, quirks, and limitations, and learning a new package can be time-consuming. In addition to that, how do you effectively evaluate different packages? How do you know which one is the best for your use case? This is where Vicinity comes in. Instead of learning a new interface for each new package or backend, Vicinity provides a unified interface for all backends. This allows you to easily experiment with different indexing methods and distance metrics and choose the best one for your use case. Vicinity also provides a simple way to evaluate the performance of different backends, allowing you to measure the queries per second and recall. Quickstart Install the package with: pip install vicinity Optinally, install any of the supported backends, or simply install all of them with: pip install vicinity[all] The following code snippet demonstrates how to use Vicinity for nearest neighbor search: import numpy as np from vicinity import Vicinity from vicinity.datatypes import Backend, Metric # Create some dummy data items = ["triforce", "master sword", "hylian shield", "boomerang", "hookshot"] vectors = np.random.rand(len(items), 128) # Initialize the Vicinity instance (using the basic backend and cosine metric) vicinity = Vicinity.from_vectors_and_items(vectors=vectors, items=items, backend_type=Backend.BASIC, metric=Metric.COSINE) # Create a query vector query_vector = np.random.rand(128) # Query for nearest neighbors with a top-k search results = vicinity.query([query_vector], k=3) # Query for nearest neighbors with a threshold search results = vicinity.query_threshold([query_vector], threshold=0.9) Saving and loading a vector store: vicinity.save('my_vector_store') vicinity = Vicinity.load('my_vector_store') Evaluating a backend: # Use the first 1000 vectors as query vectors query_vectors = vectors[:1000] # Evaluate the Vicinity instance by measuring the queries per second and recall qps, recall = vicinity.evaluate( full_vectors=vectors, query_vectors=query_vectors, ) Main Features Vicinity provides the following features: * Lightweight: Minimal dependencies and fast performance. * Flexible Backend Support: Use different backends for vector storage and search. * Serialization: Save and load vector stores for persistence. * Evaluation: Easily evaluate the performance of different backends. * Easy to Use: Simple and intuitive API. Supported Backends The following backends are supported: * BASIC: A simple (exact matching) flat index for vector storage and search. * HNSW: Hierarchical Navigable Small World Graph (HNSW) for ANN search using hnswlib. * USEARCH: ANN search using Usearch. This uses a highly optimized version of the HNSW algorithm. * ANNOY: "Approximate Nearest Neighbors Oh Yeah" for approximate nearest neighbor search. * PYNNDescent: ANN search using PyNNDescent. * FAISS: All FAISS indexes are supported: + flat: Exact search. + ivf: Inverted file search. + hnsw: Hierarchical Navigable Small World Graph. + lsh: Locality Sensitive Hashing. + scalar: Scalar quantizer. + pq: Product Quantizer. + ivf_scalar: Inverted file search with scalar quantizer. + ivfpq: Inverted file search with product quantizer. + ivfpqr: Inverted file search with product quantizer and refinement. NOTE: the ANN backends do not support dynamic deletion. To delete items, you need to recreate the index. Insertion is supported in the following backends: FAISS, HNSW, and Usearch. The BASIC backend supports both insertion and deletion. Backend Parameters Backend Parameter Description Default Value Annoy metric Similarity metric to use (dot, "cosine" euclidean, cosine). trees Number of trees to use for 100 indexing. length Optional length of the dataset. None FAISS metric Similarity metric to use "cosine" (cosine, l2). Type of FAISS index (flat, ivf, index_type hnsw, lsh, scalar, pq, "hnsw" ivf_scalar, ivfpq, ivfpqr). nlist Number of cells for IVF 100 indexes. m Number of subquantizers for PQ 8 and HNSW indexes. nbits Number of bits for LSH and PQ 8 indexes. Number of bits for the refine_nbits refinement stage in IVFPQR 8 indexes. HNSW metric Similarity space to use "cosine" (cosine, l2). ef_construction Size of the dynamic list during 200 index construction. m Number of connections per 16 layer. PyNNDescent metric Similarity metric to use "cosine" (cosine, euclidean, manhattan). n_neighbors Number of neighbors to use for 15 search. Usearch metric Similarity metric to use (cos, "cos" ip, l2sq, hamming, tanimoto). connectivity Number of connections per node 16 in the graph. expansion_add Number of candidates considered 128 during graph construction. expansion_search Number of candidates considered 64 during search. Installation The following installation options are available: # Install the base package pip install vicinity # Install all backends pip install vicinity[all] # Install specific backends pip install vicinity[annoy] pip install vicinity[faiss] pip install vicinity[hnsw] pip install vicinity[pynndescent] pip install vicinity[usearch] License MIT About Lightweight Nearest Neighbors with Flexible Backends minishlab.github.io/ Topics python ai embeddings annoy faiss hnsw hnswlib vector-database Resources Readme License MIT license Activity Custom properties Stars 45 stars Watchers 1 watching Forks 0 forks Report repository Releases 4 v0.3.0 Latest Dec 1, 2024 + 3 releases Packages 0 No packages published Contributors 2 * @Pringled Pringled Thomas van Dongen * @stephantul stephantul Stephan Tulkens Languages * Python 99.5% * Makefile 0.5% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.