https://github.com/neuml/txtai Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Blog * Solutions + For + Enterprise + Teams + Startups + Education + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} neuml / txtai Public * Notifications * Fork 282 * Star 3.3k Build AI-powered semantic search applications neuml.github.io/txtai License Apache-2.0 license 3.3k stars 282 forks Star Notifications * Code * Issues 12 * Pull requests 0 * Actions * Security * Insights More * Code * Issues * Pull requests * Actions * Security * Insights neuml/txtai This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 2 branches 28 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/n] Use Git or checkout with SVN using the web URL. [gh repo clone neuml/] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @davidmezzetti davidmezzetti Update notebook ... 88fb779 Feb 2, 2023 Update notebook 88fb779 Git stats * 950 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows docker docs examples models src/python/txtai test/python .coveragerc .gitignore .pre-commit-config.yaml .pylintrc CITATION.cff LICENSE Makefile README.md apps.jpg demo.gif logo.png mkdocs.yml pyproject.toml setup.py View code [ ] Build AI-powered semantic search applications Why txtai? Installation Examples Semantic Search Pipelines Workflows Model Training Applications Documentation Further Reading Contributing README.md [logo] Build AI-powered semantic search applications Version GitHub last commit GitHub issues Join Slack Build Status Coverage Status --------------------------------------------------------------------- txtai executes machine-learning workflows to transform data and build AI-powered semantic search applications. demo Traditional search systems use keywords to find data. Semantic search applications have an understanding of natural language and identify results that have the same meaning, not necessarily the same keywords. search search Backed by state-of-the-art machine learning models, data is transformed into vector representations for search (also known as embeddings). Innovation is happening at a rapid pace, models can understand concepts in documents, audio, images and video. Summary of txtai features: * Large-scale similarity search with multiple index backends ( Faiss, Annoy, Hnswlib) and support for external vector databases * Create embeddings for text snippets, documents, audio, images and video * Machine-learning pipelines that run question-answering, labeling, transcription, translation, summarization, LLM prompts and more * -[?][?] Workflows to join pipelines together and aggregate business logic. txtai processes can be microservices or full-fledged indexing workflows. * [?][?] Build with Python or YAML. API bindings available for JavaScript, Java, Rust and Go. * [?] Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes) Applications range from similarity search to NLP-driven data extractions that generate structured data. Semantic workflows transform and find data driven by user intent. flows flows The following applications are powered by txtai. apps Application Description paperai Semantic search and workflows for medical/scientific papers codequestion Semantic search for developers tldrstory Semantic search for headlines and story text neuspo Fact-driven, real-time sports event and news site txtai is built with Python 3.7+, Hugging Face Transformers, Sentence Transformers and FastAPI Why txtai? why why In addition to traditional search systems, a growing number of semantic search solutions are available, so why txtai? * Up and running in minutes with pip or Docker # Get started in a couple lines from txtai.embeddings import Embeddings embeddings = Embeddings({"path": "sentence-transformers/all-MiniLM-L6-v2"}) embeddings.index([(0, "Correct", None), (1, "Not what we hoped", None)]) embeddings.search("positive", 1) #[(0, 0.2986203730106354)] * Build applications in your programming language of choice via the API # app.yml embeddings: path: sentence-transformers/all-MiniLM-L6-v2 CONFIG=app.yml uvicorn "txtai.api:app" curl -X GET "http://localhost:8000/search?query=positive" * Connect machine learning models together to build intelligent data processing workflows * Works with both small and big data - scale when needed * Supports micromodels all the way up to large language models (LLMs) * Low footprint - install additional dependencies when you need them * Learn by example - notebooks cover all available functionality Installation install install The easiest way to install is via pip and PyPI pip install txtai Python 3.7+ is supported. Using a Python virtual environment is recommended. See the detailed install instructions for more information covering optional dependencies, environment specific prerequisites, installing from source, conda support and how to run with containers. Examples examples examples The examples directory has a series of notebooks and applications giving an overview of txtai. See the sections below. Semantic Search Build semantic/similarity/vector/neural search applications. Notebook Description Overview of the functionality Open Introducing txtai >[?] provided by txtai In Colab Build an Embeddings index Index and search Hugging Face Open with Hugging Face Datasets In Datasets Colab Build an Embeddings index Index and search a data source with Open from a data source word embeddings In Colab Add semantic search to Add semantic search to existing Open Elasticsearch search systems In Colab Similarity search with Embed images and text into the same Open images space for search In Colab Distributed embeddings Distribute an embeddings index Open cluster across multiple data nodes In Colab Content storage, SQL, object Open What's new in txtai 4.0 storage, reindex and compressed In indexes Colab Deep dive into the file formats Open Anatomy of a txtai index behind a txtai embeddings index In Colab Custom Embeddings SQL Add user-defined functions to Open functions Embeddings SQL In Colab Open Model explainability Explainability for semantic search In Colab Domain-specific natural language Open Query translation queries with query translation In Colab Question matching with semantic Open Build a QA database search In Colab Composable search with vector, SQL Open Embeddings components and scoring components In Colab Explore topics, data connectivity Open Semantic Graphs and run network analysis In Colab Topic modeling backed by a BM25 Open Topic Modeling with BM25 index In Colab Prompt-driven search with Embeddings-guided and Prompt-driven Open LLMs search with Large Language Models In (LLMs) Colab Pipelines Transform data with NLP-backed pipelines. Notebook Description Extractive QA with Introduction to extractive Open In txtai question-answering with txtai Colab Extractive QA with Run extractive question-answering Open In Elasticsearch queries with Elasticsearch Colab Extractive QA to build Build structured datasets using Open In structured data extractive question-answering Colab Apply labels with zero Use zero shot learning for labeling, Open In shot classification classification and topic modeling Colab Building abstractive Run abstractive text summarization Open In text summaries Colab Extract text from Extract text from PDF, Office, HTML Open In documents and more Colab Text to speech Generate speech from text Open In generation Colab Transcribe audio to Convert audio files to text Open In text Colab Translate text between Streamline machine translation and Open In languages language detection Colab Generate image Captions and object detection for Open In captions and detect images Colab objects Near duplicate image Identify duplicate and near-duplicate Open In detection images Colab API Gallery Using txtai in JavaScript, Java, Rust Open In and Go Colab Workflows Efficiently process data at scale. Notebook Description Run pipeline workflows >[?] Simple yet powerful constructs to Open In efficiently process data Colab Transform tabular data with Transform, index and search Open In composable workflows tabular data Colab Tensor workflows Performant processing of large Open In tensor arrays Colab Entity extraction workflows Identify entity/label Open In combinations Colab Workflow Scheduling Schedule workflows with cron Open In expressions Colab Push notifications with Generate and push notifications Open In workflows with workflows Colab Pictures are a worth a Generate webpage summary images Open In thousand words with DALL-E mini Colab Run txtai with native code Execute workflows in native code Open In with the Python C API Colab Model Training Train NLP models. Notebook Description Train a text labeler Build text sequence classification Open In models Colab Train without labels Use zero-shot classifiers to train Open In new models Colab Train a QA model Build and fine-tune Open In question-answering models Colab Train a language model Build new language models Open In from scratch Colab Export and run models Export models with ONNX, run Open In with ONNX natively in JavaScript, Java and Colab Rust Export and run other Export and run models from Open In machine learning models scikit-learn, PyTorch and more Colab Applications Series of example applications with txtai. Links to hosted versions on Hugging Face Spaces also provided. Application Description Basic Basic similarity search example. Data from the similarity original txtai demo. search Book similarity search application. Index book Local Book search descriptions and query using natural language run statements. only Image Image similarity search application. Index a search directory of images and run searches to identify images similar to the input query. Summarize Summarize an article. Workflow that extracts text an article from a webpage and builds a summary. Wiki search Wikipedia search application. Queries Wikipedia API and summarizes the top result. Build and execute txtai workflows. Connect Workflow summarization, text extraction, transcription, builder translation and similarity search pipelines together to run unified workflows. Documentation Full documentation on txtai including configuration settings for pipelines, workflows, indexing and the API. Further Reading further * Introducing txtai, AI-powered semantic search built on Transformers * Tutorial series on Hashnode | dev.to * What's new in txtai 5.0 | 4.0 * Getting started with semantic search * Run machine-learning workflows to transform data and build AI-powered semantic search applications with txtai * Semantic search on the cheap * Serverless vector search with txtai * Insights from the txtai console Contributing For those who would like to contribute to txtai, please see this guide. About Build AI-powered semantic search applications neuml.github.io/txtai Topics python search nlp api machine-learning deep-learning microservice cloud-native image-search semantic-search machine-learning-pipelines similarity-search video-search document-search audio-search vector-search machine-learning-workflows contextual-search neural-search txtai Resources Readme License Apache-2.0 license Stars 3.3k stars Watchers 58 watching Forks 282 forks Releases 28 v5.2.0 Latest Dec 20, 2022 + 27 releases Packages 0 No packages published Used by 62 * @cox-j * @gowindarajanTeBS * @muazhari * @kingsdigitallab * @muazhari * @trueblood * @avshrudai1 * @hsm207 + 54 Contributors 9 * @davidmezzetti * @csheargm * @0xflotus * @adin786 * @0206pdh * @lipusz * @hsm207 * @csnelsonchu * @hi019 Languages * Python 99.4% * Other 0.6% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.