[HN Gopher] Show HN: Phospho - Text Analytics for LLM Apps (Post...
       ___________________________________________________________________
        
       Show HN: Phospho - Text Analytics for LLM Apps (Posthog for
       Prompts)
        
       Hello HN! Pierre and Paul here. We are building an open source text
       analytics tool for user inputs and LLM app outputs The repo is
       https://github.com/phospho-app/phospho and landing is
       https://phospho.ai  Most people building with LLMs today don't have
       quantified evaluation and usage metrics on the interactions between
       users and their product. The only solution is to read every message
       (or a sample) to get a sense of what is going on. You can't improve
       your product without understanding who your users are and how they
       are using it. Nobody would launch a website without standard
       analytics today; the same principle should apply to LLM products.
       We made phospho to analyze the large amounts of text from user
       inputs and LLM app outputs, and give you quantified and actionable
       insights.  You first log messages and set up semantic events. Eg:
       "user is talking about sports", "assistant didn't quote the
       source". We then run asynchronous jobs to detect if events are
       present in the text or not. To do so, we use GPT-4 for the first
       few events, and then downsize to smaller fine-tuned models (cheaper
       & faster). It works with any LLM provider (OpenAI, Mistral,
       Ollama...). No proxy, no monkey patch, and no OpenAI key needed.
       You can link phospho to your users' feedback, and even use the
       platform to annotate some messages yourself. This helps you design
       step by step a custom evaluation pipeline that runs automatically,
       fits your needs, and enables you to iterate.  Results are available
       in dashboards, as dataframes, or via API. You can also directly
       leverage the events in your app to trigger actions in real-time
       with the API or via webhooks.  Deploy everything with Docker, or
       use the hosted cloud version. We have Python/Javascript SDK and an
       API. License is Apache 2.0.  Give it a spin and see where we're at:
       https://github.com/phospho-app/phospho We're interested in both
       feature requests and roasts. Let us know what you think!
        
       Author : PL_Venard
       Score  : 34 points
       Date   : 2024-03-13 15:14 UTC (7 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | mottobotto wrote:
       | I feel like as devs there is an omerta on the real performances
       | of AI apps.Cherry picked twitter screens get social media
       | highlights and money flowing into the field, but we all know that
       | in practice it delivers on 1% of the value promised.
       | 
       | Unsure how much of it relies on the models themselves, and how
       | much on the fact that optimizing prompts for a propper signal is
       | so f* hard
       | 
       | I hope more tools will make this easier. Will take a look at
       | phospho, thanks man
        
       | tzm wrote:
       | Very cool project with so much potential. Will follow along.
       | Thanks for sharing.
        
       | Oras wrote:
       | How does it compare to portkey which is backed by YC too?
        
         | PLBjt wrote:
         | Portkey is a proxy, so you will have to call their API, and
         | they will route the request to a LLM provider. Then, they
         | provide you with some observability tools to look at these
         | requests -> LLM router + observability phospho is not a proxy,
         | you log the messages and feedbacks to our API. We then extract
         | insights and detect events inside the text -> text and product
         | analytics
        
       ___________________________________________________________________
       (page generated 2024-03-13 23:00 UTC)