https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/

Simon Willison's Weblog

Subscribe

Pelicans on a bicycle. I decided to roll out my own LLM benchmark:
how well can different models render an SVG of a pelican riding a
bicycle?

I chose that because a) I like pelicans and b) I'm pretty sure there
aren't any pelican on a bicycle SVG files floating around (yet) that
might have already been sucked into the training data.

My prompt:

    Generate an SVG of a pelican riding a bicycle

I've run it through 16 models so far - from OpenAI, Anthropic, Google
Gemini and Meta (Llama running on Cerebras), all using my LLM CLI
utility. Here's my (Claude assisted) Bash script: generate-svgs.sh

Here's Claude 3.5 Sonnet (2024-06-20) and Claude 3.5 Sonnet
(2024-10-22):

[claude-3-5] [claude-3-5]

Gemini 1.5 Flash 001 and Gemini 1.5 Flash 002:

[gemini-1] [gemini-1]

GPT-4o mini and GPT-4o:

[gpt-4o-min] [gpt-4o]

o1-mini and o1-preview:

[o1-mini] [o1-preview]

Cerebras Llama 3.1 70B and Llama 3.1 8B:

[cerebras-l] [cerebras-l]

And a special mention for Gemini 1.5 Flash 8B:

[gemini-1]

The rest of them are linked from the README.

Posted 25th October 2024 at 11:56 pm

Recent articles

  * Gemini 2.0 Flash: An outstanding multi-modal LLM with a sci-fi
    streaming mode - 11th December 2024
  * ChatGPT Canvas can make API requests now, but it's complicated -
    10th December 2024
  * I can now run a GPT-4 class model on my laptop - 9th December
    2024

svg 38 ai 966 openai 225 generative-ai 824 llama 64 llms 818 llm 118 
anthropic 100 gemini 47 cerebras 6 pelican-riding-a-bicycle 9

  * Colophon
  * (c)
  * 2002
  * 2003
  * 2004
  * 2005
  * 2006
  * 2007
  * 2008
  * 2009
  * 2010
  * 2011
  * 2012
  * 2013
  * 2014
  * 2015
  * 2016
  * 2017
  * 2018
  * 2019
  * 2020
  * 2021
  * 2022
  * 2023
  * 2024