https://www.robinsloan.com/lab/what-are-we-even-doing-here/

Blog About Moonbound Shop

This is a post from Robin Sloan's lab blog & notebook. You can visit
the blog's homepage, or learn more about me.

Is the doc bot docs, or not?

July 8, 2025

Upgrading my Shopify email notification templates this morning,
I asked Shopify's LLM-powered developer documentation bot this ques
tion:

    What's the syntax, in Liquid, to detect whether an order in an
    email notification contains items that will be fulfilled through
    Shopify Collective?

I'd done a few traditional searches, but couldn't find an answer, so
I thought, okay, let's try the doc bot!

The reply came, quick and clean:

{% if order.tags contains 'Shopify Collective' %}
  <!-- The order contains at least one item fulfilled through Shopify Collective -->
  <p>Some items in your order are being fulfilled by a partner merchant through Shopify Collective.</p>
{% endif %}

Looks great. I added this code, but it didn't work; the condition
wasn't being satisfied, even when orders had the necessary tag, vis
ible on the admin screen.

Shopify doesn't provide a way to test unconventional email formats
without actually placing real orders, so I did my customary dance of
order-refund, order-refund, order-refund. My credit card is going to
get locked one of these days.

Turns out, the Shopify Collective tag isn't present on the order at
the time the confirmation email is generated. It's added later, pos
sibly just moments later, by some other cryptic Shopify process.

I don't think this is documented anywhere, so in that sense, it's not
surprising the documentation bot got it wrong. My question is: what's
the point of a doc bot that just takes a guess?

More pointedly: is the doc bot docs, or not?

And very pointedly indeed: what are we even doing here??

I've sent other queries to the doc bot and received helpful replies;
I'd categorize these basically as search: "How do I do X?" "What's
the syntax for Y?" But I believe this is a situation in which the
cost of bad advice outweighs the benefit of quick help by 10X, at
least. I can, in fact, figure out how to do X using the real docs.
Only the doc bot can make things up.

If it was Claude making this kind of mistake, I'd be annoyed but not
surprised. But this is Shopify's sanctioned helper! It waits twin
kling in the header of every page of the dev site. I suppose there
are domains in which just taking a guess is okay; is the official doc
umentation one of them?

I vote no, and I think a freestyling doc bot undermines the effort
and care of the folks at Shopify who work -- in my estimation very 
successfully -- to produce documentation that is thorough and
accurate.

P.S. In the end, I adapted the code from the section of the notifica
tion that lists the items in the order -- a bit circuitous, but it
works:

{% assign is_collective_order = false %}
{% for line_item in line_items %}
  {% if line_item.product.tags contains 'Shopify Collective' %}
    {% assign is_collective_order = true %}
  {% endif %}
{% endfor %}

The product-level Shopify Collective tag is available and reliable at
the time the notification is generated.

To the blog home page

I'm Robin Sloan, a writer, printer, & manufacturer. The best thing to
do here is sign up for my email newsletter:

[                    ] [Subscribe]
This website doesn't collect any information about you or your
reading. It aspires to the speed and privacy of the printed page.

Don't miss the colophon. Hony soyt qui mal pence