[HN Gopher] The HTTP Query Method
___________________________________________________________________
The HTTP Query Method
Author : todsacerdoti
Score : 97 points
Date : 2024-09-16 04:36 UTC (18 hours ago)
(HTM) web link (www.ietf.org)
(TXT) w3m dump (www.ietf.org)
| MuffinFlavored wrote:
| A simple query with a direct response: QUERY
| /contacts HTTP/1.1 Host: example.org Content-
| Type: example/query Accept: text/csv
| select surname, givenname, email limit 10
|
| Not quite full SQL (no JOIN or WHERE in any examples I see)
|
| Hmm... as long as you handle authentication/authorization
| correctly, why is it bad?
|
| It's a way to pluck certain JSON fields that were otherwise going
| to be returned? Kind of like one of the benefits of GraphQL? Will
| this catch on?
| SahAssar wrote:
| The request body is just an example, this does not specify the
| query language, just the http method.
| cjpearson wrote:
| I think it would be a good idea to remove these examples from
| the draft as it gives people the impression that this method is
| SQL-over-HTTP. A more typical body might be a JSON object with
| various filter criteria.
| pimlottc wrote:
| The WHERE is implied by the endpoint ("contacts")
| MuffinFlavored wrote:
| That would be the FROM in my opinion. WHERE would be ?
| filters, etc.
| pimlottc wrote:
| Oops, what am I thinking, you're right
| jessekv wrote:
| This paragraph is the key:
|
| > The QUERY method provides a solution that spans the gap between
| the use of GET and POST. As with POST, the input to the query
| operation is passed along within the content of the request
| rather than as part of the request URI. Unlike POST, however, the
| method is explicitly safe and idempotent, allowing functions like
| caching and automatic retries to operate.
| TYPE_FASTER wrote:
| Yes. I was skeptical at first. Why add another method? This
| explains why.
| solardev wrote:
| GraphQL reads are like that. You POST a query with the
| content inside the request body. This would be a nicer verb
| for it.
| ako wrote:
| OData provides GraphQL functionality but using GET + URL
| parameters, for large queries it's really hard to read or
| edit. Using the request body makes sense for readability,
| so OData also offers the option to query with POST +
| request body.
| nonethewiser wrote:
| But doesn't this beg the question: why not GET with query
| params?
|
| I'm not necessarily against it - I've had the urge to send a
| body on a GET request before (cant recall or justify the use-
| case, however).
|
| Reasons I can think of:
|
| - browsers limit url (and therefore) query param size
|
| - general dev ergonomics
|
| The practical limits (ie, not standard specified limits) on
| query param size seems fair but its worth mentioning there are
| practical limits to body size as well introduced by web
| servers, proxies, cloud api gateways, etc or ultimately
| hardware. From what I can see this is something like 2kb for
| query params in the lower end case (Chrome's 2083 char limit)
| and 10-100mb (but highly variable and potentially only hardware
| bound) for body size.
|
| In both cases it's worth stating that the spec is being
| informed by practical realities outside of the spec. In terms
| of the spec itself, there is no limit to either body size or
| query param length. How much should the spec be determined by
| particular implementation details of browsers, cloud services,
| etc.?
| nine_k wrote:
| Size can be a very practical concern if your query parameter
| is e.g. a picture.
|
| Not showing the URL in various logs can be a concern if your
| query parameters are sensitive.
| nonethewiser wrote:
| A picture as an input for a GET seems like a very strange
| use-case. Can you elaborate?
|
| I do agree on the quirkiness of encoding a bunch of data in
| the URL. Its nice to decouple the endpoint from the input
| at a more fundamental level.
| bakje wrote:
| Use cases like reverse image search come to mind
| treve wrote:
| With a request body you can also rely on all the other
| standard semantics that request bodies afford, such as
| different content-types, content encoding. Query parameters
| are also often considered less secure for things like PII and
| (less important these days) query parameters don't really
| define character encoding.
|
| But generally the most important reason is that you can take
| advantage of the full range of mimetypes, and yes
| practicality speaking there's a limit on how much you should
| stuff in a query parameter.
|
| This has resulted in tons of protocols using POST, such as
| GraphQL. This is a nice middle ground.
| chrsig wrote:
| Servers have to limit header size as well, since content-
| length isn't available. Without a bounded header size the
| client can just send indefinitely.
|
| Also just impossible to read, and url encoding is full of
| parser alignment issues. I have to imagine that QUERY would
| support a json request body.
| MBCook wrote:
| This avoid changing the definition of GET. Who knows how many
| middle boxes would mess with things if you did that because
| they "know" what GET means and so their thing is "safe".
|
| Until GET changes.
|
| People aren't using QUERY yet so that problem doesn't exist.
| cryptonector wrote:
| Reasons: - URI length limits (not just
| browsers, but also things like Jersey) -
| q-params are ugly and limiting by comparison to an
| arbitrarily large and complex request body
| with a MIME type
|
| However, q-params are _super_ convenient because the query is
| then part of the URI, which you can then cut-n-paste around.
| Of course, a convenient middle of the road is to use a QUERY
| which returns not just query results but _also_ a URI with
| the query expressed as q-params so you can cut-n-paste it.
| TacticalCoder wrote:
| > Unlike POST, however, the method is explicitly safe and
| idempotent, allowing functions like caching and automatic
| retries to operate.
|
| A shitload of answers to GET requests, although cacheable, are
| stale though. If you issue a GET for a page which contains
| stuff like _" Number of views: xxx"_ or _" Number of users
| logged in: yyy"_ or _" Last updated on: yyyy-mm-dd"_ there goes
| idempotency.
|
| _Some_ GET requests are actually idempotent ( "Give me the
| value of AAPL at close on 2021-07-21") but many aren't.
|
| Stale data won't break much but there's a world between "it's
| an idempotent call" and "I'm actually likely to be seeing stale
| data if I'm using a cached value".
|
| I mean... Enter in you browser "https://example.org", that's a
| GET. And that's definitely not idempotent for most sites we're
| all using on a daily basis.
| stackghost wrote:
| I suppose the counterargument is that "users logged in" or
| "views" is stale all the time, because it could have changed
| while the response was in flight.
|
| If you really need live data for "views" or what have you,
| then perhaps the front end should be querying the backend
| repeatedly via a separate POST.
| Timon3 wrote:
| Idempotency in the context of HTTP requests isn't about the
| response you receive, but about the state of the resource on
| the server. You're supposed to be able to call _GET
| /{resource}/{id}_ any amount of times without the resource
| _{id}_ being different at the end. You can 't do the same for
| POST, as _POST /{resource}_ could create a new entity with
| every call.
|
| A view counter also doesn't break this, as the view counter
| isn't the resource you're interacting with. As long as you're
| not modifying the resource on a `GET` call, the request is
| idempotent.
| kbolino wrote:
| Strictly speaking, an _inline_ view counter, which is
| incremented on each GET and included in the response, would
| break idempotence. Similarly, a PUT request that implicitly
| modifies a "last updated" field would also break
| idempotence. These are pretty minor violations, though,
| which arguably don't change any "semantics" of the
| response.
| Timon3 wrote:
| If the view counter is part of the resource itself, then
| yes, incrementing it on GET breaks the idempotence
| contract - but it should be obvious that by breaking
| idempotence, you're breaking idempotence. If it's part of
| the response without being part of the resource, you're
| not breaking idempotence - otherwise things like updating
| a JS library would also break idempotence, but it
| doesn't, since it's not part of the resource.
| kbolino wrote:
| Idempotence does not mean immutability, it means that two or
| more identical operations have the same effect on the
| resource as a single one. Since GET operations, by virtue of
| also being safe, generally have no effect at all, this is
| almost always trivially true. Just because the resource's
| content changed _for some other reason_ doesn 't mean GET is
| not idempotent.
| paulddraper wrote:
| Correct.
|
| GET, DELETE, HEAD, OPTIONS, PUT are idempotent.
|
| POST is not. (Thus existence of Idempotency-Key, etc)
| cryptonym wrote:
| Can be useful for some APIs.
|
| The use-case 4.2 with both Content-Location and Location feels
| weird to me. Not sure you would want multiple urls with different
| meanings. Isn't it harder to keep it idempotent if we are
| generating URLs for request and result? Not sure Location is
| generally meaningful in 200... that may impact other RFC.
|
| Could be interesting to see a sample where query is created but
| result will be available later. That's probably just the 303 See
| Other with a Retry-After?
| pgris2 wrote:
| I don't like using the body. GET's are easily shareable,
| bookmarkeable, etc, (even editable by humans). etc. thans to
| query strings. I rather have a GET with better (shorter)
| serialization for parameters and a standarized max-length of 16k
| or something like that.
| imbnwa wrote:
| They discuss this desire in the draft:
|
| >A server MAY create or locate a resource that identifies the
| query operation for future use. If the server does so, the URI
| of the resource can be included in the Location header field of
| the response (see Section 10.2.2 of [HTTP]). This represents a
| claim that a client can send a GET request to the indicated URI
| to repeat the query operation just performed without resending
| the query parameters.
| lokar wrote:
| HTTP is used in many contexts where no human will see the
| request
| westurner wrote:
| There is currently no way to determine whether there's, for
| example, an application/json representation of a URL that returns
| text/html by default.
|
| OPTIONS and PROPFIND don't get it.
|
| There should be an HTTP method (or a .well-known url path prefix)
| to query and list every Content-Type available for a given URL.
|
| From https://x.com/westurner/status/1111000098174050304 :
|
| > _So, given the current web standards, it 's still not possible
| to determine what's lurking behind a URL given the correct
| headers? Seems like that could've been the first task for
| structured data on the internet_
| tarasglek wrote:
| Yeah no way to ask for openapi schema definition.
| treve wrote:
| Servers can return something like: Link:
| </foo>; rel="alternate" type="application/json"
|
| You could return multiple of these as a response to a HEAD
| request for example.
|
| You could also use the Accept header in response to a HTTP
| OPTIONS request, or as part of a 415 error response.
| Accept: application/json, text/html
|
| https://httpwg.org/specs/rfc9110.html#field.accept (yes Accept
| can be used in responses)
|
| .well-known is not a good place for this kind of thing. That
| discovery mechanism should only be used in situations where
| only a domainname is known, and a feature or endpoint needs to
| be discovered for a full domain. In most cases you want a link.
|
| The building blocks for most of this stuff is certainly there.
| There's a lot of wheels being reinvented all the time.
| westurner wrote:
| > _https://httpwg.org/specs/rfc9110.html#field.accept (yes
| Accept can be used in responses)_
|
| I think that would solve.
|
| HTTP servers SHOULD send one or more Accept: {content_type}
| HTTP headers in the HTTP Response to an HTTP OPTIONS Request
| in order to indicate which content types can be requested
| from that path.
| mg wrote:
| The first example shows a GET request and states that it is
| "common and interoperable query". But that "if the query
| parameters extend to several kilobytes or more of data it may not
| be, because many implementations place limits on their size".
|
| So how about simply removing those limits on GET requests?
| pimlottc wrote:
| The installed base of HTTP processing devices is ridiculously
| huge. There is no practical way to change the existing behavior
| of GET.
| IncreasePosts wrote:
| There are thousands of pieces of software which have those
| limitations built in intentionally or unintentionally though.
| Using a new verb avoids that issue
| packetlost wrote:
| I wonder how caching will work when the body has the potential to
| be huge? You could hash it to save on RAM, but I imagine that
| would get "expensive" at scale.
| cryptonym wrote:
| A robust cache implementation would probably put a limit on
| request content-length, either denying the request or
| forwarding it as-is and bypassing cache.
| packetlost wrote:
| I mean, yeah? The standard does say *MAY* cache content. It
| also says the cache server should try to do semantic
| interpretation of the body if possible, which I think is a
| misstep. IMO it would be better to have canonical
| representations of whatever format you're using and then let
| the cache layer rely on hashing or similar mechanisms. More
| complicated edge servers/proxies is _not_ what we want
| because it leads to vulnerabilities.
| nine_k wrote:
| Huge as in 50 kB? Fine. Huge as in 50 MB? Well, pass it through
| maybe if your NVMe device is running out of space.
| paulddraper wrote:
| https://en.wikipedia.org/wiki/Hash_function
| thrance wrote:
| I often find myself creating endpoints like `POST
| /<resource>/query`, which should absolutely be a GET if they
| could have bodies. Why add yet more to the web specs when we
| could get rid of the "no body" exception on GET?
| nonethewiser wrote:
| > I often find myself creating endpoints like `POST
| /<resource>/query`
|
| Why?
| k__ wrote:
| So, QUERY is a POST but with GET semantics?
|
| Sounds like GraphQL won.
| marcosdumay wrote:
| It's way more than GraphQL.
|
| Searching and analyzing large things is a real need. It won't
| go away by changing your protocol.
| paulddraper wrote:
| GraphQL queries are a specific usecase for HTTP QUERY.
|
| (Today they rely on HTTP POST.)
| dehrmann wrote:
| Years ago, I had an interview question that was essentially
| "model a restful query API." I'm not sure if it was the gotcha or
| that it was a bad question, but rest isn't a good fit. GET
| lacking a request body doesn't help, and except for long-running
| queries, modeling things as resources doesn't work. You can
| either design an API that isn't very restful or do confusing
| things to squeeze it into that box.
| the_arun wrote:
| For taking advantage of `Cache-Control` don't we need the URI to
| be unique? For eg. /Contacts?id=abcd with graphql query in the
| body? the id can be a hash/signature of query string?
| treve wrote:
| No, the key is not just the URI. It's potentially the URI +
| method + Some request headers + some response headers. The Vary
| header lets you control which request headers should be
| considered.
|
| This method specifically includes the request body in that key.
| giobox wrote:
| I was just working with OpenSearch's API recently, who sort of
| abuse the semantics of the HTTP spec by using GETs with a body to
| perform search queries, largely to solve a similar problem. A
| "QUERY" message type would probably easily replace the GET with a
| body used by OpenSearch etc. I'd go as far as to argue thats
| largely what this new QUERY type is - official recognition for a
| "GET" request with a body.
|
| > https://opensearch.org/docs/latest/api-reference/search/
| BugsJustFindMe wrote:
| It's not abuse of the spec. The spec says that GET is allowed
| to have a body. Whether an endpoint must honor it is left
| undefined, and this has lead people to believe wrongly that
| that means endpoints are not allowed to expect it. But
| certainly the endpoint itself defines what it does with your
| requests, and that's what actually matters.
|
| QUERY is just GET once more with feeling, because people are
| worried about existing software that made the wrong decision
| about denying or stripping GET bodies and would need to be
| updated to allow them through, and detecting an unimplemented
| verb is I guess maybe simpler than detecting some middle layer
| being a dick and dropping your data.
| treve wrote:
| This is not a correct interpretation of the spec, as far as I
| can tell. I did some more research on this + primary sources
| on my blog if you're interested in why and why people are
| confused about this:
|
| https://evertpot.com/get-request-bodies/
| divbzero wrote:
| Related:
|
| _Why an HTTP Get Request Shouldn't Have a Body_ -
| https://www.baeldung.com/cs/http-get-with-body - July 2024
| jkrems wrote:
| > When doing so, caches SHOULD first normalize request content to
| remove semantically insignificant differences, thereby improving
| cache efficiency, by: [...]
|
| That part sounds like it's asking for trouble. I'm curious if
| this will make it to the final draft. If the client mis-
| identifies which parts of the request body are semantically
| insignificant, the result would be immediate cache poisoning and
| fun hard-to-debug bugs.
|
| If it's meant as a "MAY", then that seems kind of meaningless: If
| the client for some reason knows that one particular aspect of
| the request body is insignificant, it could just generate request
| bodies that are normalized in the first place..?
| orf wrote:
| Yeah, that's nuts and is obviously flawed behaviour that can
| interact poorly with any number of things - not least of all
| any kind of checksums within the response.
|
| I'm surprised to see that in a RFC.
|
| Edit: it's only for the cache key:
|
| > Note that any such normalization is performed solely for the
| purpose of generating a cache key; it does not change the
| request itself.
|
| Still super dangerous.
|
| Edit edit: I just typed out a long message on the GitHub issue
| tracker for this, but submitting it errored and I've lost all
| the content. Urgh
| cryptonector wrote:
| Request body normalization is not really feasible, not without
| having the normalization function be specified by the request
| body's MIME type, and MIME types specify no such thing.
| Besides, normalization of things like JSON is fiendishly
| tricky, and may not be feasible in the case of many MIME types.
| IMO this should be removed from the spec.
|
| Instead the _server_ should normalize _if it can and wants to_
| , and the resulting URI should be used by the cache. The 3xx
| approach might work well for this, whereas having the server
| immediately assign a Content-Location: URI as I propose
| elsewhere in this thread would not allow for server-side
| normalization in time for a cache.
| apitman wrote:
| Looks interesting. Sadly it will likely be hamstring by CORS
| rules. When designing an API, I frequently send all non-GET
| requests as POSTs with content type text/plain in order for them
| to classify as simple requests and avoid CORS preflights, which
| add an entire round trip per request. Obviously only safe if
| you're doing proper authorization with a token or something.
| Another fun bit is that you have to put the token in the query
| string, going against best practices for things like OAuth2,
| because the Authorization header isn't CORS approved. CORS
| enforcement is an abomination.
| cryptonector wrote:
| Because of the semantics of QUERY CORS rules should apply as to
| GETs -- one would think anyways. The Internet-Draft ought to
| say something about CORS though, that's for sure.
| mtrovo wrote:
| I can't stop thinking how HTTP REST is just an abuse on the
| original design of HTTP and the whole thing of using HTTP verbs
| to add meaning to requests is not a very good abstraction for RPC
| calls.
|
| The web evolved from being a tool to access documents on
| directories to this whole apps in the cloud thing and we kept
| using the same tree abstraction to sync state to the server,
| which doesn't make a lot of sense in lots of places.
|
| Maybe we need a better abstraction to begin with, something like
| discoverable native RPCs using a protocol designed for it like
| thrift or grpc.
| TZubiri wrote:
| En.wikipedia.org/wiki/constitutionalism
| jarjoura wrote:
| HTTP as a pure transport protocol keeps coming back to the
| default, because it works. Its superpower is that it pushes
| stateless (and secure) design from end to end. You have to
| fight a losing battle to get around that, so if you play along,
| you can end up with better power efficiency, resiliency and
| scalability.
|
| REST is just very simple to understand and easy to prototype
| with. There's better abstractions on top of HTTP, like GraphQL
| and gRPC (as you mentioned), but you can layer those on after
| you have a working solution and are looking for more
| performance.
|
| HTTP3 is on the way this decade and I'm excited for its
| promises. Given how long it took for HTTP2 to standardize, I'm
| not optimistic it will be soon, but it does mean we have a path
| forward.
| Two4 wrote:
| HTTP2 has already been mostly replaced by HTTP3, iirc. We now
| mostly have a split between HTTP1.1 And 3, if I'm remembering
| an article I read correctly.
| cryptonector wrote:
| > The web evolved from being a tool to access documents on
| directories to this whole apps in the cloud thing and we kept
| using the same tree abstraction to sync state to the server,
| which doesn't make a lot of sense in lots of places.
|
| The first part is correct, but hard disagree on the rest. HTTP
| makes a lot of sense for RPC-ish things because a) it can do
| those things better than RPC, b) HTTP can do things that RPC
| typically can't (like: content type negotiations and
| conversions, caching, online / indefinite length content
| transmission, etc).
|
| HTTP is basically a filesystem access protocol with extra
| semantics made possible by a) headers, b) MIME types, and if
| you think of some "files" as active/dynamic resources rather
| than static resources, then presto, you have a pretty good
| understanding of HTTP. ("Dynamic" means code processes a
| request and produces response content, possibly altering state,
| that a plain filesystem can't possibly do. An RDBMS is an
| example of "dynamic", while a filesystem is an example of
| "static".)
|
| REST is quite fine. It's very nice actually, and much nicer
| than RPC. And everything that's nice about REST and not nice
| about RPC is to do with those extensible headers and MIME
| types, and especially semantics and cache controls.
|
| But devs always want to generate APIs from IDLs, so RPC is
| always so tempting.
|
| As for RPC, there's nothing terribly wrong with it _as long as_
| one can generate async-capable APIs from IDLs. The thing
| everyone hates about RPC is that typically it gets synchronous
| interfaces for distributed computations, which is a mistake.
| But RPC _protocols_ do not imply synchronous interfaces -- that
| 's just a mistake in the codegen tools designs.
|
| Ultimately the non-RESTful thing about RPCs that sucks is that
| - no URIs (see below response to your point about
| discovery) - nothing is exposed about RPC
| idempotence, which makes "routing" fraught
| - lack of 3xx redirects, which makes "routing" hard
| - lack of cache controls - lack of online
| streaming ("chunked" encoding w/ indefinite
| content-length)
|
| Conversely, the things that make HTTP/REST good are:
| - URIs!! - idempotence is explicitly part
| of the interface because it is part of the protocol
| - generic status codes including redirects
| - content type negotiation - conditional requests
| - byte range requests - request/response body
| streaming
|
| > Maybe we need a better abstraction to begin with, something
| like discoverable native RPCs using a protocol designed for it
| like thrift or grpc.
|
| That's been tried. ONC RPC and DCE RPC for example had a
| service discovery system. It's not enough, and it can't be
| enough. You really need URIs. And you really need URIs to be
| embeddable in contents like HTML, XML, JSON, etc. -- by
| convention if need be (e.g., in JSON it can only be by
| convention / schema). You also need to be able to send extra
| metadata in request/response headers, including URIs.
|
| (Re: URIs and URIs headers, HATEOAS really depends on very
| smart user-agents, which basically haven't materialized because
| it turns out that HTML+JS _is enough_ to make UIs good, and so
| URIs in headers are not useful for UIs, but they _are_ useful
| for APIs.)
|
| It took me a long time to understand all of this, that REST _is
| right_ and typical RPCs are lame. Many of the points are very
| subtle, and you might have to build a RESTful application that
| uses many of these features of HTTP /REST in order to come
| around -- that's a lot to ask for!
|
| The industry seems to be constantly vacillating between REST
| and RPC. SOAP came and went; no one misses it. gRPC is the RPC
| of the day, but I think in the end the only nice thing about it
| is binary encodings and schema, and that it won't survive in
| the long run.
| cryptonector wrote:
| So basically a GET with a request body, with the query as the
| request body rather than as URI local-part query-parameters.
| TZubiri wrote:
| So a POST with different semantics?
| cryptonector wrote:
| Yes. A POST that is idempotent, safe, cacheable (POST
| normally isn't any of those).
|
| Alternatively this is a standardization of "GET-with-request-
| body" (which is NOT allowed by the spec but which some
| implement anyways).
| PaulDavisThe1st wrote:
| IETF 2029: The HTTP ASK Method
|
| IETF 2034: The HTTP TELLME Method
|
| IETF 2038: The HTTP WHATIF Method
| cryptonector wrote:
| Those sound like good April 1st RFC fodder.
| arcuri82 wrote:
| it looks great. Covering a few pain points in modelling APIs when
| you need to retrieve data idempotently but the URL is not
| identifying a specific resource, ie, "The content returned in
| response to a QUERY cannot be assumed to be a representation of
| the resource identified by the target URI".
|
| This is a "work in progress". Is there any estimation of when it
| will be finalized by? Something like during 2025, and
| frameworks/libraries starting to support it by something like
| 2026? Just to have a reference, anyone remember how long it took
| for PATCH?
| cryptonector wrote:
| This appears to be a Working Group item of the HTTPbis WG [0]
| which has a Zulip stream [1], a mailing list [2][3], a site
| [4], and a GitHub organization [5], thus lots of ways to send
| feedback. The Internet-Draft itself has a GitHub repository
| [6]. For this I think a GitHub issue/issues [7] would be best;
| some already exist now for issues in this thread.
| [0] https://datatracker.ietf.org/wg/httpbis/about/ [1]
| https://zulip.ietf.org/#narrow/stream/225-httpbis [2]
| mailto:httpbisa@ietf.org (I think) [3]
| https://mailarchive.ietf.org/arch/browse/httpbisa/ [4]
| https://httpwg.org/ [5] https://github.com/httpwg
| [6] https://github.com/httpwg/http-extensions
| https://github.com/httpwg/http-extensions/blob/main/draft-ietf-
| httpbis-safe-method-w-body.xml [7]
| https://github.com/httpwg/http-extensions/issues
___________________________________________________________________
(page generated 2024-09-16 23:02 UTC)