https://www.http-feeds.org/ HTTP Feeds Asynchronous event streaming and data replication with plain HTTP APIs. HTTP feeds is a minimal specification for polling events over HTTP: * An HTTP feed provides a HTTP GET endpoint * that returns a chronological sequence (!) of events * serialized in CloudEvents event format * in batched responses using the media type application/ cloudevents-batch+json * and respects the lastEventId query parameter to scroll through further items * to support infinite polling for real-time feed subscriptions. HTTP feeds can be used to decouple systems asynchronously without message brokers, such as Kafka or RabbitMQ. Example GET /inventory HTTP/1.1 Host: https://example.http-feeds.org HTTP/1.1 200 OK Content-Type: application/cloudevents-batch+json [{ "specversion" : "1.0", "type" : "org.http-feeds.example.inventory", "source" : "https://example.http-feeds.org/inventory", "id" : "1c6b8c6e-d8d0-4a91-b51c-1f56bd04c758", "time" : "2021-01-01T00:00:01Z", "subject" : "9521234567899", "data" : { "sku": "9521234567899", "updated": "2022-01-01T00:00:01Z", "quantity": 5 } },{ "specversion" : "1.0", "type" : "org.http-feeds.example.inventory", "source" : "https://example.http-feeds.org/inventory", "id" : "292042fb-ab04-4653-af90-19a24032bffe", "time" : "2021-12-01T00:00:15Z", "subject" : "9521234512349", "data" : { "sku": "9521234512349", "updated": "2022-01-01T00:00:12Z", "quantity": 0 } },{ "specversion" : "1.0", "type" : "org.http-feeds.example.inventory", "source" : "https://example.http-feeds.org/inventory", "id" : "fa3e2a22-398c-4d02-ad08-9415e43178e6", "time" : "2021-01-01T00:00:22Z", "subject" : "9521234567899", "data" : { "sku": "9521234567899", "updated": "2022-01-01T00:00:21Z", "quantity": 4 } }] Client calls again with the last processed event id. GET /inventory?lastEventId=fa3e2a22-398c-4d02-ad08-9415e43178e6 HTTP/1.1 Host: https://example.http-feeds.org HTTP/1.1 200 OK Content-Type: application/cloudevents-batch+json [] An empty array signals the end of the feed. Polling A client can continue polling in an infinite loop to subscribe to a feed. Simple Polling A client calls the endpoint with the last known event id as lastEventId query parameter in an endless loop. If the response is an empty array, the client reached the end of the stream and waits some time to make another call to get events that occured in the meantime. Pseudocode: endpoint = "https://example.http-feeds.org/inventory" lastEventId = null while true: try: response = GET endpoint + "?lastEventId=" + lastEventId for event in response: process event lastEventId = event.id if response is empty: wait N seconds except: wait N seconds Client must persist the id of the last processed event as lastEventId for further fetches. The client's event processing must be idempotent (at-least-once delivery semantic). The id may be used for idempotency checks. Long Polling The server may also support long polling for lower latency. The client adds a timeout query parameter to specify the max period of milliseconds to wait for an response. Client Pseudocode: endpoint = "https://example.http-feeds.org/inventory" lastEventId = null timeout = 5000 // 5000 milliseconds is a good timeout period for long polling while true: try: response = GET endpoint + "?lastEventId=" + lastEventId + "&timeout=" + timeout for event in response: process event lastEventId = event.id // no client wait step within the loop except: // Delay the next request only in case of a server error wait N seconds If there are no newer events available, the server keeps the connection open until new events arrive or the defined period timed out. The server then sends the response (with the new events or an empty array) and the client can immediately perform another call. The latency can be improved, as the server can react to new events efficiently by implementing an internal event notification, change data capture triggers, or performing a high-frequency polling to the database. The cost of long polling is that the server needs to handle more open connections concurrently. This may become an issue with more than 10K connections. Event ID The event.id is used as lastEventId to scroll through further events. This means that events need to be strongly ordered to retrieve subsequent events. Events may also get deleted from the feed, e.g. through Compaction and Deletion. The server must still respect the original position and send only newer events, even when an event with the lastEventId was deleted. One way to achieve this is to use an time-ordered UUIDv6 as event ID (see IETF draft and Java-Library). This a viable option, especially if only one server appends new events to the feed, but might be a problem with multiple servers when the clocks are not in-sync. An alternative is to use a database sequence that is used as part of the event.id and interpreted when querying the database for the next batch. Example: The event.id 0000001000001::5f8de8ff-30d8-4fab-8f5a-c32f326d6f26 contains a database sequence 0000001000001 and a random UUID. Event Feeds HTTP feeds can be used to provide an API for publishing immutable domain events to other systems. It is quite common that one http feed endpoint includes different event types that belong to the same bounded context. Aggregate Feeds HTTP feeds can be used to provide an API for data collections of mutable objects (aka aggregates, master data) to other systems. An aggregate is identified through its subject. An aggregate feed must contain every aggregate at least once. Every created aggregate and each update leads to an appended feed entry with the full current state. Feed consumers can subscribe an aggregate feed to perform near real-time data synchronization to build local read models and to trigger actions when new or updated data is received. A feed consumer has a consistent state when reaching the end of the feed. The server should implement Compaction and may implement Deletion based on business requirements. Compaction Each aggregate update leads to an additional entry in the feed. It is good practice to keep the feed small to enable a quick synchronization of new clients. When feed items include the full current state of the resource, older feed items for the same aggregate may be outdated. Clients that read the feed from the beginning will therefore occur an inconsistent state for a short time. To mitigate, entries may be deleted from the feed when another entry was appended to the feed with the same subject. Example: There is an update for subject 9521234567899. HTTP/1.1 200 OK Content-Type: application/cloudevents-batch+json [{ "specversion" : "1.0", "type" : "org.http-feeds.example.inventory", "source" : "https://example.http-feeds.org/inventory", "id" : "1c6b8c6e-d8d0-4a91-b51c-1f56bd04c758", "time" : "2021-01-01T00:00:01Z", "subject" : "9521234567899", "data" : { "sku": "9521234567899", "updated": "2022-01-01T00:00:01Z", "quantity": 5 } },{ "specversion" : "1.0", "type" : "org.http-feeds.example.inventory", "source" : "https://example.http-feeds.org/inventory", "id" : "292042fb-ab04-4653-af90-19a24032bffe", "time" : "2021-12-01T00:00:15Z", "subject" : "9521234512349", "data" : { "sku": "9521234512349", "updated": "2022-01-01T00:00:12Z", "quantity": 0 } },{ "specversion" : "1.0", "type" : "org.http-feeds.example.inventory", "source" : "https://example.http-feeds.org/inventory", "id" : "fa3e2a22-398c-4d02-ad08-9415e43178e6", "time" : "2021-01-01T00:00:22Z", "subject" : "9521234567899", "data" : { "sku": "9521234567899", "updated": "2022-01-01T00:00:21Z", "quantity": 4 } }] After a compaction run, the first entry is gone: HTTP/1.1 200 OK Content-Type: application/cloudevents-batch+json [{ "specversion" : "1.0", "type" : "org.http-feeds.example.inventory", "source" : "https://example.http-feeds.org/inventory", "id" : "292042fb-ab04-4653-af90-19a24032bffe", "time" : "2021-12-01T00:00:15Z", "subject" : "9521234512349", "data" : { "sku": "9521234512349", "updated": "2022-01-01T00:00:12Z", "quantity": 0 } },{ "specversion" : "1.0", "type" : "org.http-feeds.example.inventory", "source" : "https://example.http-feeds.org/inventory", "id" : "fa3e2a22-398c-4d02-ad08-9415e43178e6", "time" : "2021-01-01T00:00:22Z", "subject" : "9521234567899", "data" : { "sku": "9521234567899", "updated": "2022-01-01T00:00:21Z", "quantity": 4 } }] Deletion Some aggregates need to be deleted, e. g. by regulatory requirements. Aggregate feeds use a method field with value DELETE to signal the deletion of an subject to consumers that built a local read model before. When aggregate was deleted, the server must append a DELETE entry with the subject to delete and no data content. { "specversion" : "1.0", "type" : "org.http-feeds.example.inventory", "source" : "https://example.http-feeds.org/inventory", "id" : "06b13630-e4c3-4d85-a669-ce66fc4daa75", "time" : "2021-12-31T00:00:01Z", "subject" : "9521234567899", "method": "DELETE" } Clients must delete this aggregate or otherwise handle the removal. The server should start a compaction run afterwards to delete previous entries for the same aggregate. Data Model An HTTP feed endpoint must support these query parameters: Query Type Mandatory Description Parameter The last event id that was processed. lastEventId String Optional Used to scroll through further events. May be missing or null to start from the beginning of the feed. A timeout is set, when long-polling should be used and is supported by the timeout Number Optional server. Max waiting time in milliseconds for long-polling, after which the server must send a response. A typical value is 5000. The response body contains an array of events that comply with the CloudEvents Specification in the application/cloudevents-batch+json format. Field Type Event Aggregate Description Feed Feed The currently supported specversion String Mandatory Mandatory CloudEvents specification version. A unique value (such as a UUID) for this event. It can be used to implement id String Mandatory Mandatory deduplication/idempotency handling in downstream systems. It is used as the lastEventId in subsequent calls. The type of the event. May be used to specify and deserialize the payload. A type String Mandatory Mandatory feed may contain different event types. It SHOULD be prefixed with a reverse-DNS name. The source system that source String Mandatory Mandatory created the event. Should be a URI of the system. The event addition time String Mandatory Mandatory timestamp. ISO 8601 UTC date and time format. Key to identify the business object. It doesn't have to be unique within the feed. This subject String n/a Mandatory should represent a business key such as an order number or sku. Used for compaction and deletion, if implemented. The HTTP equivalent method type that the feed item performs on the subject. method String n/a Optional PUT indicates that the subject was created or updated. DELETE indicates that the subject was deleted. Defaults to PUT. datacontenttype String Optional Optional Defaults to application/ json. The payload of the item. data Object Mandatory Optional Defaults to JSON. May be missing, e.g. when the method was DELETE. Further metadata may be added, e.g. for traceability. Authentication HTTP feeds may be protected with HTTP authentication. The most common authentication schemes are Basic and Bearer. The server may filter feed items based on the principal. When filtering is applied, caching may be unfeasible. Caching Servers may set appropriate response headers, such as Cache-Control: public, max-age=31536000, when a batch is full and will not be modified anymore. Libraries and Examples * http-feeds-server-spring-boot-starter * http-feeds-server-spring-boot-example * http-feeds-server-aws-serverless-example About This site is maintained by Jochen Christ. Jochen works as tech lead at INNOQ and is also author of WhichJDK.com, Remote Mob Programming, and Data Mesh Architecture. HTTP feeds has evolved from rest-feeds. Contributions are highly appreciated, e.g. libraries or examples in different languages or frameworks help a lot. Found an error or something is missing? Please raise an issue or create a pull request. This specification is published under CC BY 4.0. [noscript]