HTTP/2+HTTP/1.1+WebSocket server written with Lazarus (Free Pascal) lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- iLya2IK · 2021-02-25 12:54 The fpweb library is certainly good, but it clearly lacks modern features - namely, full-fledged http/1.1 with persistent connections, and also http/2. I completed my search for the desired analog by creating a server with all the necessary. I would like to present to your attention my project - WCHTTPServer. The project builds on the fcl-web library and extends it to increase functionality: Client management using cookies (saving and maintaining sessions). Saving information about clients and sessions in SQLite database.Saving information about the latest requests and saving logs in SQLite database.Multithreading preparation and execution of requests based on thread pools e.g. helpful classes to work with EventSources.Client rankings based on the frequency of client requests.Built-in support for gzip and deflate compression methods including decompression of client-side requests.The WebSocket (RFC 6455) protocol is supported with the "permessage-deflate" extension (RFC 7692).Ability to start the server both in HTTP/2 (RFC 7540) mode and in HTTP/1.1 mode.Ability to start the server both in HTTP/2 mode and in HTTP 1.1 mode.Modified OpenSSL modules (added necessary TLS extensions) in order to create and maintain HTTP/2 connections.Added the ability to save the master key and a random set of client data on the server-side (necessary for debugging TLS dumps using WireShark).Works under both Windows and Linux Link to project on GitHub : https://github.com/iLya2IK/wchttpserver Now the project is at its final stage, but it requires a careful look from the outside. On the forum, I would like to see suggestions for improving the functionality and bug reports. Thanks. ---------------------------------------------------------------------- PierceNg · 2021-02-25 13:42 Great stuff! Thanks for making this available. I only started looking at HTTP/2 and know next to nothing about it. I did find https://github.com/summerwind/h2spec. Can you try it on your implementation and share your views on the output please. ---------------------------------------------------------------------- iLya2IK · 2021-02-25 16:03 Quote from: PierceNg on February 25, 2021, 02:42:26 pmGreat stuff! Thanks for making this available. Thanks Quote from: PierceNg on February 25, 2021, 02:42:26 pmI did find https://github.com/summerwind/h2spec. Can you try it on your implementation and share your views on the output please. Nice advice I'll try it. I used before h2load. But I think h2spec models a lot of additional non-standard situations. ---------------------------------------------------------------------- iLya2IK · 2021-02-26 16:02 I checked the server with the h2spec utility: 41 passed from 94 tests. I found some bugs in my code caused by a misunderstanding of the http2 protocol (mostly in the part of informing clients about malformed requests and protocol errors). I also found that tests have their own bugs and some ambiguous testing algorithms. For example - Test 5.1/1 waits for the GOAWAY frame with PROTOCOL_ERROR, but section 6.1 in RFC 7540 requires GOAWAY frame with STREAM_CLOSED error. Another example - Test to emulate maximum concurrent streams. On my server, the test program was unable to create so many concurrent streams - no matter how I increase or decrease the MAX_CONCURRENT_STREAMS value - they seem to be quickly snatched up by worker threads. The author has yet to work out his testing program better, but now it is a powerful tool and I intend to use it further to improve the reliability and adequacy of my server. ---------------------------------------------------------------------- ASBzone · 2021-02-26 16:12 Thanks for this great work. You may want to post this in the "Third Party" board of the forum, though: https://forum.lazarus.freepascal.org/index.php/board,19.0.html ---------------------------------------------------------------------- PierceNg · 2021-02-27 00:14 Quote from: iLya2IK on February 26, 2021, 05:02:41 pmI checked the server with the h2spec utility: 41 passed from 94 tests. I found some bugs in my code caused by a misunderstanding of the http2 protocol (mostly in the part of informing clients about malformed requests and protocol errors). I also found that tests have their own bugs and some ambiguous testing algorithms. For example - Test 5.1/1 waits for the GOAWAY frame with PROTOCOL_ERROR, but section 6.1 in RFC 7540 requires GOAWAY frame with STREAM_CLOSED error. Another example - Test to emulate maximum concurrent streams. On my server, the test program was unable to create so many concurrent streams - no matter how I increase or decrease the MAX_CONCURRENT_STREAMS value - they seem to be quickly snatched up by worker threads. The author has yet to work out his testing program better, but now it is a powerful tool and I intend to use it further to improve the reliability and adequacy of my server. This is great - both your implementation and h2spec will improve through this interaction. Natural generative adversarial neural networks at work. ---------------------------------------------------------------------- iLya2IK · 2021-02-28 18:25 Testing status now using h2spec: 94 tests, 84 passed, 0 skipped, 10 failed Failed tests Code: XML  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   4. HTTP Frames    4.2. Frame Size            × 1: Sends a DATA frame with 2^14 octets in length        -> The endpoint MUST be capable of receiving and minimally processing frames up to 2^14 octets in length.           Expected: HEADERS Frame (stream_id:1)             Actual: Connection closed            Comment: Error in h2spec - DATA payload size didn't equal the value of "content-length" header.   5. Streams and Multiplexing    5.1. Stream States      5.1.2. Stream Concurrency              × 1: Sends HEADERS frames that causes their advertised concurrent stream limit to be exceeded          -> The endpoint MUST treat this as a stream error of type PROTOCOL_ERROR or REFUSED_STREAM.             Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)                       RST_STREAM Frame (Error Code: PROTOCOL_ERROR)                       GOAWAY Frame (Error Code: REFUSED_STREAM)                       RST_STREAM Frame (Error Code: REFUSED_STREAM)                       Connection closed               Actual: DATA Frame (length:845, flags:0x01, stream_id:201)            Comment: Ambiguous testing algorithm in h2spec. Streams quickly snatched up by worker threads     5.5. Extending HTTP/2            × 2: Sends an unknown extension frame in the middle of a header block      -> The endpoint MUST treat as a connection error of type PROTOCOL_ERROR.           Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)                     Connection closed             Actual: Timeout            Comment: I didn't find any references in RFC 7540 that unknown extension frame in the middle                    of a header block should cause a connection error   6. Frame Definitions    6.5. SETTINGS      6.5.3. Settings Synchronization              × 1: Sends multiple values of SETTINGS_INITIAL_WINDOW_SIZE        -> The endpoint MUST process the values in the settings in the order they apper.             Expected: DATA Frame (length:1, flags:0x00, stream_id:1)               Actual: DATA Frame (length:845, flags:0x01, stream_id:1)              Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.     6.9. WINDOW_UPDATE      6.9.1. The Flow-Control Window        × 1: Sends SETTINGS frame to set the initial window size to 1 and sends HEADERS frame          -> The endpoint MUST NOT send a flow- controlled frame with a length that exceeds the space available.           Expected: DATA Frame (length:1, flags:0x00, stream_id:1)             Actual: DATA Frame (length:845, flags:0x01, stream_id:1)                   Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.        × 2: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1          -> The endpoint MUST sends a GOAWAY frame with a FLOW_CONTROL_ERROR code.             Expected: GOAWAY Frame (Error Code: FLOW_CONTROL_ERROR)               Actual: Timeout                   Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.        × 3: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream          -> The endpoint MUST sends a RST_STREAM frame with a FLOW_CONTROL_ERROR code.             Expected: RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR)               Actual: Timeout              Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.       6.9.2. Initial Flow-Control Window Size                × 1: Changes SETTINGS_INITIAL_WINDOW_SIZE after sending HEADERS frame        -> The endpoint MUST adjust the size of all stream flow-control windows.             Expected: DATA Frame (length:1, flags:0x00, stream_id:1)               Actual: DATA Frame (length:845, flags:0x01, stream_id:1)                       Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.        × 2: Sends a SETTINGS frame for window size to be negative          -> The endpoint MUST track the negative flow-control window.             Expected: DATA Frame (length:1, flags:0x00, stream_id:1)               Actual: Timeout              Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.   8. HTTP Message Exchanges    8.1. HTTP Request/Response Exchange      8.1.2. HTTP Header Fields        8.1.2.2. Connection-Specific Header Fields                    × 1: Sends a HEADERS frame that contains the connection-specific header field            -> The endpoint MUST respond with a stream error of type PROTOCOL_ERROR.               Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)                         RST_STREAM Frame (Error Code: PROTOCOL_ERROR)                         Connection closed               Actual: DATA Frame (length:845, flags:0x01, stream_id:1)              Comment: I know that this violates the requirements of RFC 7540, but I sincerely consider them                          redundant and the type of reaction to the header value should remain on the server side ---------------------------------------------------------------------- trev · 2021-02-28 22:33 @iLya2IK: Great work! I've added a link to the Wiki Web Development Portal. ---------------------------------------------------------------------- iLya2IK · 2021-03-08 13:47 The project has undergone major changes related to debugging of flow control (according to sec.5.2 RFC 7540). Here are the actual results of testing with utilities. Code: XML  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i # h2spec -p 8080 http2 -t -k -o 2094 tests, 91 passed, 0 skipped, 3 failedHypertext Transfer Protocol Version 2 (HTTP/2)  4. HTTP Frames  4.2. Frame Size            × 1: Sends a DATA frame with 2^14 octets in length        -> The endpoint MUST be capable of receiving and minimally processing frames up to 2^14 octets in length.           Expected: HEADERS Frame (stream_id:1)             Actual: Connection closed            Comment: Error in h2spec - DATA payload size didn't equal the value of "content-length" header.     5. Streams and Multiplexing    5.5. Extending HTTP/2            × 2: Sends an unknown extension frame in the middle of a header block        -> The endpoint MUST treat as a connection error of type PROTOCOL_ERROR.           Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)                   Connection closed             Actual: Timeout            Comment: I didn't find any references in RFC 7540 that unknown extension frame in the middle                     of a header block should cause a connection error   8. HTTP Message Exchanges    8.1. HTTP Request/Response Exchange      8.1.2. HTTP Header Fields        8.1.2.2. Connection-Specific Header Fields                    × 1: Sends a HEADERS frame that contains the connection-specific header field            -> The endpoint MUST respond with a stream error of type PROTOCOL_ERROR.               Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)                         RST_STREAM Frame (Error Code: PROTOCOL_ERROR)                         Connection closed               Actual: DATA Frame (length:845, flags:0x01, stream_id:1)              Comment: I know that this violates the requirements of RFC 7540, but I sincerely consider them                         redundant and the type of reaction to the header value should remain on the server side       And here are the results of testing with h2load Code: XML  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i # h2load -n32000 -c32 -m10 --header=connection:keep-alive --header=cookie:cid=11  https://localhost:8080        starting benchmark...spawning thread #0: 32 total client(s). 32000 total requestsTLS Protocol: TLSv1.2Cipher: ECDHE-RSA-AES256-GCM-SHA384Server Temp Key: X25519 253 bitsApplication protocol: h2progress: 10% doneprogress: 20% doneprogress: 30% doneprogress: 40% doneprogress: 50% doneprogress: 60% doneprogress: 70% doneprogress: 80% doneprogress: 90% doneprogress: 100% done finished in 17.22s, 1858.68 req/s, 1.54MB/srequests: 32000 total, 32000 started, 32000 done, 32000 succeeded, 0 failed, 0 errored, 0 timeoutstatus codes: 32000 2xx, 0 3xx, 0 4xx, 0 5xxtraffic: 26.46MB (27746560) total, 126.38KB (129408) headers (space savings 95.01%), 25.79MB (27040000) data                   min         max         mean         sd        +/- sdtime for request:    16.59ms    327.96ms    150.22ms     35.42ms    77.80%time for connect:     7.17ms       1.96s       1.15s    618.49ms    62.50%time to 1st byte:    59.55ms       2.00s       1.20s    629.72ms    62.50%req/s           :      58.09       72.88       61.87        4.42    81.25% # h2load -n32000 -c32 -t8 -m1 --h1 --header=connection:keep-alive --header=cookie:cid=11  https://localhost:8080-t: warning: the number of threads is greater than hardware cores.starting benchmark...spawning thread #0: 4 total client(s). 4000 total requestsspawning thread #1: 4 total client(s). 4000 total requestsspawning thread #2: 4 total client(s). 4000 total requestsspawning thread #3: 4 total client(s). 4000 total requestsspawning thread #4: 4 total client(s). 4000 total requestsspawning thread #5: 4 total client(s). 4000 total requestsspawning thread #6: 4 total client(s). 4000 total requestsspawning thread #7: 4 total client(s). 4000 total requestsTLS Protocol: TLSv1.2Cipher: ECDHE-RSA-AES256-GCM-SHA384Server Temp Key: X25519 253 bitsApplication protocol: http/1.1progress: 10% doneprogress: 20% doneprogress: 30% doneprogress: 40% doneprogress: 50% doneprogress: 60% doneprogress: 70% doneprogress: 80% doneprogress: 90% doneprogress: 100% done finished in 29.06s, 1101.26 req/s, 1.01MB/srequests: 32000 total, 32000 started, 32000 done, 32000 succeeded, 0 failed, 0 errored, 0 timeoutstatus codes: 32000 2xx, 0 3xx, 0 4xx, 0 5xxtraffic: 29.39MB (30816000) total, 2.53MB (2656000) headers (space savings 0.00%), 25.79MB (27040000) data                   min         max         mean         sd        +/- sdtime for request:     7.99ms    208.16ms     26.31ms      5.94ms    93.71%time for connect:     8.22ms       3.08s       1.25s    943.18ms    62.50%time to 1st byte:    99.48ms       3.12s       1.31s    913.25ms    62.50%req/s           :      34.42       38.74       36.31        1.25    62.50% ---------------------------------------------------------------------- iLya2IK · 2021-04-22 07:16 Improved support for TLS v1.3, optimized the speed of the network code (read latency increased when the server was idle). ---------------------------------------------------------------------- iLya2IK · 2021-04-26 13:38 Decompression of client-side requests added. Only the deflate unpacking method is currently supported on the server-side Look at main.js and server.cnf. To implement compression on client-side used zlib.js https://github.com/imaya/zlib.js Code: Javascript  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i if (content.length > 200) {    var plain = new Uint8Array(toUTF8Array(content));    content = new Zlib.Deflate(plain, {compressionType: Zlib.Deflate.CompressionType.DYNAMIC}).compress();              xhr.setRequestHeader('Content-Encoding', 'deflate');      } ---------------------------------------------------------------------- iLya2IK · 2021-05-05 15:12 The latest update adds support for the websocket protocol, and therefore the demo site has been changed. As an example of interaction over the websocket protocol, the JSON-RPC application level protocol (well, or some of its similarity  - you can change it to your own) is used. Now I am working on adding the permessage-deflate extension (RFC 7692) ---------------------------------------------------------------------- mr-highball · 2021-05-06 04:09 Cool. Apparently I already starred your repo, but this comment is here for an extra ⭐ ---------------------------------------------------------------------- iLya2IK · 2021-05-06 06:13 Quote from: mr-highball on May 06, 2021, 06:09:43 amCool. Apparently I already starred your repo, but this comment is here for an extra ⭐ Thank you for your support ---------------------------------------------------------------------- iLya2IK · 2021-05-15 13:37 The structure of the project has been put in order. Added lpk file. The "permessage-deflate" extension for websocket has been fully implemented.