Indy UDP packets losses in duplex lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- vladbfg · 2024-07-19 12:34 Hello all, Before write module for my project I write simple test using TIdUDPServer. First, code of sender, that send 2000 packs 50 bytes every 300ms by timer Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TFmain.UPDSenderTimerTimer(Sender: TObject);  var i : integer;    s:string;    buf : TidBytes;begin  s:= #$7E + #0#0#0+#$7E;  s:= s+#$7E + #0#0#0+#$7E;  s:=s+s+#0#0#0#0#0;// 10+10+5  s:=s+s;// 50   buf := bytesof(s);  for i:=1 to MaxThreads(2000) do begin    idUDPServer.SendBuffer(EditDebug3.Text, 1003, buf);    txp := txp+1;  txb := txb+length(s);  end;end;  I tested and checked it by Wireshark and all looks like good, I see thousands packets and counts in capture are equal my sended count txp Second, I write echo server, it receives pack and send back. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TFMain.IdUDPServer1UDPRead(AThread: TIdUDPListenerThread;  const AData: TIdBytes; ABinding: TIdSocketHandle);  var sansi : ansistring = '';begin  RxB:= RxB + length(AData);  RxP:= RxP + 1;   lastip := ABinding.PeerIP;  lastport := ABinding.PeerPort;   SetLength(sansi, Length(AData));  Move(Pointer(AData)^, Pointer(sansi)^, Length(AData));   Packs.Add(sansi);  //IdUDPServer1.SendBuffer(ABinding.PeerIP, ABinding.PeerPort, AData); end; After that, I notice that not all packs are achieved the echo-server. For example I sended 100,000 packs, received 80,000 packs,  but wireshark shows in receive side 100,000packs. Then I commented  //IdUDPServer1.SendBuffer(ABinding.PeerIP, ABinding.PeerPort, AData); echo sending and all 100thds packs by rxp  are acheived the aplication|socket. So, my question is how to make it work in duplex mode? May be it is nedeed set big buffers or timeouts, separately  TX and RX works fine. ---------------------------------------------------------------------- cdbc · 2024-07-19 13:02 Hi Define a protocol for your duplex, e.g.: handshakes, ack, nack etc... Regards Benny ---------------------------------------------------------------------- vladbfg · 2024-07-22 04:24 Quote from: cdbc on July 19, 2024, 03:02:25 pmHi Define a protocol for your duplex, e.g.: handshakes, ack, nack etc... Regards Benny What does protocol have to do with it? There is no protocol (my own protocol, I send raw data over UDP).  Protocol doesn't matter. P.S. You probably didn't understand the question- I need to transfer data in both direction simultaneously, simultaneously Rx + Tx. ---------------------------------------------------------------------- TRon · 2024-07-22 05:33 Quote from: vladbfg on July 22, 2024, 06:24:40 amWhat does protocol have to do with it? Everything  QuoteThere is no protocol (my own protocol, I send raw data over UDP).  Protocol doesn't matter. Please read up on UDP f.e. wikipedia Quote... UDP is a connectionless protocol meaning that messages are sent without negotiating a connection and that UDP doesn't keep track of what it has sent.[1][2] .. Note "connectionless" QuoteP.S. You probably didn't understand the question- I need to transfer data in both direction simultaneously, simultaneously Rx + Tx. I think cdbc understood the question as intended but that is just my hunch. Are you aware that Indy socket implementation uses blocking sockets ? ---------------------------------------------------------------------- vladbfg · 2024-07-22 06:08 Quote from: TRon on July 22, 2024, 07:33:00 amQuote... UDP is a connectionless protocol meaning that messages are sent without negotiating a connection and that UDP doesn't keep track of what it has sent.[1][2] .. Note "connectionless" And what? Did you read my post carefully? ALL packets reach their destination in both directions separately without losses. Moreover, all packets reach the network card, but do not reach the program (or socket), while I am sending packets (while receiving). QuoteAre you aware that Indy socket implementation uses blocking sockets ?Thank you for this. Ok, the socket cannot receive a packet while it is transmitting. But why is the data lost? Why is the data not accepted into the buffer by the operating system (or socket realization) so that I can pick it up later? ---------------------------------------------------------------------- cdbc · 2024-07-22 06:14 Hi edit: Answer to your latest Q: UDP doesn't guarantee delivery. TCP does! If by /simultaneous/, you mean "Sending while receiving", then you need >=2 ports. Otherwise you can do something like this:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TFmain.UPDSenderTimerTimer(Sender: TObject);  var i : integer;    s:string;    buf,MyAck : TidBytes; (* MyAck is the result buffer, to check handshake *)begin  s:= #$7E + #0#0#0+#$7E;  s:= s+#$7E + #0#0#0+#$7E;  s:=s+s+#0#0#0#0#0;// 10+10+5  s:=s+s;// 50   buf := bytesof(s);  for i:=1 to MaxThreads(2000) do begin    idUDPServer.SendBuffer(EditDebug3.Text, 1003, buf);    txp := txp+1;  txb := txb+length(s);    (* here you read the 'ACK' from the server, the /echo/ in your case*)    idUDPServer.ReadBuffer(MyAck); // you'd have to figure out how it goes    (* this would have to be a 'blocking' operation, to avoid overrun, here *)    (* you check, that the 'ack' is correct, otherwise resend packet till it is *)  end;end; Now you should be able to do this:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TFMain.IdUDPServer1UDPRead(AThread: TIdUDPListenerThread;  const AData: TIdBytes; ABinding: TIdSocketHandle);  var sansi : ansistring = '';begin  RxB:= RxB + length(AData);  RxP:= RxP + 1;   lastip := ABinding.PeerIP;  lastport := ABinding.PeerPort;   SetLength(sansi, Length(AData));  Move(Pointer(AData)^, Pointer(sansi)^, Length(AData));   Packs.Add(sansi);  (* now the server waits for this to come through, before sending next *)  IdUDPServer1.SendBuffer(ABinding.PeerIP, ABinding.PeerPort, AData); end;That's what I mean with "protocol", never mind the rudimentary example... Regards Benny ---------------------------------------------------------------------- vladbfg · 2024-07-22 06:39 Quote from: cdbc on July 22, 2024, 08:14:51 amHi edit: Answer to your latest Q: UDP doesn't guarantee delivery. TCP does! Even a cat understands this. You suggested using tcp simulation with ack/nack message, ok, but this is not a solution to the problem. If you use this you will get ping-pong protocol, not full duplex bidirectional stream. Quote you mean "Sending while receiving", then you need >=2 ports.Yes, that's exactly what I meant I'll say it again: ALL packets reach the network card. But the packets do not reach the application while the application is sending packets That's my question, How to fix it without any crutches. How to configure a socket or OS to accept and remember incoming packets while sending. ---------------------------------------------------------------------- Remy Lebeau · 2024-07-22 07:02 Quote from: vladbfg on July 19, 2024, 02:34:30 pmFirst, code of sender, that send 2000 packs 50 bytes every 300ms by timer Why are you using TIdUDPServer to send the test packets, instead of using TIdUDPClient? Quote from: vladbfg on July 19, 2024, 02:34:30 pmI write echo server, it receives pack and send back. The correct way to send a reply back to the sender in the OnUDPRead event is to use ABinding.SendTo() instead of TIdUDPServer.SendBuffer().  The ABinding parameter represents the listening socket that actually received the packet and fired the OnUDPRead event, so you should use the same socket to send the reply back.  If you use SendBuffer() instead, it will use the 1st socket in the TIdUDPServer.Bindings collection, which might not be the correct socket if there are multiple bindings defined (such as listening on separate IPv4 + IPv6 sockets, or a separate socket per network interface, etc). Quote from: vladbfg on July 19, 2024, 02:34:30 pmAfter that, I notice that not all packs are achieved the echo-server. For example I sended 100,000 packs, received 80,000 packs,  but wireshark shows in receive side 100,000packs. The most likely cause is your server reading packets slower than the sender is sending them, so the listening socket's buffer eventually fills up and discards packets.  Unlike TCP, UDP does not guarantee delivery.  You need to either slow down the rate of the sender, or speed up the rate of the reading, or increase the buffer size of the reader socket so it can hold more packets in cache, eg: Code: [Select]for I := 0 to IdUDPServer1.Bindings.Count-1 do begin   IdUDPServer1.Bindings[I].SetSockOpt(Id_SOL_SOCKET, Id_SO_RCVBUF, 1024*1024*10); // 10MB end; Quote from: vladbfg on July 19, 2024, 02:34:30 pmSo, my question is how to make it work in duplex mode? It already is. That is not the problem. Quote from: vladbfg on July 19, 2024, 02:34:30 pmMay be it is nedeed set big buffers Most likely. Quote from: cdbc on July 22, 2024, 08:14:51 amIf by /simultaneous/, you mean "Sending while receiving", then you need >=2 ports. That is not correct.  A *socket* can send and read at the same time.  A *thread* cannot, and the OnUDPRead event is triggered in a worker thread. But you can multiple bindings running in parallel. And you can have other threads sending while the server is reading. Quote from: vladbfg on July 22, 2024, 08:39:21 amHow to configure a socket or OS to accept and remember incoming packets while sending. It already does. But it can hold only so many packets by default. If a new packet arrives and the socket's buffer is full, the packet is silently discarded. ---------------------------------------------------------------------- cdbc · 2024-07-22 07:23 Hi QuoteThat is not correct.  A *socket* can send and read at the same time.  A *thread* cannot, and the OnUDPRead event is triggered in a worker thread. But you can multiple bindings running in parallel. And you can have other threads sending while the server is reading.Thanks Remy, I've learnt something new today  Regards Benny ---------------------------------------------------------------------- vladbfg · 2024-07-22 11:07 Quote from: Remy Lebeau on July 22, 2024, 09:02:03 amCode: [Select]IdUDPServer1.Bindings[0].SetSockOpt(Id_SOL_SOCKET, Id_SO_RCVBUF, 1024*1024*10); // 10MB Thx!  bingo! it works without losses now. Thanks very much, Remy . QuoteThe correct way to send a reply back to the sender is to have the OnUDPRead event handler use ABinding.SendTo() instead of TIdUDPServer.SendBuffer().Is it better/faster? I watched code, there are such calls:   sendbuffer(1)-> sendBuffer(2) ->                   GStack.ResolveHost();                   Binding.SendTo(); Yes, sendTO faster ---------------------------------------------------------------------- Remy Lebeau · 2024-07-22 15:40 Quote from: vladbfg on July 22, 2024, 01:07:08 pmQuoteThe correct way to send a reply back to the sender is to have the OnUDPRead event handler use ABinding.SendTo() instead of TIdUDPServer.SendBuffer().Is it better/faster? I watched code, there are such calls:   sendbuffer(1)-> sendBuffer(2) ->                   GStack.ResolveHost();                   Binding.SendTo(); Yes, sendTO faster See the update I just made to my previous post.  It is not just a matter of being faster, of also of using correct semantics. ---------------------------------------------------------------------- vladbfg · 2024-07-23 08:02 Quote from: Remy Lebeau on July 22, 2024, 09:02:03 amWhy are you using TIdUDPServer to send the test packets, instead of using TIdUDPClient? Because I need receive/send and packets on both sides sporadically. I don't want use model client-server, both apps acts as "servers" and waits incoming packs, and both can send packs, and uses one port. [App1] Tx  >> datastream1 >>  Rx[App2]   |   parallel and [App1] Rx  << datastream2 <<  Tx[App2]  |    asynchronously Quote it will use the 1st socket in the TIdUDPServer.Bindings collection, which might not be the correct socket if there are multiple bindings definedThx for explanation. I use one binding (which is automatically created upon activation server idudpsrv.active:=true) QuoteA *thread* cannot, and the OnUDPRead event is triggered in a worker thread. But you can multiple bindings running in parallel. And you can have other threads sending while the server is readingIs it possible to pass one socket to 2(N) threads and each thread will be able to send data over the same socket? Just interested. ---------------------------------------------------------------------- Thaddy · 2024-07-23 08:51 Are you aware UDP is a potentionally lossy protocol? In other words it is fire and forget? Because that is basically everything where all your questions and possibly misunderstanding come from: UDP has no guaranteed delivery and is not designed like that. It is only fast, but not lossless. ---------------------------------------------------------------------- TRon · 2024-07-23 09:51 Quote from: Remy Lebeau on July 22, 2024, 09:02:03 amThe most likely cause is your server reading packets slower than the sender is sending them, so the listening socket's buffer eventually fills up and discards packets.  Unlike TCP, UDP does not guarantee delivery.  You need to either slow down the rate of the sender, or speed up the rate of the reading, or increase the buffer size of the reader socket so it can hold more packets in cache, eg: Quote from: Thaddy on July 23, 2024, 10:51:04 amAre you aware UDP is a potentionally lossy protocol? In other words it is fire and forget? Because that is basically everything where all your questions and possibly misunderstanding come from: UDP has no guaranteed delivery and is not designed like that. It is only fast, but not lossless. Besides not guaranteeing delivery and potential loss it also doesn't guarantee order (unless indy has that covered) so depending on what TS actually wants to achieve some sort of protocol is required. ---------------------------------------------------------------------- Thaddy · 2024-07-23 10:12 Indy does not order UDP packages. That is programmer responsability in the case of UDP. On very stable networks, with high bandwidth, like in the Netherlands, UDP can achieve near TCP accuracy, but UDP was never meant for that. https://en.wikipedia.org/wiki/User_Datagram_Protocol#Attributes Using UDP for streaming real-time audio or video, where a frame drop is not critical, UDP comes alive, but never use it for anything else. ---------------------------------------------------------------------- vladbfg · 2024-07-23 11:11 Quote from: Thaddy on July 23, 2024, 10:51:04 amAre you aware UDP is a potentionally lossy protocol? In other words it is fire and forget? Because that is basically everything where all your questions and possibly misunderstanding come from: UDP has no guaranteed delivery and is not designed like that. It is only fast, but not lossless. With all due respect, you did not understand my question and apparently did not read the entire thread carefully. I'm not looking for a lossless protocol. In my case, the losses were due to the buffer depth setting and slow reading. In an abstract local network with sufficient bandwidth there should be no packet loss at all. Quotewhere a frame drop is not critical, UDP comes alive, but never use it for anything else Sounds like too categorical a statement. And yes, you are right, RTP uses UDP as transport. In turn, I can recommend that you read about such protocols as SNMP, SNTP/NTP, RIP, DHCP and may be google experimental QUIC and others. PS: DNS ---------------------------------------------------------------------- MarkMLl · 2024-07-23 11:56 Quote from: vladbfg on July 23, 2024, 01:11:50 pmWith all due respect, you did not understand my question and apparently did not read the entire thread carefully. I'm not looking for a lossless protocol. In my case, the losses were due to the buffer depth setting and slow reading. In an abstract local network with sufficient bandwidth there should be no packet loss at all. I'll break the habit of a lifetime, and stick up for Thaddy here. With all due respect, /you/ haven't read the documentation: UDP is a lossy protocol with no guarantee of delivery order, even on a local network. The fact that you've seen all the expected packets using Wireshark is meaningless, since it has a separate point at which it intercepts packets and is immune to kernel- and app-level buffering issues. It would also be irrelevant if Wireshark consistently showed packets arriving in the correct order by by the time you read them some were swapped: the protocol, considered end-to-end, does not guarantee order, and it is entirely possible for a system's buffering to dequeue them arbitrarily. If you want something reliable, then either build sequence numbering and acknowledgements on to of UDP (which is a point somebody made earlier), use TCP, or use something like SCTP which should be OK locally even though many ISPs drop it. MarkMLl ---------------------------------------------------------------------- vladbfg · 2024-07-23 13:31 Quote from: MarkMLl on July 23, 2024, 01:56:37 pmUDP is a lossy protocol with no guarantee of delivery order, even on a local network. About order: I didn't say anything about order, I said all packs must be delivered. So, network devices (swithes) decide to discard the UDP packet? May be NetCard decides drop the frame with UDP data and transfer the frame with TCP data? I'll say it again: in a normal local network no packets should be lost. PS: If your local network (swithes/PC/other HW) is losing packets in normal state (no overload bandwidth), look for hardware problems. ---------------------------------------------------------------------- rvk · 2024-07-23 14:03 Quote from: MarkMLl on July 23, 2024, 01:56:37 pmI'll break the habit of a lifetime, and stick up for Thaddy here. With all due respect, /you/ haven't read the documentation: UDP is a lossy protocol with no guarantee of delivery order, even on a local network. All packages were received by the network card just fine. None were lost. Only because of the buffering in the network driver the packages were lost. Agreed, udp doesn't guarantee delivery but that wasn't the problem here. TS. You might want to remove the sending of packages from the receive event. Now the receive event takes too long (because you have the overload of sending). It would be better to, if you want a sort of echo, to push the data to a buffer, and do the sending in another thread. That way the receive end will stay clean and fast and you probably receive all data. Now you increased the buffer but that will only get you so far into it craps out again (with more data). Of course you need to determine a good size for your buffer but that might be better than adjusting the network driver buffer like it is done now. ---------------------------------------------------------------------- H₂SO₄ · 2024-07-23 14:21 Quote from: vladbfg on July 23, 2024, 01:11:50 pmIn an abstract local network with sufficient bandwidth there should be no packet loss at all. UDP does not guarantee delivery, period. There's no way around it - that's how the protocol is defined, and anything else is essentialy relying on undefined behaviour. Software is like a contract: don't be surprised when someone (metaphorically) sues you after you break its terms. ---------------------------------------------------------------------- Thaddy · 2024-07-23 14:41 Quote from: rvk on July 23, 2024, 04:03:06 pmAll packages were received by the network card just fine. by accident.... Or the throw of a dice. Or living in the Benelux or the Baltic states. And even then it is by accident.. At least on this one I get overwealming support from others that are experts. ---------------------------------------------------------------------- rvk · 2024-07-23 14:46 Quote from: Thaddy on July 23, 2024, 04:41:33 pmQuote from: rvk on July 23, 2024, 04:03:06 pmAll packages were received by the network card just fine. by accident.... Or the throw of a dice. At least on this one I get overwealming support from others that are experts. Or they are received because of good quality network (like we have here). Guys, we know udp doesn't guarantee delivery (everyone agrees here) but if you loose 20% of your packages, which was the problem here, then there is something seriously wrong. In this case it was the buffering due to not reading the packages fast enough. True, don't expect all packages to be delivered (for that you need other protocols or management) but when you loose that many, you need to find out why (which is discovered already). (Same goes for voip which usually also uses udp.) ---------------------------------------------------------------------- Thaddy · 2024-07-23 14:50 Rik, we are spoiled. Try Scotland Wales, Northen England or even France or Austria. Let alone the U.S. (where the infrastructure is happless, not helpless) 20 % package loss is actually quite good... allows for video streaming ---------------------------------------------------------------------- rvk · 2024-07-23 14:52 Quote from: Thaddy on July 23, 2024, 04:50:58 pmRik, we are spoiled. Try Scotland Wales or even France or Austria. Let alone the U.S. (which is happless, not helpless)  20 % package loss is actually quite good... I wonder how voip performs there then. Would be a nightmare I guess. ---------------------------------------------------------------------- Thaddy · 2024-07-23 14:54 Depending on where you are.But thanks to udp it still is understandable in many cases. You and me are really spoiled. ( I forgot about udp over tcp.) ---------------------------------------------------------------------- Warfley · 2024-07-23 16:19 Quote from: vladbfg on July 23, 2024, 03:31:33 pmSo, network devices (swithes) decide to discard the UDP packet? May be NetCard decides drop the frame with UDP data and transfer the frame with TCP data? Yes every device in the network can decide to drop packets for any reason whatsoever. For example a switch can start dropping packages because it's overheating to protect the hardware from damage on a hot summer day. Also radiation, there is a low chance that any data in transmission will randomly get a bit flipped from background radiation. Also every device in the network has fixed size buffers, which when they overflow you get package loss. You may think now why would you ever encounter package loss in a local network with low traffic? The answer to that is funnily enough TCP. TCP has a mechanism called congestion control. Basically TCP will increase it's sending rate until package loss is detected, in order to figure out the maximum capacity of the network and use this to adjust the sending rate. So TCP is implemented in a way that it will literally DOS your switches in order to figure out what the maximum transmission speed is. Smart routers like used by ISPs in the backend will treat TCP and UDP packets differently, and have different package loss rates for them. Your cheap home switches don't. So a single TCP connection, e.g. someone using the web, can already lead to local network traffic loss. ---------------------------------------------------------------------- Warfley · 2024-07-23 16:24 Quote from: rvk on July 23, 2024, 04:52:34 pmI wonder how voip performs there then. Would be a nightmare I guess. There is a whole area of research for this called Quality of Service. Basically how it works is that the internet backbones will look into the packages and categorize them according to their requirements. VoIP is one of the use cases which is relatively robust against package loss, but requires very small delays. So these packages will have usually a smaller queue that guarantees that a package is either dropped early or sent out early, to minimize delay. TCP on the other hand requires a steady throughput, but as described in my previous post, relies on package loss for it's congestion control. So TCP packages are dropped very frequently, even before the buffers are full, each package will get a certain chance to be dropped when it arrives. Online gaming usually requires both, low delay and low package loss and therefore are usually prioritized in general. Video Streaming requires low package loss but can live with a high delay, so it get's the largest buffers. And so much more. I had a whole module in University about this stuff. Long story short it's very complicated and there are no generic statements like x% package loss possible/meaningful ---------------------------------------------------------------------- Remy Lebeau · 2024-07-24 06:28 Quote from: vladbfg on July 23, 2024, 10:02:13 amThx for explanation. I use one binding (which is automatically created upon activation server idudpsrv.active:=true) Are you aware that activating an Indy server without any bindings defined may create 2 bindings, not just 1?  If you want to ensure only 1 binding, you should define it yourself before activating. Quote from: vladbfg on July 23, 2024, 10:02:13 amIs it possible to pass one socket to 2(N) threads and each thread will be able to send data over the same socket? Yes, provided adequate serialized access to the socket so multiple sends don't overlap each other.  For a message-oriented socket like UDP, the default serialization provided by the OS should suffice since sends are atomic.  This is not true for stream-oriented sockets like TCP, in which case you have to serialize in your own code, such as with a mutex.