Basic Data Exchange over home network lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- AlanTheBeast · 2021-05-09 18:18 I have a RPi4 project using Ultibo RTL (no OS). That end of the project is generating a lot of data and I'm not happy with using the SD card to 'import' it to the Mac or same to transport updated run time program to the Pi. I'd like a bare bones ethernet solution where my Mac can send/receive arbitrary data from the Pi and v-v.  (I'd be writing the code for each machine).  Call the Mac program Boss and the Pi program Sensor. I'm not sure if TCP/IP is too much or if UDP is too little.  In the later case I can easily implement checksumming and such to verify the data. I have no idea how to do this. Is there an absolutely bare bones FP demo of using UDP or TCP/IP on a local network between two machines? The local network has a WiFi-modem-router and a switch in between the two machines.  This is all wired, not WiFi. Thanks. Update: I've found some code for the Ultibo RTL unit called Winsock2.  It compiled and is allegedly sending a test message 1/second to my Mac from the Pi.  (Local IP and port). I see mention here of INet, but it won't "uses" on the mac (FPC compiler).... hmm - seems to be for Windows and Linux, not Mac.  ! ---------------------------------------------------------------------- MarkMLl · 2021-05-09 20:18 Are you moving packets of data described by a fixed-size record or streams? MarkMLl p.s. You've probably seen my considered opinion of LNet elsewhere. ---------------------------------------------------------------------- AlanTheBeast · 2021-05-09 20:37 Fixed size packets is fine.  That is to say the packets may vary in size from packet to packet, but not be a stream. I have not seen your opinion but your tone says it all... ---------------------------------------------------------------------- PierceNg · 2021-05-10 01:58 Quote from: AlanTheBeast on May 09, 2021, 08:18:15 pmI have a RPi4 project using Ultibo RTL (no OS). I'd like a bare bones ethernet solution where my Mac can send/receive arbitrary data from the Pi and v-v.  (I'd be writing the code for each machine).  MQTT https://mqtt.org/ is designed for IoT networking. Here's an implementation for Ultibo: https://github.com/pjde/ultibo-mqtt. Search the forum for threads of MQTT for FPC / Lazarus on the usual 'regular' operating systems. ---------------------------------------------------------------------- MarkMLl · 2021-05-10 07:31 Quote from: AlanTheBeast on May 09, 2021, 10:37:17 pmFixed size packets is fine.  That is to say the packets may vary in size from packet to packet, but not be a stream. In that case either follow @PierceNg and use whatever you can find available, or stick to UDP which is a very thin wrapper over IP which is a very thin wrapper over Ethernet or SLIP. Avoid TCP unless you really need it, it's a rabbit hole. LNet might have been OK before threads were universally implemented, but is somewhat... abstruse. I use it for Telnet and have added a couple of bits to the variant bundled as a tool with FPC, but there are some nasty bugs in there which have defied resolution and if I need to do more I'll quite simply reimplement Telnet from scratch. MarkML ---------------------------------------------------------------------- AlanTheBeast · 2021-05-10 19:44 Quote from: PierceNg on May 10, 2021, 03:58:17 amMQTT https://mqtt.org/ is designed for IoT networking. Thanks! Quote from: MarkMLl on May 10, 2021, 09:31:49 amIn that case either follow @PierceNg and use whatever you can find available, or stick to UDP which is a very thin wrapper over IP which is a very thin wrapper over Ethernet Thanks! Probably get to this on Friday. ---------------------------------------------------------------------- MarkMLl · 2021-05-10 20:29 The problem with TCP- and I note that MQTT uses it- is that while the trivial protocol isn't too bad in practice it relies on a very large number of "options". There is truly vast scope for error, to the extent that between its inception in the 1970s and (say) 2010 not just some but /most/ implementations had bugs that could be exploited for intrusion or a (D)DoS. Saying that it's for strictly local use so it doesn't have to be robust is, as I'm quite sure you're aware, no defence. I don't know what would be involved in an "SCTP-lite", i.e. a minimal implementation for something like Ultibo, but it might be worth reviewing. Later: sniffing around for "SCTP-Lite", https://www.pdl.cmu.edu/mailinglists/ips/mail/msg03648.html discusses the addition of framing on top of TCP, but this would obviously assume all the baggage of TCP. There's obviously been no movement on that in the intervening 19 (!) years, but that's not to say that there isn't a need for it. So what have I got against using TCT where it's not needed? These for a start: https://lcamtuf.coredump.cx/oldtcp/tcpseq.html https://lcamtuf.coredump.cx/newtcp/ /If/ I had time and incentive to look at this (which I don't), I think I'd set up MQTT on two PCs using SCTP and then investigate what parts of the protocol were actually needed. But https://www.uni- due.de/~be0001/sctplib/index.html or http://www.sctp.de/sctp- download.html would probably be a good start. MarkMLl ---------------------------------------------------------------------- AlanTheBeast · 2021-05-15 11:54 Quote from: MarkMLl on May 10, 2021, 10:29:47 pmSaying that it's for strictly local use so it doesn't have to be robust is, as I'm quite sure you're aware, no defence. Not sure what you're alluding to, but what I am doing is a project where I need some very basic communication between some Pi's and my computer.  This is not release code, but tooling for my own needs.  For data 'robustness' I will indeed be implementing a basic checksum such as Fletcher and keeping the blocks small (256 byte or so) with a mechanism to signal bad blocks back to the sender.  Not sure if I'll throw block serial numbering and recovery on top of that.  The files I'll be transporting are pretty small generally. I could do something basic where the sender sends an arbitrary number of blocks, say 10 and not send anymore until it receives a 'pass' for a block by seq number.  If it receives a pass, then send a new block; if it receives a fail, then send a replacement for the block that failed.  If a timeout expires, then abandon the transfer. Or, given that the file sizes will be pretty small (several MB at most), just 1 block at a time, repeating blocks if failed and a general timeout. ---------------------------------------------------------------------- MarkMLl · 2021-05-15 12:14 Quote from: AlanTheBeast on May 15, 2021, 01:54:43 pmQuote from: MarkMLl on May 10, 2021, 10:29:47 pmSaying that it's for strictly local use so it doesn't have to be robust is, as I'm quite sure you're aware, no defence. Not sure what you're alluding to, but what I am doing is a project where I need some very basic communication between some Pi's and my computer.  This is not release code, but tooling for my own needs. Come on old chap. You've been knocking around industry for long enough to know that roughly 50% of the software (and much hardware) in circulation is stuff knocked together on those terms, which somebody (else) has then pressed into service where it's publicly visible. https://www.theregister.com/2020/04/27/who_me/ MarkMLl ---------------------------------------------------------------------- AlanTheBeast · 2021-05-15 12:27 Quote from: PierceNg on May 10, 2021, 03:58:17 amQuote from: AlanTheBeast on May 09, 2021, 08:18:15 pmI have a RPi4 project using Ultibo RTL (no OS). I'd like a bare bones ethernet solution where my Mac can send/receive arbitrary data from the Pi and v-v.  (I'd be writing the code for each machine).  MQTT https://mqtt.org/ is designed for IoT networking. Here's an implementation for Ultibo: https://github.com/pjde/ultibo-mqtt. Search the forum for threads of MQTT for FPC / Lazarus on the usual 'regular' operating systems. In the end that's way over the top of what I want to do. I really need something very basic on the Mac end to do UDP in Free Pascal.  I'll implement what I need under that. ---------------------------------------------------------------------- AlanTheBeast · 2021-05-15 18:52 Quote from: MarkMLl on May 15, 2021, 02:14:57 pm Come on old chap. You've been knocking around industry for long enough to know that roughly 50% of the software (and much hardware) in circulation is stuff knocked together on those terms, which somebody (else) has then pressed into service where it's publicly visible. https://www.theregister.com/2020/04/27/who_me/ Amusing story.  But really - all I want is the simplest of procedures or functions that I can use to listen to the other machine and talk to it over UDP. This is what I have for the Pi to get going that allegedly works (it compiles and is threaded ... whether anything is getting to the Mac... not sure). What I can't figure out is an equivalent for the Mac side... Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure B asicUDP(p:pointer);const i:integer=0;var buf:string; IPAddress:string;  Winsock2UDPClient:TWinsock2UDPClient; beginWinsock2UDPClient:=TWinsoc k2UDPClient.Create;repeat  IPAddress:=Winsock2UDPClient.LocalAddress; sleep(1000) until ipaddress<>''; Winsock2UDPClient.RemoteHost := '10.7.219.69';Winsock2UDPClient.RemotePort := StrToInt('61013');Winsock2UDPClient.Connect;repeat  i:=i+1;  buf:='Test string '+inttostr(i);  Winsock2UDPClient.SendData(PChar(buf), Length(buf));  sleep(1000);until false;end; ---------------------------------------------------------------------- MarkMLl · 2021-05-15 19:46 I agree that UDP's a good bet subject to a very thin ack/nak protocol layer. Remember that it's a connectionless protocol, so what you need is something like Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i (* The result of this may be subsequently retrieved using InetDomainSocket().  Assume that any existing socket was deleted when its owner terminated. Create  a fresh Inet Domain socket and save the handle, on failure save an error code.  Both success and failure are silent.*)procedure CreateInetDomainSocket(port: integer); var  sockAddr: TSockAddr; begin // TODO : This could usefully announce what IP addresses are being listened on. // Scenario: the host computer is restarted, and comes up with a DHCP-allocated// address which no longer corresponds to that being assumed by other WatchXxx// instances assuming it's the supervisor.   Write(StdErr, 'Creating new Internet domain socket (UDP, port ', port, ')... ');  inetDomainSocketHandle := fpSocket(PF_INET, SOCK_DGRAM, 0);  try    sockAddr.sin_family:= AF_INET;    sockAddr.sin_port:= hToNS(port);    sockAddr.sin_addr.s_addr:= 0;    FillChar(sockAddr.sin_zero[0], SIZEOF(sockAddr.sin_zero), #0);    if fpBind(inetDomainSocketHandle, @sockAddr, SIZEOF(sockAddr)) = 0 then      WriteLn(StdErr, 'OK')    else begin      WriteLn(StdErr, 'failed, might need CAP_DAC_OVERRIDE,CAP_NET_BIND_SERVICE,CAP_NET_RAW=p+e');      inetDomainSocketHandle := -Abs(ErrNo)    end  except    WriteLn(StdErr, 'failed, might need CAP_DAC_OVERRIDE,CAP_NET_BIND_SERVICE,CAP_NET_RAW=p+e');    inetDomainSocketHandle := -Abs(ErrNo)  end (* Assume that this will be deleted automatically at program termination.       *) end { CreateInetDomainSocket } ;  from which you can then read using something like Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   FillByte(recvBuffer, SizeOf(recvBuffer), 0);    FillByte(sa, SizeOf(sa), 0);    saLength := SizeOf(sa);    messageLength := fpRecvFrom(socket, @recvBuffer, SizeOf(recvBuffer), 0, @sa, @saLength)    sender := NetAddrToStr(sa.sin_addr) + ':' + IntToStr(NToHS(sa.sin_port));  which will normally be inside a thread, or given a timeout using fpSelect(). MarkMLl ---------------------------------------------------------------------- AlanTheBeast · 2021-05-15 20:09 Thanks Mark, I'll get on that in the morning. ---------------------------------------------------------------------- MarkMLl · 2021-05-16 08:16 Note that as part of your setup protocol you'll need to check how long UDP datagrams can be. This tends to vary depending on what /exactly/ is in your network path, but in the worst case you'll find them truncated to somewhere around 1.5K and you might usefully consider the situation when the length changes in the middle of a session. Also consider "sliding window" for your acks... you can almost certainly do without it but you can usefully know about it. MarkMLl