lNet updated for macOS, FreeBSD, Windows - Linux epoll problem lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- trev · 2022-03-19 12:01 Over the last few days I've been updating the lNet (Lightweight Networking Library) to compile and run successfully on FreeBSD, macOS and Windows using FPC 3.2.2 or FPC 3.3.1. It now does. macOS happily uses TLS 1.3 (thanks to MIA Forum user Tz) for SMTP;  FreeBSD happily uses TLS 1.2 for SMTP; and Windows 32 and 64 bit happily use TLS 1.2 for SMTP. I should mention that I've only thoroughly tested the SMTP protocol which was the itch I wanted to scratch because macOS has no framework for SMTP at all and so a third-party library was required. I have added the Lazarus SMTP client example to the Wiki here. The updated lNet library is on GitHub here. Alas, my Linux (Ubuntu 20.10) attempts met with a little less success. If I comment out the TLS handshake (FSMTP.StartTLS;) , the SMTP transaction flies through with no problems. If I leave the TLS handshake in, it succeeds but then the application locks up with the error: Error on epoll [4]: Interrupted system call which comes from the Linux only file lib/sys/lepolleventer.inc and never sends the email to my FreeBSD sendmail server which waits patiently to no avail. I very rarely ever use any Linux and I'll happily concede that I know nothing about its workings. My searching threw up a few similar epoll issues but did not enlighten me sufficiently to come up with a fix. So, if anyone feels like trying to tame the beast, you'll need to change the only FreeBSD ifdef in lib/lnetssl.pp to Linux because Linux, like FreeBSD, always seems to want to use SSL 2/3 rather than TLS 1.x and fails, so I force it to use TLS 1.2. macOS happily picks TLS 1.3 all by itself. Have fun :-) [Updated to add Windows] ---------------------------------------------------------------------- MarkMLl · 2022-03-19 12:39 I'm neither sufficiently up to speed on TLS nor a sufficiently good Linux systems programmer to be able to help directly, but out of curiosity: is this based on the master Lnet sources which I think are on Github these days, or on the variant that comes with FPC (utils/fppkg/lnet in the sources)? The reason I ask is that the latter has some extra Telnet support stuff which I think never made it into the main version. MarkMLl ---------------------------------------------------------------------- trev · 2022-03-19 13:09 I forked the these  lNet sources on GitHub after dabbling with the FPC ones and then discovering warnings that they were only for FPC development and should be avoided. QuoteI'm neither sufficiently up to speed on TLS nor a sufficiently good Linux systems programmer I resemble both aspects of that remark ;-) I think the TLS handshake is ok as apart from the client saying so, my mail server also says: QuoteMar 19 23:46:51 shadow sm-mta[75484]: STARTTLS=server, relay=ubuntu.example.org [192.168.1.26], version=TLSv1.2, verify=NO, cipher=ECDHE-RSA-AES256-GCM-SHA384, bits=256/256 ---------------------------------------------------------------------- MarkMLl · 2022-03-19 13:46 I find Lnet very difficult internally, and only use it for Telnet... and the next time I need that I'll redo from scratch. I modded the FPC ones to support the Hercules mainframe emulator, which used subcommands and demanded a terminal type that aren't widely implemented. Apropos TLS, I saw a situation late last month where an accounting package would no longer send invoices via Office-365 which turned out to be something related to TLS version. I mention this because there seems to be an unholy number of security patches being pushed out by the various OS vendors at the moment, and there's always a possibility that the test system which is giving you problems has had its configuration forcibly changed recently. MarkMLl ---------------------------------------------------------------------- trev · 2022-03-20 06:01 I've updated lNet - there's no longer an ifdef for FreeBSD. There's one for Darwin/macOS when setting the TLS preferred order, everything else gets a forced TLS 1.2 and down. As for the example SMTP client, I forgot to mention that for Linux (GTK2) you need to scatter application.processmessages around liberally (once in each of the On... functions and in the main loop) so that the log is updated - this wasn't needed for macOS (Cocoa) or FreeBSD (GTK2). Weird. ---------------------------------------------------------------------- MarkMLl · 2022-03-20 07:58 Quote from: trev on March 20, 2022, 07:01:37 amyou need to scatter application.processmessages around liberally Which is one of the reasons I really don't like Lnet: I suspect it predates threads, and I've seen some odd effects when one end or the other of a connection has been dropped unexpectedly which I've not been able to resolve  due to the complexity of the code. MarkMLl ---------------------------------------------------------------------- trev · 2022-03-20 09:41 The scattering seems confined to Linux. I'm working on Windows at the moment and like macOS and FreeBSD it doesn't require the scattering. And... it now works on Windows 32 bit and 64 bit (tested on Win 10 only, both 32 bit and 64 bit). ---------------------------------------------------------------------- trev · 2022-03-22 07:54 With my latest commit, FreeBSD, macOS and Windows now all choose TLS v1.3. Yay. Linux still chooses TLS v1.2 and dies after the handshake with the epoll error. Something to do with the Linux-specific lib/sys/lepolleventer.inc file obviously needs a tweak. The other operating systems use the BSD/Win inc files and are happy little Vegemites. ---------------------------------------------------------------------- marcov · 2022-03-22 09:08 Quote from: trev on March 19, 2022, 01:01:32 pmError on epoll [4]: Interrupted system call On Linux, syscalls are not reliable. To avoid deadlock, the kernel can let them fail with EsysEINTR, and all syscalls should be guarded with repeat loops to restart the syscall in such case. See e.g. the write() call in rtl/unix/sysfile.inc  (see below)  and grep for eintr in that directory Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Function Do_Write(Handle:thandle;Addr:Pointer;Len:Longint):longint; var j : cint;Begin  repeat    Do_Write:=Fpwrite(Handle,addr,len);    j:=geterrno;  until (do_write<>-1) or ((j<>ESysEINTR) and (j<>ESysEAgain));  If Do_Write<0 Then   Begin    Errno2InOutRes;    Do_Write:=0;   End  else   InOutRes:=0;End;  IOW EINTR and EAGAIN are not "real" errors but things that can happen in normal program flow IIRC this is an scheme used in older Unices, but Linux stuck to it because the supreme Penguin thought it to be elegant in its simplicity. ---------------------------------------------------------------------- MarkMLl · 2022-03-22 10:10 Quote from: marcov on March 22, 2022, 10:08:01 amOn Linux, syscalls are not reliable. To avoid deadlock, the kernel can let them fail with EsysEINTR, and all syscalls should be guarded with repeat loops to restart the syscall in such case. See e.g. the write() call in rtl/unix/sysfile.inc  (see below)  and grep for eintr in that directory I'm not arguing in the slightest but I thought the fp...() functions wrapped that retry automatically? MarkMLl ---------------------------------------------------------------------- Thaddy · 2022-03-22 10:14 Quote from: MarkMLl on March 22, 2022, 11:10:15 amI'm not arguing in the slightest but I thought the fp...() functions wrapped that retry automatically? These are not wrappers as such at all. They are mostly merely straight header translations to Pascal syntax. So if it does not work for FPC, it probably does not work on any other compiler for the same platform. ---------------------------------------------------------------------- marcov · 2022-03-22 10:45 Quote from: MarkMLl on March 22, 2022, 11:10:15 amI'm not arguing in the slightest but I thought the fp...() functions wrapped that retry automatically? I don't know why you thought that: rtl/linux/ossysc.inc: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function Fpwrite(fd: cint;buf:pchar; nbytes : size_t): ssize_t; [public, alias : 'FPC_SYSC_WRITE']; begin Fpwrite:=do_syscall(syscall_nr_write,Fd,TSy sParam(buf),nbytes);end;  One of the problems of doing it in the above routine, is that if the runtime switches to libc (android?), it is still needed on a higher level. ---------------------------------------------------------------------- MarkMLl · 2022-03-22 11:04 Thanks both for the clarification re ESysEINTR etc. My suspicion is that I was confused by recollection of very old discussion of the distinction between libc and syscalls, explained succinctly by e.g. the somewhat more recent https://www.gnu.org/software/libc/manual/html_node/Interrupted- Primitives.html (via https://lists.freepascal.org/fpc- pascal/2017-August/051997.html). MarkMLl ---------------------------------------------------------------------- han · 2022-04-21 11:53 >>The updated lNet library is on GitHub here. I'm glad somebody addressed some problems.  The Visual package still indicates V.0.6.6. Would it not be better to increase the version to 0.6.6a or 0.6.7? Han ---------------------------------------------------------------------- zamronypj · 2022-04-22 22:38 @trev I create issue https://github.com/trevoz/lnet/issues/1 regarding FPC_FULLVERSION number. Regards, ---------------------------------------------------------------------- trev · 2022-04-25 05:28 Quote from: han on April 21, 2022, 01:53:46 pm>>The updated lNet library is on GitHub here. I'm glad somebody addressed some problems.  The Visual package still indicates V.0.6.6. Would it not be better to increase the version to 0.6.6a or 0.6.7? I don't believe I touched the visual package code at all. ---------------------------------------------------------------------- trev · 2022-04-25 05:30 Quote from: zamronypj on April 23, 2022, 12:38:23 am@trev I create issue https://github.com/trevoz/lnet/issues/1 regarding FPC_FULLVERSION number. Thanks, I'll attend to it when I return home and have access to my Mac computers again.