[SOLVED] CURL throws Error CURLE_REMOTE_FILE_NOT_FOUND but only on
newer OS/CURL
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
Hartmut · 2023-06-09 15:16

I hope I picked the right subforum.

My little demo (see below) reads 1 mail-header via IMAP. It runs
perfect on Ubuntu 18.04 (with CURL 7.58). But on Ubuntu 22.04 (with
CURL 7.81) the same demo throws Error "CURLE_REMOTE_FILE_NOT_FOUND".
Don't know what happens on Windows, because my Windows 7 is too old
for Internet activities.

I'm a beginner to CURL and need your help.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i {$mode
objfpc}{$H+} uses sysutils,classes,libcurl; const   url_imap    =
'imaps://imap.gmx.net'; {wanted URL}   imap_folder = 'INBOX';        
      {wanted IMAP-Foldername}   user_pw     = 'username:password';  
 {insert your username:password}{$IFNDEF LINUX}                      
    {Windows-only: Certificate-Bundle: }   fspec_crt: PChar =
'C:\Progs\Curl\bin\curl-ca-bundle.crt';{$ENDIF}   curl_Verb = true;  
  {CURL with verbose output?}   FspecTmp = 'curl.tmp'; {temporary file
for a mail header} {$IFDEF LINUX}type size_t = dword; {is on Windows
in Unit system}{$ENDIF} var hCurl: pCurl; {= ^pointer = Curl-
Handle} function DoWrite(buf: Pointer; size: size_t; nmemb: size_t;  
             SS: Pointer): size_t; cdecl;   {Helper for function
curl_read_file(). Writes 'size*nmemb' bytes from buffer    'buf' into
a stream 'SS'. Returns the number of written bytes}   begin 
 result:=TStream(SS).Write(buf^,size*nmemb);   end; function
curl_read_file(url, user_pw, creq, fspecO: ansistring): CURLcode; 
 {reads from URL 'url' via CURL and writes the result into file
'fspecO'.    In: - user_pw = Username + Password as
'<username>:<password>'.        - creq: optional custom request.   
Out: returns as result the CURL-errorcode or 'CURLE_OK'}   var FS:
TFileStream;       pC: pChar;       e: CURLcode;   begin 
 writeln('curl_read_file(): url="' + url + '", creq="' + creq +
'"');    if curl_Verb then                                           
    {verbose?}      curl_easy_setopt(hCurl,CURLOPT_VERBOSE,[True]); 
 curl_easy_setopt(hCurl,CURLOPT_URL,[PChar(url)]);           {set
Server-URL} 
 curl_easy_setopt(hCurl,CURLOPT_USERPWD,[PChar(user_pw)]);    {set
user + pw}{$IFNDEF LINUX} 
 curl_easy_setopt(hCurl,CURLOPT_CAINFO,[fspec_crt]); {set Certificate-
Bundle}{$ENDIF}    if creq <> '' then pC:=PChar(creq)              
{optional Custom Request: }                 else pC:=nil; 
 curl_easy_setopt(hCurl,CURLOPT_CUSTOMREQUEST,[pC]);  
 FS:=TFileStream.Create(fspecO,fmCreate);              {output into a
file: }   curl_easy_setopt(hCurl,CURLOPT_WRITEFUNCTION,[@DoWrite]); 
 curl_easy_setopt(hCurl,CURLOPT_WRITEDATA,[Pointer(FS)]);  
 e:=curl_easy_perform(hCurl);                          {execute CURL-
command}   FS.Free;writeln('Result1=', e);   exit(e);   end; function
Test_read_IMAP(nr: integer): CURLcode;   {reads the Header of Mail
number 'nr' into file 'FspecTmp' and returns as    result the CURL-
errorcode or 'CURLE_OK'}   var url: ansistring;       e: CURLcode; 
 begin   url:=url_imap + '/' + imap_folder + '/;UID=' + IntToStr(nr) 
   + '/;SECTION=HEADER'; {read only the Header}  
 hCurl:=curl_easy_init; {initialize CURL}   if not
system.Assigned(hCurl) then exit(CURLE_FAILED_INIT);    try
e:=curl_read_file(url,user_pw,'',FspecTmp);   finally     
 curl_easy_cleanup(hCurl);   end; {try}   exit(e);   end; var e:
CURLcode;begin {main}e:=Test_read_IMAP(1); {reads mail-header from the
1st mail}writeln('Result2=', e);end.
I enabled "verbose CURL output" and attached the 2 outputs with CURL
7.58 and with CURL 7.81 and a diff-screenshot. What I see is, that
CURL 7.58 uses TLS 1.2, while CURL 7.81 uses TLS 1.3. Can this be the
problem? Or what else?

I use FPC 3.2.0 (but have the same error with FPC 3.2.2). Attached is
a compilable demo-project. Thanks in advance.

----------------------------------------------------------------------
Hartmut · 2023-06-13 15:04

Hmm, no answers after 4 days. Did I pick the wrong subforum? Did I
describe my problem not clear enough? Then please ask...

----------------------------------------------------------------------
rvk · 2023-06-13 15:32

You are hitting 2 different servers.
The imap implementation could be different.

Maybe you should first read out the UID before actually querying the
BODY.

Difference in FETCH 1 (BODY) and UID FETCH 1.
https://forums.oracle.com/ords/apexds/post/uid-fetch-1-vs-
fetch-1-285-uid-4595

So it's not certain your first message (1) is really UID = 1.
You shouldn't do a curl_read_file directly but first check out what
the real output of the command.

(Though I'm not an imap expert)

BTW. I send you a PM.

----------------------------------------------------------------------
Hartmut · 2023-06-13 17:22

Thank you very much Rik for helping me.

Quote from: rvk on June 13, 2023, 05:32:31 pmYou are hitting 2
different servers. The imap implementation could be different.
I did not realize that before.

QuoteMaybe you should first read out the UID before actually querying
the BODY.
My real program 1st reads in all UUID's and then selects 1 mail-
number. Do you mean that?
But my demo shall read only the header, not the body of a mail.

QuoteDifference in FETCH 1 (BODY) and UID FETCH 1.
https://forums.oracle.com/ords/apexds/post/uid-fetch-1-vs-
fetch-1-285-uid-4595
Sorry, I don't understand what you want to tell me and what is said in
this link. I don't know Java or JavaMail.
What is meant with 'UID'? I only "know" UUID's like in this list:
Code: Text  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i 1
1MmSgj-1pDbi21FIH-00iJVZ2 1MqfgY-1qfFac3pOi-00chkj3
1N5CpL-1qIpbz1s4T-010ylP4 1MAg1q-1pzjcy0FWi-00B8Co5
1MmFpi-1qbTMv42e8-00h7s9...I call the 1st column 'mail number' and the
2nd column 'UUID'. How do you call them?

I'm a beginner to CURL and to all of that mail stuff. Please be more
verbose. Thanks a lot.

----------------------------------------------------------------------
rvk · 2023-06-13 18:14

I'm not sure if you can do it like that (using URL to retrieve a
mailbody directory).
Normally you would connect to imap server and issue commands.

For the UID. With FETCH 1 you get the first mail in a box.
But with UID FETCH 1 you get the mail with UID 1.
But as I understand it, UID 1 isn't the same as the first mail.

So the error could be just that mail with UID 1 doesn't exist.

You could do a A001 UID SEARCH ALL to search for all UID's. When you
use that UID in the UID FETCH x.

If you want to do this manually you can do this:

Code: Bash  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i openssl
s_client -connect imap.gmx.net:993 -quiet a1 login username
password a2 select inbox a3 uid search all
You get a list with UIDs (which for me definitely are not in the lower
range).

Code: Bash  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i a3 uid
search all* SEARCH 364420 364746 364899 365310 365367 365464 365583
365584 365585 365586 365587 365588 365589 365590a3 OK Search completed
(0.001 + 0.000 secs).
So you need to use one of those UIDs.

Problem is that I don't know if you can get those UID numbers via an
URL.
(you can get them if you connect to the server and issue that command)

BTW. Synapse has some functionality build in to do all this.

----------------------------------------------------------------------
Hartmut · 2023-06-14 14:44

The problem seems to be solved!

Quote from: rvk on June 13, 2023, 08:14:18 pmIf you want to do this
manually you can do this:
Code: Bash  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i openssl
s_client -connect imap.gmx.net:993 -quieta1 login username passworda2
select inboxa3 uid search allYou get a list with UIDs (which for me
definitely are not in the lower range).

I did this and got this list:
Code: Text  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i a3 uid
search all* SEARCH 8168 8706 8707 8708 8709 8710 8711 8712 8713 8714
8715 8716 8717 8718 8719 8720 8721 8722 8723 8724...8872 8873 8874
8875 8876 8877a3 OK UID SEARCH completedAn when I passed one of these
numbers to my function Test_read_IMAP() it worked successfully.
So rvk was right, that after '/;UID=' a UID from above list has to be
passed and not a relative mail-index, like I did.

Because I want to get a mail-header not for a UID, but for a relative
mail-index, I found with google the right syntax: I only had to change
line 68 in my demo from
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
url:=url_imap + '/' + imap_folder + '/;UID=' + IntToStr(nr) +
'/;SECTION=HEADER';to
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
url:=url_imap + '/' + imap_folder + ';MAILINDEX=' + IntToStr(nr) +
';SECTION=HEADER';and now my demo works as I want.

I made a quick test with this change in my bigger real-world-program
and it worked. So the problem seems to be solved (needs more deep
testing in the next days).

Thanks a lot to rvk for his help. This is a great forum.