Synapse, OAuth2, Imap & Gmail
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
Frank · 2019-01-22 19:55

Hi All,
  Has anyone had any success with Synapse(svn), OAuth2, Imap & Gmail?
  I stumbled across a great jump point by rvk (google_oauth2.pas &
example), but that only deals with SMTP. As well I found a
Delphi/Win32/Indy10 example. I am looking for a Win/Mac solution using
Lazarus/Synapse. I can get Imap to work IF I turn on "Allow Less
Secure App Access", but I'd rather not do that... OAuth2 seems to be
the way to go.

I really like rvk's approach, it would be nice to continue that, if
possible.

Any Ideas? Frank

----------------------------------------------------------------------
rvk · 2019-01-22 20:15

My google_oauth2.pas can also be used to access imap and even google
drive.

I use it to put a concept mail via imap into the drafts-folder of
gmail, before the user sends it off.

You can use the access_token retrieved via that unit to gain access.

----------------------------------------------------------------------
Frank · 2019-01-22 20:20

@rvk : Thanks, is there a chance I'm using an old example ? Where
might I find the most up-to-date? Frank

EDIT: Last recorded edit in 'google_oauth2.pas' was, my version,
2016-12-12.
         I am able to make SMTP/Google drive work with
google_oauth2.pas.

----------------------------------------------------------------------
rvk · 2019-01-22 20:49

Yeah, it's been a while since I did anything with the original source.
The latest is at https://github.com/rvk01/google-oauth2

In there I showed with a TmySMTPSend how you should do the AUTH
XOAUTH2 with the GetXOAuth2Base64 (which was a combination of EMail
and the Access_token) via Gmail SMTP.

With Gmail IMAP it's a similar process. Below is my TmyIMAPSend you
can use (which is inherited from the Synapse one).

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i // ========
=====================================================================t
ype  TmyIMAPSend = class(TIMAPSend)  public    OAuth2: String;   
function Login: Boolean;    function DoXOAuth2(const Value: string):
Boolean;    function ChallengeError(): string;  end; function
TmyIMAPSend.Login: Boolean;var  S: string;begin  FSelectedFolder :=
'';  FSelectedCount := 0;  FSelectedRecent := 0;  FSelectedUIDvalidity
:= 0;  Result := false;  FAuthDone := false;  if not Connect then    
Exit;  S := string(FSock.RecvString(FTimeout));  if Pos('* PREAUTH',
S) = 1 then      FAuthDone := true  else    if Pos('* OK', S) = 1
then      FAuthDone := false  else      Exit;  if Capability then 
begin    // * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID
XLIST CHILDREN X-GM-EXT-1    // XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN
AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER AUTH=XOAUTH    //
Showmessage(FullResult.Text);    if Findcap('IMAP4rev1') = '' then   
   Exit;    if FAutoTLS and (Findcap('STARTTLS') <> '') then      if
StartTLS then          Capability;  end;   Result :=
Self.DoXOAuth2(OAuth2);  // Result := AuthLogin; end; function
TmyIMAPSend.DoXOAuth2(const Value: string): Boolean;var  S:
string;begin  S := IMAPcommand('AUTHENTICATE XOAUTH2 ' + Value); 
Result := S = 'OK';  // Showmessage(S);  // x := StrToIntDef(Copy(S,
1, 3), 0);  // Result := (x = 235);end; function
TmyIMAPSend.ChallengeError(): string;var  S: string;begin  Result :=
'';  Sock.SendString('' + CRLF);  repeat    S :=
string(Sock.RecvString(FTimeout));    if Sock.LastError <> 0 then    
  break;    if Result <> '' then        Result := Result + CRLF;   
Result := Result + S;  until Pos('-', S) <> 4;end;// =================
============================================================
Then you can use this to access IMAP from Google.

Here is an example from my program to push a mail into the Drafts-
folder. (It's a subroutine for a larger procedure to send mail via
Gmail IMAP, Gmail SMTP, Microsoft Office365 and regular Windows Mapi,
so some variables might be unknown in this subroutine standalone)

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   function
SendItViaGMailImap: Cardinal;  const    google_client_id =
'1111111111-xxxxxxxxxxxxxx.apps.googleusercontent.com';   
google_client_secret = 'xxxxxxxxxxxxxxxxxxxxxxx';  var    gOAuth2:
TGoogleOAuth2;    Scopes: GoogleScopeSet;  var    Imap: TmyIMAPSend; 
 ts: TStringList;    Uid: int64;    UidStr: string;    Ok: Boolean;  
Match: TMatch;    Url: String;  begin    Result := MAPI_E_FAILURE;   
ts := TStringList.Create;    Imap := TmyIMAPSend.Create;    gOAuth2 :=
TGoogleOAuth2.Create(google_client_id, google_client_secret);    try  
    Scopes := [GoogleScope.goMail];      gOAuth2.refresh_token :=
CipherString(ReadRegistry('GMail', 'refresh_token', ''),
'my_app_name', false);      gOAuth2.access_token :=
CipherString(ReadRegistry('GMail', 'access_token', ''), 'my_app_name',
false);      gOAuth2.GetAccess(Scopes, false); // <- DO NOT get from
file      if gOAuth2.email <> '' then      begin       
DeleteRegistry('GMail', 'Gebruikersnaam');       
DeleteRegistry('GMail', 'Wachtwoord');        WriteRegistry('GMail',
'refresh_token', CipherString(gOAuth2.refresh_token, 'my_app_name',
true));        WriteRegistry('GMail', 'access_token',
CipherString(gOAuth2.access_token, 'my_app_name', true));      end   
 else      begin        Result := MAPI_E_INVALID_SESSION;       
Exit;      end;       //
https://developers.google.com/gmail/imap_extensions?csw=1     
Imap.AutoTLS := false;      //
https://www.limilabs.com/blog/oauth2-gmail-imap-service-account     
Imap.Username := '';      Imap.Password := '';      Imap.OAuth2 :=
gOAuth2.GetXOAuth2Base64;      Imap.TargetHost := 'imap.gmail.com';  
  Imap.TargetPort := '993';      Imap.FullSSL := true;     
Imap.Sock.SSL.SSLType := LT_all;      // Imap.Sock.SSLDoConnect();   
 if not Imap.Login() then      begin        ShowMessage('Fout: ' +
Imap.ChallengeError);        Result := MAPI_E_LOGIN_FAILURE;       
Exit;      end;       (*        Imap.List('*', ts);       
ShowMessage(ts.Text);        ShowMessage(Imap.SelectedFolder);     
*)       Ok := Imap.SelectFolder('[Gmail]/Concepten');      if not Ok
then Ok := Imap.SelectFolder('[Gmail]/Drafts');      if not Ok then  
  begin        Imap.Logout();        Result :=
MAPI_E_INVALID_SESSION;        Exit;      end;       ts.Text :=
CreatePlainEML;      Imap.AppendMess(Imap.SelectedFolder, ts);      //
S5 OK [APPENDUID 628624526 53] (Success)      Match :=
TRegEx.Match(Imap.ResultString, '\[APPENDUID (.*?) (.*?)\]');      if
Match.Success and (Match.Groups.Count = 3) then      begin       
UidStr := Match.Groups[2].Value;         if Imap.IMAPcommand('UID
FETCH ' + UidStr + ' (X-GM-MSGID)') = 'OK' then        begin         
// * 13 FETCH (X-GM-MSGID 1590941195342179459 UID 61)          Match
:= TRegEx.Match(Imap.FullResult.Text, '\(X-GM-MSGID (.*?) UID
(.*?)\)');          if Match.Success and (Match.Groups.Count = 3)
then          begin            UidStr := Match.Groups[1].Value;      
    Uid := StrToInt64(UidStr);             Url :=
'https://mail.google.com/mail/#drafts?compose=' +
lowercase(IntToHex(Uid, 16));            Url :=
'https://mail.google.com/mail?authuser=' + gOAuth2.email + '#drafts/'
+ lowercase(IntToHex(Uid, 16));             // Url :=
'https://mail.google.com/mail/#drafts/' + lowercase(IntToHex(Uid,
16));            //
https://mail.google.com/mail/u/0/#drafts/%23thread-a%3ammiai-xxxx    
      // https://mail.google.com/mail/u/0/#drafts?compose=xxx        
   BrowseURL(Url);            Result := SUCCESS_SUCCESS;          
end;        end;      end;       Imap.Logout();     finally     
Imap.Free;      ts.Free;      gOAuth2.Free;    end;   end;
Hope you can get it working.
(source is from my Delphi application but it should work in Lazarus
too)

If you can't get it to work I could expand the example on github to
include IMAP (when I get some time).

----------------------------------------------------------------------
Frank · 2019-01-22 21:14

@rvk : And presto, It is working !!! Thank you for your help !!! Frank

----------------------------------------------------------------------
rvk · 2019-01-22 21:43

Great.

One note: I got an e-mail recently that Google is updating their
policies regarding the scopes you can use and what scopes are needing
their approval. The https://mail.google.com/ scope (full access) needs
extra approval and you need to provide a privacy policy to include on
the authentication screen for the user. You could use the
gmail.compose scope (for just accessing drafts and sending mail) but
that also need their approval. If you don't ask approval the user gets
an extra security screen. I think this will apply from 15 februari
onwards. You can see in the authentication-screen-designer if you need
such approval.

QuoteIn October 2018, we announced that, in January 2019, new Gmail
API policies for restricted scopes will go into effect. We want to let
you know that, starting today, you can submit your app(s) that use
restricted scopes for verification. Please review the full policy and
OAuth FAQ for more information including the secure handling
requirement.https://developers.googleblog.com/2018/10/elevating-user-
trust-in-our-api.html

----------------------------------------------------------------------
Frank · 2019-01-23 10:59

@rvk : Once again thanks, my project is aimed at a closed audience,
but it's nice to know in advance.
           By the way, is pop3 possible with a similar approach?

  Frank

----------------------------------------------------------------------
rvk · 2019-01-23 11:20

I'm not sure about POP3. You might not even need to set "allow less
secure apps" to access Gmail via POP3.

B.T.W. Instead of setting "allow less secure apps" when accessing
Gmail via SMTP you can also create a specific password for your
application. You can do that here:
https://myaccount.google.com/apppasswords. That way you can use SMTP
without the fancy authentication method shown in the github example.
It's just what you prefer.

So you have:
Sending mail:
*) SMTP with setting "allow less secure apps" (not advised anymore)
*) SMTP with special created password for specific app (slightly
better solution)
*) SMTP with authentication via authorize screen and access_token (see
github example)
*) IMAP with authentication via authorize screen and access_token (see
above example)
*) Directly pushing mail via Gmail API-calls (see below example)

Reading mail:
*) POP3 (not sure if you need any special setting)
*) IMAP with authentication via authorize screen and access_token
*) Directly reading mail via Gmail API-calls (see documentation)

Direct Gmail API-call (JSON-requests):
The method I've shown earlier used IMAP. For IMAP you need the full
access scope (https://mail.google.com/). You can however also use API-
calls and push a JSON to the Gmail server. In that case you don't need
full access but can use gmail.compose. Here is an example. (I'm not
sure if it works completely because it's been a while since I tried
this but you see how it's done)

(You see here that a header "Authorization: Bearer" is used for
authentication)

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   function
SendItViaGMailAPI: Cardinal;  const    google_client_id =
'1111111111-xxxxxxxxxxxxxx.apps.googleusercontent.com';   
google_client_secret = 'xxxxxxxxxxxxxxxxxxxxxxx';  var    gOAuth2:
TGoogleOAuth2;    Scopes: GoogleScopeSet;  var    HTTP: THTTPSend;   
Response: TStringList;    obj: ISuperObject;    json: ISuperObject;  
Url: STring;   begin    Result := MAPI_E_FAILURE;    gOAuth2 :=
TGoogleOAuth2.Create(google_client_id, google_client_secret);    try  
    Scopes := [GoogleScope.goMail];      gOAuth2.refresh_token :=
CipherString(ReadRegistry('GMail', 'refresh_token', ''),
'my_app_name', false);      gOAuth2.access_token :=
CipherString(ReadRegistry('GMail', 'access_token', ''), 'my_app_name',
false);      gOAuth2.GetAccess(Scopes, false); // <- DO NOT get from
file      if gOAuth2.email <> '' then      begin       
DeleteRegistry('GMail', 'Gebruikersnaam');       
DeleteRegistry('GMail', 'Wachtwoord');        WriteRegistry('GMail',
'refresh_token', CipherString(gOAuth2.refresh_token, 'my_app_name',
true));        WriteRegistry('GMail', 'access_token',
CipherString(gOAuth2.access_token, 'my_app_name', true));      end   
 else      begin        Result := MAPI_E_INVALID_SESSION;       
Exit;      end;       // https://developers.google.com/gmail/api/v1/re
ference/users/drafts/create?hl=nl      Url :=
'https://www.googleapis.com/upload/gmail/v1/users/' + gOAuth2.email +
'/drafts';      Response := TStringList.Create;      HTTP :=
THTTPSend.Create;      try        json := SO;        // json.S['auth']
:= 'oauth2Client';        // json.S['userId'] := 'me';       
json.S['message.raw'] := CreatePlainEML;        
WriteStrToStream(HTTP.Document, ansistring(CreatePlainEML));       
json := nil;         HTTP.MimeType := 'message/rfc822';       
HTTP.Headers.Clear;        HTTP.Headers.Add('Authorization: Bearer ' +
gOAuth2.access_token);        HTTP.Headers.Add('Accept:
application/json');        if HTTP.HTTPMethod('POST', Url) then      
begin          Response.LoadFromStream(HTTP.Document);           //
gaat momenteel mis          ShowMessage(Response.Text);           obj
:= SO(Response.Text);          Url :=
'https://mail.google.com/mail?authuser=' + gOAuth2.email + '#drafts/'
+ urlencode(obj.S['message.id']);          BrowseURL(Url);          
Result := SUCCESS_SUCCESS;        end;       finally       
HTTP.Free;        Response.Free;      end;     finally       
gOAuth2.Free;    end;   end;
You can look at the API-documentation at
https://developers.google.com/gmail/api/guides/
You see in the example I used upload/gmail/v1/users/userId/drafts as
per example here:
https://developers.google.com/gmail/api/v1/reference/users/drafts/crea
te

I'm sure it's possible to read mail too (but in that case you would
again need the full access scope).

----------------------------------------------------------------------
Frank · 2019-01-23 11:57

@rvk Thank you for the information, I believe an application specific
password might be just the ticket.  I have written "Google Drive API"
specific code (& use it in my project), but the temptation to use
Synapse/imap/pop3 was there (I have legacy projects that could use it
as well)...

Unfortunately I have a day job to attend to, but I'll attempt your
suggestions tonight.

And again, Thanks 
     Frank

----------------------------------------------------------------------
Remy Lebeau · 2019-01-23 18:54

Quote from: Frank on January 23, 2019, 12:57:22 pmI believe an
application specific password might be just the ticket.

I use app-specific passwords when accessing Gmail via Indy and it
works just fine.  Will work fine in Synapse, too, and you don't need
OAuth to use it.

----------------------------------------------------------------------
rvk · 2019-01-23 19:01

One reason for using OAuth over app specific password would be user-
convenience.
Na usernane and no password to enter in the application.
A app specific password can.only be created by the user manually (as
far as I know).

But both methods should work fine.

----------------------------------------------------------------------
Frank · 2019-01-23 19:08

@Remy Lebeau : Thanks. Just setup apppassword for my legacy project &
now it works with the code that is probably 10 years old (had to use
"Allow Less Secure" before)(personal email program).

@rvk : Thanks, as above... Now you've given me '3' possibilities -
pop3/apppassword, imap/OAuth2 & GMail API...  Again, Thank you for
your time & expertise.
  
  Frank