Microsoft Translation + Synapse
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
bobkos · 2019-04-10 08:11

Hello everyone,

I'm trying to use Microsoft cognitive translation service in my app in
accordance with API described here and here is details. With the
following example code:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i unit
Unit1; {$mode objfpc}{$H+} interface uses  Classes, SysUtils, Forms,
Controls, Graphics, Dialogs, StdCtrls,  HttpSend, ssl_openssl,
ssl_openssl_lib; type   { TForm1 }   TForm1 = class(TForm)    Button1:
TButton;    procedure Button1Click(Sender: TObject);  private   
FWebSession:THTTPSend;  public   end; var  Form1:
TForm1; implementation {$R *.lfm} { TForm1 } procedure
TForm1.Button1Click(Sender: TObject);var  ADocString:string;begin 
FWebSession:=THTTPSend.Create;  with FWebSession do begin    Clear;  
Headers.Add('Ocp-Apim-Subscription-Key: <HERE IS MY VALID KEY>');   
Headers.Add('Content-Type: application/json');   
ADocString:='[{"Text":"car"}]';   
Document.WriteBuffer(Pointer(ADocString)^,Length(ADocString));   
Headers.Add(Format('Content-Length: %d',[Document.Size]));   
Document.Position:=0;    if HTTPMethod('POST','https://api.cognitive.m
icrosofttranslator.com/translate?api-version=3.0&from=en&to=fr') then
begin      Document.SaveToFile('answer.json');    end;  end; 
FreeAndNil(FWebSession);end; end. 
Howere I'm getting server's error HTTP/1.1 400 Bad Request
Code: [Select]--Header
HTTP/1.1 400 Bad Request
Content-Type: text/html; charset=us-ascii
Date: Wed, 10 Apr 2019 07:54:29 GMT
Connection: close
Content-Length: 339
--Body
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML
4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-
ascii"></HEAD>
<BODY><h2>Bad Request - Invalid Header</h2>
<hr><p>HTTP Error 400. The request has an invalid header name.</p>
</BODY></HTML>

I believe that the problem is somewhere around Document(TMemoryStream)
(not in format itself), because when I'm using Microsoft's search
service which is not requirying any docs to be post, only headers and
params via URL then it's work perfectly.

Just for your information the folowing curl link work without
troubles:
Code: [Select]curl -X POST
"https://api.cognitive.microsofttranslator.com/translate?api-
version=3.0&from=en&to=fr" -H "Ocp-Apim-Subscription-Key: <HERE IS MY
VALID KEY>" -H "Content-Type: application/json" -d "[{'Text':'car'}]"

Can anyone suggest what I'm doing wrong?
Thank you.

----------------------------------------------------------------------
bobkos · 2019-04-12 01:21

So no any ideas? What is the different between
THttpSend.Document.LoadFromFile and curl's -d -@word.txt? Why curl is
accepting while THttpSend is not?

----------------------------------------------------------------------
Thaddy · 2019-04-12 07:39

AllowRedirect?

----------------------------------------------------------------------
bobkos · 2019-04-13 08:53

Sorry I didn't get you. Could you expand please?

----------------------------------------------------------------------
Thaddy · 2019-04-13 09:36

The microsoft website  is a front-end server that  uses redirections
to other  ms webservers,. In Synapse redirections are not allowed by
default, but you can set Thttpsend.AllowRedirect to true, I believe by
now.

Otherwise use this example:
https://www.matthewhipkin.co.uk/codelib/follow-http-redirects-using-
ararat-synapse/

It may also be that you are using and old version of synapse and/or
OpenSSL
You need the trunk version of synapse at
https://sourceforge.net/projects/synalist/ and a recent version from
OpenSSL and select the TLS1.2 protocol as default. Azure doesn't
support ssl2,ssl3, tls 1.0 and tls1.1. These are deprecated because
severe security issues.

----------------------------------------------------------------------
bobkos · 2019-04-13 11:01

Thank you Thaddy

In case of redirection I should get something like 301 or 302
ResultCode isn't it? But I'm getting 400 error. So I suppose there is
no any redirection.
Meanwhile I'm using latest versions of Synapse and OpenSSL.

As I mentioned above in case of using another service (search for
example) it's working well, the only one difference between this two
cases is that I must post json request in case of translator service.
Therefore I suppose that in case of any incompatability I couldn't
perform even searching request, don't you think so?

Anyway appreciate your advise. Maybe there are some other reasons
could be? Or is there any possibility to analyze how curl sending the
data?

----------------------------------------------------------------------
Thausand · 2019-04-13 11:18

Quote from: bobkos on April 13, 2019, 01:01:22 pmOr is there any
possibility to analyze how curl sending the data?
- program curl have manual debug https://curl.haxx.se/docs/manual.html
- program wireshark https://www.wireshark.org/
- online https://requestbin.com/
- online http://httpbin.org/

Important write: not send key online test.

add:
I write before to many quote. now correct.

----------------------------------------------------------------------
Thaddy · 2019-04-13 11:28

https://www.lifewire.com/how-to-fix-a-400-bad-request-error-2617988
which would fit bad protocol.

----------------------------------------------------------------------
bobkos · 2019-04-13 11:57

Thanks Thausand and Thaddy

Will go deeper, and try find what is going wrong.

----------------------------------------------------------------------
Thausand · 2019-04-13 13:38

Deep no take many time.

netcat
Code: [Select]nc -l 9000

request localhost
Code: [Select]curl -X POST "http://localhost:9000" -H "Ocp-Apim-
Subscription-Key: <HERE IS MY VALID KEY>" -H "Content-Type:
application/json" -d "[{'Text':'car'}]"

netcat write
Code: [Select]POST / HTTP/1.1
Host: localhost:9000
User-Agent: curl/7.52.1
Accept: */*
Ocp-Apim-Subscription-Key: <HERE IS MY VALID KEY>
Content-Type: application/json
Content-Length: 16

[{'Text':'car'}]


curl verbose
Code: [Select]curl -v -X POST
"https://api.cognitive.microsofttranslator.com/translate?api-
version=3.0&from=en&to=fr" -H "Ocp-Apim-Subscription-Key: <HERE IS MY
VALID KEY>" -H "Content-Type: application/json" -d "[{'Text':'car'}]"
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 40.90.141.99...
* TCP_NODELAY set
* Connected to api.cognitive.microsofttranslator.com (40.90.141.99)
port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection:
ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.microsofttranslator.com
*  start date: Mar 20 16:45:38 2018 GMT
*  expire date: Mar 20 16:45:38 2020 GMT
*  subjectAltName: host "api.cognitive.microsofttranslator.com"
matched cert's "*.cognitive.microsofttranslator.com"
*  issuer: C=US; ST=Washington; L=Redmond; O=Microsoft Corporation;
OU=Microsoft IT; CN=Microsoft IT TLS CA 1
*  SSL certificate verify ok.
> POST /translate?api-version=3.0&from=en&to=fr HTTP/1.1
> Host: api.cognitive.microsofttranslator.com
> User-Agent: curl/7.52.1
> Accept: */*
> Ocp-Apim-Subscription-Key: <HERE IS MY VALID KEY>
> Content-Type: application/json
> Content-Length: 16
>
* upload completely sent off: 16 out of 16 bytes
< HTTP/1.1 401 Unauthorized
< Content-Type: text/plain; charset=utf-8
< X-RequestId: TRAN.13B3.B1E9A8B8
< Access-Control-Expose-Headers: X-RequestId
< X-Content-Type-Options: nosniff
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Date: Sat, 13 Apr 2019 13:29:27 GMT
< Content-Length: 111
<
* Curl_http_done: called premature == 0
* Connection #0 to host api.cognitive.microsofttranslator.com left
intact
{"error":{"code":401000,"message":"The request is not authorized
because credentials are missing or invalid."}}


curl debug
Code: [Select] curl --trace trace.txt -X POST
"https://api.cognitive.microsofttranslator.com/translate?api-
version=3.0&from=en&to=fr" -H "Ocp-Apim-Subscription-Key: <HERE IS MY
VALID KEY>" -H "Content-Type: application/json" -d "[{'Text':'car'}]"
Note: Unnecessary use of -X or --request, POST is already inferred.
{"error":{"code":401000,"message":"The request is not authorized
because credentials are missing or invalid."}}
debug read file trace.txt (i no write)

Can make more good if have key

----------------------------------------------------------------------
bobkos · 2019-04-13 15:16

trace file is attached.
everything seems the same...

----------------------------------------------------------------------
Thausand · 2019-04-13 15:26

Quote from: bobkos on April 13, 2019, 05:16:26 pmtrace file is
attached.
everything seems the same...
You attach request send curl and have answer 

Now try trace use you program and see differ ...

I hint: it in header

----------------------------------------------------------------------
bobkos · 2019-04-13 15:41

Thanks. I see the difference in header, and already tried made it the
same as curl, but it is not gave positive result. By the way,
according to Microsoft link in my firdt topic, only 3 provided header
options marked as required.

----------------------------------------------------------------------
Thausand · 2019-04-13 15:46

Ok hello bobkos

Please i sorry need small time do some thing else. When i back (half)
hour then make project and test

Or maybe other user can help you if more quick me.

----------------------------------------------------------------------
Thausand · 2019-04-13 16:39

Ok. it not so easy when not know synapse (i not know and test)

If test you new program then netcat write:
Code: [Select]$ nc -l 9000
POST / HTTP/1.0
Host: localhost:9000
Keep-Alive: 300
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; Synapse)
Content-Type: text/html
Content-Length: 16
Ocp-Apim-Subscription-Key: <HERE IS MY VALID KEY>
Content-Type: application/json
Content-Length: 16

[{"Text":"car"}]

Error for have two time content-type is you add header.

Synapse use FWebSession.MimeType for set content-type.

Try have remove program
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
Headers.Add('Content-Type: application/json');and add program
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
FWebSession.MimeType:= 'application/json';
Then try 

I think other find differ alone compare curl and that header Accept.
So if not work try add Accept header. Also is differ UserAgent but
that not problem i think (but can try also).

----------------------------------------------------------------------
bobkos · 2019-04-13 22:33

Unfortunaterly No.

In case if I put
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
FWebSession.MimeType:= 'application/json';then the following reply
Code: [Select]{"error":{"code":415000,"message":"The Content-Type
header is missing or invalid."}}I suppose that when we have got answer
code 415, apart of 400 then previously header was ok and problem is
somewhere else.

Also not help
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
Headers.Add('Accept: */*'); and changing UserAgent. All this options I
have tried previously.

----------------------------------------------------------------------
Thausand · 2019-04-13 23:43

I Sorry bobkos.

I forget because i not see first.
Add
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  
FWebSession.Protocol:='1.1';
Other thing i change was make
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  
FWebsession.KeepAlive:= false;But maybe not want.

I read curl not set "connection" for header and i try remove from
synapse. I only can make KeepAlive true or false and synapse header
any time show "connection" in header. It no matter if connection
header there or not.

No need change UserAgent.

Quote{"error":{"code":415000,"message":"The Content-Type header is
missing or invalid."}}
I think that is error you code because i not get this error.

Have you look precise you code and set mimetype after
FWebSession.Clear line ?

Add:
This is work me:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TForm1.Button1Click(Sender: TObject);var  FWebSession:THTTPSend; 
ADocString:string;begin  FWebSession:=THTTPSend.Create;  
FWebSession.Clear;  FWebSession.Protocol:='1.1'; 
FWebSession.MimeType:='application/json';  
FWebSession.Headers.Add('Ocp-Apim-Subscription-Key: <HERE IS MY VALID
KEY>');   ADocString:='[{"Text":"car"}]';  FWebSession.Document.WriteB
uffer(Pointer(ADocString)^,Length(ADocString)); 
FWebSession.Document.Position:=0;  
FWebSession.Headers.Add(Format('Content-Length:
%d',[FWebSession.Document.Size]));   if FWebSession.HTTPMethod('POST',
'https://api.cognitive.microsofttranslator.com/translate?api-
version=3.0&from=en&to=fr') then  begin   
FWebSession.Document.SaveToFile('answer.json');  end;  
FreeAndNil(FWebSession);end;

----------------------------------------------------------------------
bobkos · 2019-04-14 11:54

You did it! Thank you Thausand.

Now I get what was wrong... When I declared type according to your
advise, instead of Code: Pascal  [Select][+][-]window.onload =
function(){var x1 = document.getElementById("main_content_section");
if (x1) { var x = document.getElementsByClassName("geshi");for (var i
= 0; i MimeType:= 'application/json' it was Code: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i MimeType:=
'Content-Type: application/json'
That was my stuipid error!

I really appreciate your support.

----------------------------------------------------------------------
Thausand · 2019-04-15 06:57

Quote from: bobkos on April 14, 2019, 01:54:23 pmThat was my stuipid
error!
He he. Is no problem. I make many error to and maybe i not write good
for you understand what i mean.

QuoteI really appreciate your support.
That ok.

You forum write error and make help me better learn english and
pascal/lazarus when try fix error you write.

I thank you for help me