Special characters lost with TFPCustomHTTPClient.SimpleGet
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
ironphil · 2023-10-30 00:54

I want to get the content of a web page with the SimpleGet method.
However, the string I received contains question marks (?) instead of
some accented characters (like é à è ê ï). If I go see the webpage in
a browser (Edge), the special characters are displayed properly. But
the page source shows that the special character are not coded in the
html way ("é" is directly in the page source instead of é for
example). I made sure everything is set to utf-8. Should the special
characters be encoded the html way on the server side before getting
the content of the page? Or am I missing something about encoding?

----------------------------------------------------------------------
Leledumbo · 2023-10-30 05:08

Quote from: ironphil on October 30, 2023, 01:54:15 amI want to get the
content of a web page with the SimpleGet method. However, the string I
received contains question marks (?) instead of some accented
characters (like é à è ê ï). If I go see the webpage in a browser
(Edge), the special characters are displayed properly. But the page
source shows that the special character are not coded in the html way
("é" is directly in the page source instead of é for example).
I made sure everything is set to utf-8. Should the special characters
be encoded the html way on the server side before getting the content
of the page? Or am I missing something about encoding?
I tried using it to get this page, where your question already
contains such character, it's as simple as:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses 
opensslsockets,  fphttpclient; begin  writeln(tfphttpclient.simpleget(
'https://forum.lazarus.freepascal.org/index.php/topic,65056.0/topicsee
n.html'));end. and both direct and HTML code are displayed properly:
Quote<div class="inner" id="msg_495250">I want to get the content of a
web page with the SimpleGet method. However, the string I received
contains question marks (?) instead of some accented characters (like
é à è ê ï). If I go see the webpage in a browser (Edge), the special
characters are displayed properly. But the page source shows that the
special character are not coded in the html way (&quot;é&quot; is
directly in the page source instead of &amp;eacute; for example). I
made sure everything is set to utf-8. Should the special characters be
encoded the html way on the server side before getting the content of
the page? Or am I missing something about encoding?</div>
Try to ALWAYS include a small compilable code that demonstrate the
problem. It could be a conversion lost due to use of the wrong string
type. for example.