Parsing URLs
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
mbdev · 2013-12-29 22:43

Hello,

I'm trying to parse a "short" URL into a "full" URL. This means that I
try to complete a URL from a http source.

Let me give you an example:
Website: http://en.wikipedia.org/wiki/Main_Page
some found links from the html source:
/wiki/Portal:Arts
/wiki/Portal:Biography"
/wiki/Portal:Geography
The function should be converted to a request read URL string
/wiki/Portal:Arts -> http://en.wikipedia.org/wiki/Portal:Arts

I already created some functions but they don't work in every case.
Does any librarie or function exists which can handle nearly all cases
of linking?

My first idea...
Code: [Select]function TWebcrawler.GetURL(Input: string): String;    
var
  uri: TIdURI;
  tmpResult: string ='';
begin
  if copy(LowerCase(Input),1,7)<>'mailto:' then
  begin
    if copy(LowerCase(Input),1,7)='http://' then uri :=
TIdURI.Create(Input) else uri := TIdURI.Create(fProtokoll+Input);

    if uri.Host='' then
    begin
      uri.Free;
      if Length(Input)>0 then if Input[1]<>'/' then Input:='/'+Input;
      uri := TIdURI.Create(fHost+Input);
    end;

    if uri.Protocol<>'' then tmpResult:=uri.Protocol+'://';
    if (uri.Username<>'') and (uri.Password<>'') then
tmpResult:=tmpResult+uri.Username+':'+uri.Password+'@';
    if uri.Host<>'' then tmpResult:=tmpResult+uri.Host;
    if uri.Port<>'' then tmpResult:=tmpResult+':'+uri.Port;
    tmpResult:=tmpResult+uri.Path+uri.Document;
    if uri.Params<>'' then tmpResult:=tmpResult+'?'+uri.Params;
    if copy(LowerCase(tmpResult),1,7)<>'http://' then
tmpResult:=fProtokoll+tmpResult;


    WriteInfo('Protokoll: '+fProtokoll);
    WriteInfo('Host: '+fHost);
    WriteInfo('Path: '+uri.Path);
    WriteInfo('Document: '+uri.Document);

    uri.Free;
  end;

  Result:=tmpResult;
end;  

procedure SetCurrentServer(CurrentURL: String);
var uri: TIdURI;
begin
  uri:=TIdURI.Create(CurrentURL);
  fProtokoll:=uri.Protocol+'://';
  fHost:=uri.Host;
  uri.Free;
end;

----------------------------------------------------------------------
Laksen · 2013-12-29 23:14

You can use the ParseURI function in the URIParser unit which comes
with freepascal by default. Complete parsing of URI's

----------------------------------------------------------------------
mbdev · 2013-12-30 12:45

Could you give me an code example because I thought that this
functions splits a url into parts.

http://www.freepascal.org/docs-html/fcl/uriparser/parseuri.html

----------------------------------------------------------------------
BigChimp · 2013-12-30 13:25

Wouldn't you do parseuri with your original string, add whatever
elements you want in the relevant part of TUri then do encodeuri?

----------------------------------------------------------------------
mbdev · 2013-12-30 13:51

I wrote this little function but it fails with the exception: Protocol
field is emty

Code: [Select]function ParseURL(CurrentURL, InputURL: String): string;
var uri, uriCurrent: TIdURI;
begin
  uri:=TIdURI.Create(InputURL);
  uriCurrent:=TIdURI.Create(CurrentURL);

  if uri.Protocol='' then uri.Protocol:=uriCurrent.Protocol;
  if uri.Host='' then uri.Protocol:=uriCurrent.Host;
  if uri.Username='' then uri.Protocol:=uriCurrent.Username;
  if uri.Password='' then uri.Protocol:=uriCurrent.Password;
  if uri.Path='' then uri.Protocol:=uriCurrent.Path;
  if uri.Document='' then uri.Protocol:=uriCurrent.Document;
  if uri.Port='' then uri.Protocol:=uriCurrent.Port;

  Result:=uri.URLEncode(InputURL);
  uri.Free;
  uriCurrent.Free; 
  WriteInfo('Current: '+CurrentURL);
  WriteInfo('Input:'+InputURL);
  WriteInfo('Code: '+Result);
  ReadLn;
end;

----------------------------------------------------------------------
mbdev · 2014-01-01 13:48

The result is the function URIParser.ResolveRelativeURI in the
uriparser unit