A Simple CGI Program - Why 'File Not Found'?
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
teamtempest · 2014-04-07 17:48

Using Lubuntu 13.10, Lazarus 1.2, FPC 2.6.2.

This is not a question about where to put a CGI executable or how to
execute it. The real question follows after a long preamble which
should (with any luck) outline what the answer isn't.

In my baby steps way I learned about Python's simple built-in web
servers. Ah! Python is already installed on Lubuntu! So I copied the
website I'm working on from WinVista to Lubuntu and put it in a
directory called 'www'.

The pages of the website are all static (that's what I'm trying to
change, but baby steps...). So I executed this command from a terminal
window in the 'www' directory.

Code: [Select]python -m SimpleHTTPServer

Now Lubuntu's default Firefox browser can access the local website via
either '127.0.0.1:8000/index.htm' (which is what the tutorial I found
said) or 'localhost:8000/index.htm' (which I suspected would also
work, and it does). All the pages work just like the actual website on
my ISP (so: one live website and two local copies, one for Windows and
one for Linux).

Incidentally this where I noticed in the terminal window log file that
Firefox spontaneously asked the server for 'favicon.ico'. Twice. The
server denied this file existed (since it didn't) - '404' both times.
Hmm. So I read up on how non-standardised all this is, sighed, created
a simple one, and put it in the root directory of all three versions
of the website. Firefox likes it. IE9 doesn't. Oh well.

Next is to try to execute CGI program from a shiny new page with its
form:

Code: [Select]<form method="post" action="cgi-bin/pricelookupcgi.exe">

which I did not expect to work - 'SimpleHTTPServer' does not have CGI
capabilities. And it didn't. The server returned a 'POST not
supported' message. Baby steps...

Okay, this time I'll try

Code: [Select]python -m CGIHTTPServer

instead. Oh, and maybe I should change that 250K Lazarus-generated
binary for Windows to the 600K Lazarus-generated binary for Linux as
well (oops!).

I still didn't expect this to work. The CGI library module in Python
automatically sends a '200 OK' response before it actually tries to
execute the script. Makes redirects within a script rather - messy -
for one thing. And as my program uses redirects, I figured I'd have to
edit that CGI module before things would behave properly. Still, I
wanted to see what would happen. Baby steps...

What I got was a message in the browser

Code: [Select]The CGI app reports an error:

File not found: '..\template\rateshow.tpl'

The thing about this particular message is that it is not coming from
the server. It is coming from my CGI program. Specifically this part:

Code: [Select]
{ show rudimentary HTML error page }
{ - our own, not the server's }

procedure UIshowerror(const mesg: String);
begin

    writeok;

    writeln( Output, '<HTML><HEAD><TITLE>CGI APP
ERROR</TITLE></HEAD><BODY>' );
    writeln( Output, '<P>The CGI application reports an error:</P>' );
    writeln( Output, '<H3>' + mesg + '</H3>' );
    writeln( Output, '</BODY></HTML>' );

end;

{ check if a file can be accessed }

function canaccess(const fname: String): Boolean;
begin

    canaccess := False;
    if ( FileExists(fname) ) then
        canaccess := True
    else
        UIshowerror( 'File not found: ' + fname );

end;

which I put in after having trouble with a local IIS7 server. Never
did figure out why it happened; never needed to after discovering the
'tinyweb' server (which runs my CGI program just fine under Windows).

But now I see the same or a similar problem is happening with a much
less sophisticated web server than IIS. At this point in the program
the user input has been accepted and all the calculations done. What
it's trying to do now is open an HTML template file and output it
after replacing various markers in the template with the results of
its calculations (sort of a primitive version of ASP).

Sez I to self, 'Self, this may be a directory separator thingy'. So
now I know about 'PathDelim' and how to use it for portability. Nope,
not the problem (all the '\'s are changed to '/'s in the error
message, though).

Nor is it a case problem - all website filenames are lower case now
(one of the first problems I discovered while playing with
'SimpleHTTPServer').

A permissions thingy? Lubuntu's file manager reports 'Anyone' can
'View', 'Only Owner' can 'Change Content' and 'Nobody' can 'Execute'.
Which I assume mean 'read, write, execute', respectively. I could even
change them using the file manger, but I would have thought universal
read access was already sufficient.

So...what next?

----------------------------------------------------------------------
engkin · 2014-04-07 23:54

Quote from: teamtempest on April 07, 2014, 07:48:23 pmSo...what next?
I tend to believe the results you received from FileExists. Use full
absolute path and see if it works.

----------------------------------------------------------------------
Leledumbo · 2014-04-08 00:06

You can't assume where your web server executes the CGI, always use
absolute path to refer to resources.

----------------------------------------------------------------------
BigChimp · 2014-04-08 14:05

Of course, you can write to a log file as well to help debug what's
going on e.g. (air code, untested, will probably saturate the ozone
layer):
Code: [Select]function canaccess(const fname: String): Boolean;
{$define cgidebug}
{$ifdef cgidebug}
var
  debuginfo: tstringlist;
{$endif}
begin

    canaccess := False;
   {$ifdef cgidebug}
    debuginfo:=tstringlist.create;
    try
      debuginfo.add('canaccess: looking for');
      debuginfo.add(fname);
      debuginfo.add('normalized path is:');
      debuginfo.add(sysutils.expandfilename(fname));
      debuginfo.savetofile('cgi.log'); // in a directory that cgi
process can write to preferably
    finally
      debuginfo.free;
    end;
  {$endif}
    if ( FileExists(fname) ) then
        canaccess := True
    else
        UIshowerror( 'File not found: ' + fname );

end;

Further debugging tips:
http://wiki.lazarus.freepascal.org/CGI_Web_Programming#Debugging_CGI

----------------------------------------------------------------------
teamtempest · 2014-04-08 15:44

Thank you for all the hints. Changing the path specification does turn
out to alter the results. Not always in the way I expect, however.
Perhaps there is something I still don't understand about the
difference between absolute and relative paths.

I changed my 'canaccess()' function to:

Code: [Select]{ check if a file can be accessed }

function canaccess(const fname: String): Boolean;
begin

    canaccess := False;
    if ( FileExists(fname) ) then
        canaccess := True
    else
        UIshowerror( 'File not found: ' + fname
                     + ' (' + ExpandFileName(fname) + ')' );

end;        

With no other changes, the reported error message became:

Code: [Select]File not found: ../template/rateshow.tpl
(/home/anton/template/rateshow.tpl)

The Python CGI webserver was started from the 'www' directory. Which
is missing from the absolute path. I decided that this meant the
relative path was interpreted in relation to the 'www' directory
(where the server started from) rather than the 'cgi-bin' directory
(where the CGI executable is).

Using my newfound knowledge of Linux paths and where my CGI app was
looking, I changed the path header of the template files from

Code: [Select]tmplPath = '..' + PathDelim + 'template' + PathDelim;

to

Code: [Select]tmplPath = 'template' + PathDelim;

reasoning that while this is still a relative path, it should now be
relative to the 'www' directory instead of its parent.

And so it is. My CGI app works as I expect. No 'File not found'
problems.

So what about absolute paths? I'm not terribly keen on the idea of
hard coding such a thing because it means potentially changing that
hard code for every place I might want to put the result. But maybe I
should try it just to check my understanding. So I changed the path
header again:

Code: [Select]tmplPath = '~' + PathDelim + 'www' + PathDelim +
'template' + PathDelim;

If I'm correct that should be a complete path from the root to my
template file. BUT...

Code: [Select]File not found: ~/www/template/rateshow.tpl
(/home/anton/www/template/rateshow.tpl)

The expanded path is absolutely correct (no pun intended). That IS
where that file is located.

Is there something I'm not understanding?

----------------------------------------------------------------------
Leledumbo · 2014-04-08 17:15

QuoteI'm not terribly keen on the idea of hard coding such a thing
because it means potentially changing that hard code for every place I
might want to put the resultUse configuration file such as TINIFile or
something else. That's what I do to all my web apps.
QuoteThe expanded path is absolutely correct (no pun intended). That
IS where that file is located.

Is there something I'm not understanding?Only the shell understands
that ~ must be expanded to /home/<username>, so you must expand it
yourself first before passing it to any API looking for files.

----------------------------------------------------------------------
teamtempest · 2014-04-09 04:17

Huh. So I played around with a few more variations of file paths:

Code: [Select]{        tmplPath = '..' + PathDelim + 'template' +
PathDelim; }
{        tmplPath = 'template' + PathDelim; }
{        tmplPath = '~' + PathDelim + 'www' + PathDelim + 'template' +
PathDelim; }
{        tmplPath = PathDelim + 'www' + PathDelim + 'template' +
PathDelim; }
{        tmplPath = 'www' + PathDelim + 'template' + PathDelim; }

Only the second variation actually worked.

The expanded version of the fourth was the same as the unexpanded
version.

The expanded version of the fifth had two consecutive 'www'
directories listed. Hmm. So it was  pre-pended with the remainder of a
complete path to the root from the directory the web server started
in. Because it's a relative path? Is that what happened in the case of
the second? That would explain why the one worked and the other
didn't.

The first gets '..' replaced by a complete path to the root minus one
directory.

The third gets '~' replaced by the home directory.

Hmm.

Dunno what shell expansion of filenames has to do with anything. I
admit I know little about them, but my limited understanding suggests
that any such expansion generally happens before a program starts
running - unless a running program is not communicating with the OS
directly but a shell intermediary?

Anyhow, clearly 'ExpandFileName()' knows how to put together a path.
That's the whole point of using it to show what's happening in the
first place, isn't it? However it manages to do so. Well then:

Code: [Select]function canfind(const basename: String): String;
var

    expname : string;

begin

    canfind := '';
    expname := ExpandFileName( basename );
    if ( FileExists(expname) ) then
         canfind := expname
    else
        UIshowerror( 'File not found: ' + basename
                     + ' (' + expname + ')' );

end;

Using this function instead of the prior 'canaccess()', both the
second and third variations work in the sense that when used my CGI
application can successfully open the template file. The first, fourth
and fifth fail with a wrong path.

My feeling at this point is that the third variation (with
'canfind()') is safer, as it should yield the same result whatever
directory a server chooses to regard relative paths as originating
from.

It still may have to be tailored to the actual deployment location
(*sigh*), but perhaps nothing a few {$IFDEF}s can't handle.

----------------------------------------------------------------------
engkin · 2014-04-09 10:06

Quote from: teamtempest on April 09, 2014, 06:17:10 amThe third gets
'~' replaced by the home directory.
..
Dunno what shell expansion of filenames has to do with anything.
..
Anyhow, clearly 'ExpandFileName()' knows how to put together a path.
Linux kernel functions do not replace/expand/understand the tilde mark
'~'. On the other hand, ExpandFileName() is an FPC/RTL function (not
OS function) that imitates the shell and does that replacement. Notice
that it will not replace ~<username> nor a directory that starts with
~<username>/ with the home directory of <username>.

----------------------------------------------------------------------
Leledumbo · 2014-04-09 15:31

QuoteBecause it's a relative path? Is that what happened in the case
of the second?Yes and yes. Basically every path that doesn't start
with / in *nix is relative path.

----------------------------------------------------------------------
teamtempest · 2014-04-10 14:22

QuoteMy feeling at this point is that the third variation (with
'canfind()') is safer, as it should yield the same result whatever
directory a server chooses to regard relative paths as originating
from.
Aw, you all know that won't work with non-*nix systems that don't
understand the '~' character. It might not even work on *nix systems
if the website root directory doesn't happen to be stored (mounted?)
somewhere in the '/user' tree (dunno; haven't tried it).

But I'm exceptionally lazy and don't want to mess around with OS
conventions any more than I have to. So I tried something that appears
to depend only on relations within the website itself.

Given this website layout:

www
  form1.htm
cgi-bin
  processform.cgi
template
  form1.tpl
  form2.tpl
  form3.tpl
  form4.tpl

'form1.htm' is what the user sees the first time the request is made.
' form1.tpl' is the same form after it's been filled in by
'processform.cgi'. It also handles the next user request (if any). The
other forms are for clarifying requests and error reporting.

Initially the forms themselves specified absolute paths to the CGI
application. The CGI application specified relative paths to the
templates. And on Windows that did not work using IIS7 as the server
because it could not find the template files. But it did work fine
using 'tinyweb' as the server.

And it didn't work on Linux using python's CGIHTTPServer. Hence this
thread.

I wrote a new function to dynamically create an absolute path to a
template file based on whether or not the current directory is the one
holding the CGI application. It assumes that the CGI application is
being called from either 'form1.htm' or a page generated from a
'*.tpl' file:

Code: [Select]{ create full path to template file }

function makeabspath(const basename: String): String;
const

     cgidir = 'cgi-bin';

     fromsiteroot = 'template' + PathDelim;
     fromcgidir = '..' + PathDelim + 'template' + PathDelim;

var

    dirname, sitepath, abspath : string;

begin

    makeabspath := '';

    dirname := AnsiLowerCase( GetCurrentDir );
    if ( Pos(cgidir, dirname) > 0 ) then
         sitepath := fromcgidir + basename
    else
         sitepath := fromsiteroot + basename;
    abspath := ExpandFileName( sitepath );
    if ( FileExists(abspath) ) then
         makeabspath := abspath
    else
        UIshowerror( 'File not found: ' + sitepath + ' (' + abspath +
')' );

end;                            

Using the python server, all the template files work correctly. This
even after I changed the "action" paths in the forms to be relative.
'form1.htm' uses 'cgi-bin/processform.cgi' but the templates use just
'processform.cgi'. It took a while to understand where the browser
considered the generated pages were in relation to the application
that produced them (roughly the time necessary to realize the address
bar had been displaying it all along). The two different "action"
paths suggest to me that the absolute template paths are in fact being
produced using both prefixes, although I haven't verified that.

The only thing not working at this point is redirection (for when the
user wants to cancel out of everything and go do something else). But
I expected that because the default python CGI server is, what shall I
say, overly optimistic about the results of calling any CGI
application.

----------------------------------------------------------------------
teamtempest · 2014-04-11 16:30

FWIW, commenting out this single line in the Python 2.7 module
'CGIHTTPServer.py'

Code: [Select]       self.send_response(200, "Script output follows")

seems to be enough to permit my script's re-directs to work as I
intend.

With that line in place, Firefox didn't seem to mind two consecutive
'200' responses (one from Python and one from my CGI script), but
didn't much like '200' followed by '303'. Commenting that line out and
saving the result with a new name in the root directory of the local
version of the website (because Python searches for names in its own
module directory first) enabled successful testing.

BTW, 'CGHTTPServer' sets a number of web-related environment variables
before executing any script, but not 'HTTP_HOST'. Just FYI...