XML CData character encoding or what?
lazarus / 1-Linux
----------------------------------------------------------------------
scasparz · 2011-12-28 02:40
Hi guys,
am using Laz 0.9.28.2-12beta on ubu11.04. Am trying to load a jpeg
onto a CData XML node. First I load the jpeg to a memorystream, then I
carry it to a an ordinary string, then I use this for the CDATA node:
CDataNode := XMLDoc.CreateCDATASection(lBuff); //where lBuff is
my string
ParentNode.AppendChild(CDataNode);
Problem is that for simple strings, it works fine. Yet when lBuff
carries anything more complex, say that jpg, although I can
SaveXMLFile, each and every time I attempt to open it through
ReadXMLFile, I get an error.
I suspect this has to do with character encoding. It is my
understanding Lazarus uses UTF-8 with the XML libs. Am I doing
something stupid or what?
Any ideas will be greatly appreciated.
----------------------------------------------------------------------
scasparz · 2011-12-28 23:52
OK, will answer my own question, for those concerned. Was not aware
that strings in XML nodes ought to be Base64 Encoded.It works now.
Sorry for it seems this is not the proper place for non Linux related
issues.
----------------------------------------------------------------------
BigChimp · 2011-12-29 13:50
scasparz, thanks for the info. If you want to, you could update the
wiki page on XML as well. I remember seeing a brief bit about CDATA
and being confused myself....
Thanks.
----------------------------------------------------------------------
scasparz · 2011-12-31 19:31
I would be more than just willing to do this, yet do you have any idea
on where I could find a wiki on 'how to author a Lazarus wiki'
someplace?
Happy a New Year for everyone
----------------------------------------------------------------------
BigChimp · 2012-01-01 08:53
Happy New Year to you and everyone else, too!
On the top left of this page you a bunch of links, one of them says
Wiki
This:
http://wiki.lazarus.freepascal.org/index.php?title=XML_Tutorial§io
n=6#Modifying_a_XML_document
is the page I'm talking about...
If you log in and press edit, you can fairly easily edit the wiki. In
the wiki, under NAVIGATION on the right, you can also click Help for
more info...
Thanks a lot for the help!
----------------------------------------------------------------------
scasparz · 2012-01-16 09:13
I have taken some time to prepare this introduction on XML with
Lazarus. This is not intended as a replacement for the original
Lazarus XML tutorial but rather as an addendum, for those concerned.
Any corrections, additions or suggestions will be strongly welcomed.
XML stands for eXtended Markup Language and represents a way of
providing structured plain text with self-explained markers known as
tags. XML bears strong similarity but differs from HTML which it
merely complements rather than compete against.
There are countless applications for XML starting with databases and
this includes namespaces too -LDAP is one such-, to configuration
files and process communication or whatever.
0. Some background for the novice
An XML document fundamentally consists of two parts:
the prolog, and the document element
A typical prolog goes like that:
A document element on the other hand consists of multiple nodes or
elements in a tree structure form like this:
root node
child node A
child node A1
child node A2
child node B
child node C
Just like in every tree structure, identation provides a readable way
of representing the structure.
Just like in most tree structures, each node has exactly one parent
node, with one obvious exception: the root node is always an orphan.
Each node may, but it also may not provide for a number of child-
nodes.
In the example above nodes A, B and C are all root node children. Node
A is provided with two children: that is nodes A1 and A2, while nodes
B and C are childless.
In the text that follows I will not make any distinction between the
root node and the document element, but I will use those two terms
interchangeably.
Typically a node starts and ends with a tag. Anything enclosed between
characters "<" and ">" represents a tag. Consider the following
example:
this is text, where
mytag is the name of a node which starts with the opening tag,
while it ends with the closing tag.
"this is text" is a name-less textnode child contained inside the
mytag node, in other words it is a mytag child.
Node names are case sensitive but can repeated within the XML tree
structure multiple times, thus having synonyms. The node value is
always a plain text string, there is no provision for representing any
other type, whether simple -say an integer- or structured. Using the
'<' or '>' inside a node value would most probably confuse any XML
parser, thus being a serious error to be avoided. As a final note,
nodes can and should be properly nested.
The XML standard itself provides that nodes can be of various kinds.
The following is another example:
Pecacastelo.jpg1.swf
Node Peca is a node which contains as a child the Pecacastelo.jpg1.swf
which is a text node.
The difference here is that the opening tag contains a couple of
attributes with names DestinoX and DestinoY and values the strings "0"
and "1" respectively.
Had node Peca not been provided with text-child-node, it would take a
much simpler form:
Notice the absence of an end tag as the character combination "/>"
provides for such.
There is no limit for the length of a node text. Similarly, there is
no limit for the number of attributes attached onto a node. However
there is an one-to-one match between an attribute name and an
attribute value. Again attribute values are plain text strings bearing
no XML formating characters.
The following example stands for a CData sectioned node:
In here our nome named node contains a CData section with data
Fernando Colombo. CData sectioned nodes are used for example to store
BLOB fields which may but also may not be, in plain text. It must be
emphasized here that any CData-section data should be in plain
readable text even when the data it stores are not. Implying that to
store non-plain readable text information on a CData-section data
-such as the contents of a BLOB field-, some kind of transformation
should be made. Enter the Base64 strings.
Lazarus supports Base64 strings through the Base64 unit. However until
FPC 2.6.0 no conversion functions to and from Base64 were provided. I
will include -kudos to markov- those two string conversion functions:
EncodeStringBase64, andDecodeStringBase64as an addendum by the end of
this post, for those who like me still work with FPC 2.4.0 Both of
these come ready-made on the Base64 unit which ships under FPC 2.6.0
1. XML Programming interfaces
There are two programming interfaces for manipulating XML.
The DOM standard (Document Object Model) originated by microsoft
-which like most of their works is extremely resource hungry-, seems
to be the most commonly used. Under DOM the entire XML document is
stored onto memory from where it gets parsed and manipulated.
Alternatively we also have SAX (Simple API for XML) which AFAIK is
currently not supported on Lazarus, yet I may well be wrong on this.
Had this been true, a package known "SAX for Pascal package" could be
considered as an alternative.
I presume readers by this stage may have noticed XML bears some
resemblance -while much different of course- with the old trusted and
proven INIFiles.
2. Manipulating XML with Lazarus
Lazarus -and FPC- support XML through the XMLRead, XMLWrite and DOM
units.
The core structure for an XML document is of course the tXMLDocument
class. Nodes are represented mostly by the tDOMNode class.
2.1 Steps on how to author an XML Document with Lazarus
Authoring an XML document under Lazarus is a five step procedure:
Start by creating a tXMLDocument instance.Proceed by creating the node
to be used as the root node, that is the document element, to be
appended by the tXMLDocument instance. This starts with the creation
of the root-node just like any other, it goes by appending it by the
tXMLDocument instance, before promoting the node as the document
element.
One-by-one create and populate the root-node children nodes and insert
them under the root node.Save the tXMLDocument instance and
finallyclean up any created instances.
Step 1: To create a tXMLDocument instance on lXMLDoc:
lXMLDoc := tXMLDocument.Create;
Step 2: root node:
lRootNd := lXMLDoc.CreateElement('text tag for the root node');
lXMLDoc.AppendChild(lRootNd); // append it under lXMLDoc
lRootNd := lXMLDoc.DocumentElement; // promote as the document
element
Step 3:
To create an ordinary node:
lNd := lXMLDoc.CreateElement('tag text');
// this is going to look like
It should be noted here that this way, the node is owned by the
document instance, hence there appears to be no reason to destroy the
node explicitly.
To create a text node:
lNd := lXMLDoc.CreateTextNode('node text here'); // note missing
node name
To create a CData-Section:
lNd := lXMLDoc.CreateCDataSection('enter your Base64 text here');
To insert an attribute to a node:
tDOMElement(lNd).SetAttribute('attribute name', 'attribute value');
To insert any kind of child node under any other node:
lParentNd.AppendChild(lChildNd);
Step 4:
WriteXMLFile(lXMLDoc, 'filename.xml');
Step 5:
FreeAndNIL(lXMLDoc);
2.2 Example in authoring an XML Document with Lazarus
The following procedure reads a .bmp picture to produce an elementary
XML document with just two nodes.
Each node named 'picture' will have just two elements:
a node named 'filename', which will contain the .bmp filename as an
attribute called 'fn' andthe .bmp itself as a CData-SectionThe
document element will be a node named db.
This is how our XML document will look like at the end.
Let me remind you again that -under DOM- the entire XML tree is loaded
onto RAM, hence this could quickly drain your system resources in
conjunction with Base64 encoded BLOB fields. Unit Base64 should be
included with your USES.
Code:
PROCEDURE Encode;
VAR
lXMLDoc : tXMLDocument;
lRootNode : tDOMNode;
lMemoryStream : tMemoryStream;
FUNCTION MemoryStreamAsString(vms: TMemoryStream): string;
// this will be used to extract the contents of the .bmp
// as an ordinary string
BEGIN { MemoryStreamAsString }
SetString(Result, vms.Memory, vms.Size)
END; { MemoryStreamAsString }
PROCEDURE LF_InsertBMP(vFileName: tFileName);
// this one inserts one .bmp node
VAR
lBMPNode : tDOMNode;
lBMPNameNode : tDOMNode;
lBMPCDataNode : tDOMNode;
lBuff : String; // common string with the jpg
info
lEncBuff : String; // it's Base64 equivalent
BEGIN { LF_InsertBMP }
// start by loading the .bmp onto lBuff
lMemoryStream.Clear;
lMemoryStream.LoadFromFile(vFileName);
lBuff := MemoryStreamAsString(lMemoryStream); // extract .bmp
contents as a common string
lEncBuff := EncodeStringBase64(lBuff); //finally encode the .bmp
contents in a Base64 string which could be stored on a CData-section
node
// build the .bmp node
lBMPNode := lXMLDoc.CreateElement('picture');
lBMPNameNode := lXMLDoc.CreateElement('filename');
tDOMElement(lBMPNameNode).SetAttribute('fn',
ExtractFileName(vFileName));
lBMPNode.AppendChild(lBMPNameNode);
lBMPCDataNode := lXMLDoc.CreateCDATASection(lEncBuff);
lBMPCDataNode.AppendChild(lCDataNode);
lRootNode.AppendChild(lBMPNode)
END; { LF_InsertBMP }
BEGIN { Encode }
TRY
// initialization
lMemoryStream := tMemoryStream.Create;
// create the lXMLDoc instance and prepare the root node
lXMLDoc := tXMLDocument.Create;
lRootNode := lXMLDoc.CreateElement('db');
lXMLDoc.AppendChild(lRootNode);
lRootNode := lXMLDoc.DocumentElement;
// read the pictures one-by-one and insert them under the root-
node
LF_InsertBMP('Apollo 16 (1024x768).bmp');
LF_InsertBMP('Astronaut & American Flag on Moon
(1024x768).bmp');
// finally write down the document
WriteXMLFile(lXMLDoc, 'db.xml')
FINALLY
// cleanup
FreeAndNIL(lMemoryStream);
FreeAndNIL(lXMLDoc)
END { finally }
END; { Encode }
Am fully aware that by using the TBase64EncodingStream as well as the
TBase64DecodingStream classes provided through the Base64 unit, the
above example could be further tightened as a code, however I believe
that as it is, serves better the causes it was created for.
2.3 Reading an XML document with Lazarus
To begin with you should read the entire XML document onto an
tXMLDocument instance using the ReadXMLFile function. This function
will create a tXMLDocument instance as well, yet this instance will
need to be explicitly destroyed when we finish.
To get the Document Element on a tDOMNode instance you should use the
DocumentElement property, eg
lRootNd := tXMLDoc.DocumentElement;
To get the root node's first node, you should use the ChildNodes.Items
property, eg
lFirstNd := lRootNd.ChildNodes.Items[0];
To get a node's value you should you should use the NodeValue
property, eg
lString := lFirstNd.NodeValue;
To get a node's first attribute you should use the
Attributes.Index[0].NodeValue propery, eg
lAttributeSting := lFirstNd.Attributes.Index[0].NodeValue;
To get the count of a node's direct first-degree children, you should
use the Childnodes.Count or Childnodes.Length properties, eg
lChildrenCountInt := lFirstNd.ChildNodes.Count;
There is no difference between Count and Length.
To get the count of a node's attributes use the Attributes.Count or
Length properties, eg
lAttributeCountInt := lFirstNd.Attributes.Count;
To extract the CDATA string from a CData-section node, use the node's
Data property, eg
lCDataStr := lNd.Data;
To locate a node named 'mytag' , you should use the FindNode method,
eg
lNd := lRootNd.FindNode('mytag');
As with authoring XML, Lazarus provides with numerous other properties
and methods which are beyond the scope of this text.
2.4 Example in reading an XML Document with Lazarus
In the following example we will read the XML document we created
before and we will extract both .bmp pictures to the './outpath'
folder.
PROCEDURE Decode;
VAR
lMemoryStream : tMemoryStream;
lXMLDoc : tXMLDocument;
lRootNode : tDOMNode;
lBMPCursor : SmallInt;
PROCEDURE LF_DecodeNode(vDOMNode: tDOMNode);
VAR
lFileName : tFileName;
lEncBuff,
lBuff : String;
BEGIN { LF_DecodeNode }
lFileName :=
vDOMNode.ChildNodes.Item[0].Attributes.Item[0].NodeValue; // extract
file name
lEncBuff := vDOMNode.ChildNodes.Item[1].NodeValue; // extract
the .bmp as a Base64 string
lBuff := DecodeStringBase64(lEncBuff); // ..before converting it
back to an ordinary string
lMemoryStream.Clear;
lMemoryStream.Write(Pointer(lBuff)^, Length(lBuff));
lMemoryStream.Position := 0;
lMemoryStream.SaveToFile('./outpath/' + lFileName)
END; { LF_DecodeNode }
BEGIN { Decode }
TRY
lMemoryStream := tMemoryStream.Create;
ReadXMLFile(lXMLDoc, 'db.xml'); // read XML doc
lRootNode := lXMLDoc.DocumentElement; // get the Document-
Element
// walk upon every root node child one-by-one
FOR lBMPCursor := 0 TO lRootNode.ChildNodes.Count - 1
DO LF_DecodeNode(lRootNode.ChildNodes.Item[lBMPCursor]) // this
is a picture node
FINALLY
// cleanup
FreeAndNIL(lMemoryStream);
FreeAndNIL(lXMLDoc)
END { finally }
END; { Decode }
3. Base64 conversion functions (for FPC < 2.6.0)
Code: [Select]FUNCTION DecodeStringBase64(CONST s:string):String;
VAR
instream,outstream: tStringStream;
decoder: tBase64DecodingStream;
BEGIN
instream := TStringStream.Create(s);
TRY
outstream := TStringStream.Create('');
TRY
decoder := TBase64DecodingStream.Create(instream, bdmmime);
TRY
outstream.copyfrom(decoder, decoder.size);
outstream.position := 0;
result := outstream.readstring(outstream.size)
FINALLY
decoder.free
END { finally }
FINALLY
outstream.free
END { finally }
FINALLY
instream.free
END { finally }
END; { DecodeStringBase64 }
FUNCTION EncodeStringBase64(CONST s:string):String;
VAR
outstream: tStringStream;
encoder: tBase64EncodingStream;
BEGIN
outstream := tStringStream.Create('');
TRY
encoder := tBase64EncodingStream.create(outstream);
TRY
encoder.write(s[1], length(s));
FINALLY
encoder.free
END; { finally }
outstream.position := 0;
result := outstream.readstring(outstream.size)
FINALLY
outstream.free
END { finally }
END; { EncodeStringBase64 }
Still the Base64 unit should be added to your code.
4. References and further reading
An excellent introduction to XML can be found at w3schools.com.
http://www.w3schools.com/xml/default.asp
Another very useful XML Tutorial from Lazarus wiki:
http://wiki.lazarus.freepascal.org/index.php?title=XML_Tutorial§io
n=6#Modifying_a_XML_document
----------------------------------------------------------------------
marcov · 2012-01-16 09:35
Quote from: scasparz on January 16, 2012, 10:13:03 am AFAIK neither
Lazarus nor Delphi provide with Base64 string transformation routines.
FPC has had stream based base64 routines for a long time (unit
base64). Since 2.6.0 there are also two simple procedures for strings
( string->string):
Code: [Select]function EncodeStringBase64(const s:string):String;
function DecodeStringBase64(const s:string):String;
----------------------------------------------------------------------
scasparz · 2012-01-16 14:22
Thanks Markov, I have implemented the necessary modifications.
regards