https://github.com/mark-summerfield/uxf Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Blog * Solutions + By Plan + Enterprise + Teams + Compare all + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} mark-summerfield / uxf Public * Notifications * Fork 1 * Star 64 Uniform eXchange Format (uxf) is a plain text human readable optionally typed storage format that supports custom types. It may serve as a convenient alternative to csv, ini, json, sqlite, toml, xml, or yaml. License GPL-3.0 license 64 stars 1 fork Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights mark-summerfield/uxf This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Clone HTTPS GitHub CLI [https://github.com/m] Use Git or checkout with SVN using the web URL. [gh repo clone mark-s] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @mark-summerfield mark-summerfield notes & doc ... c9b383e Oct 14, 2022 notes & doc c9b383e Git stats * 1,015 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time cpp astyle Oct 13, 2022 misc notes Oct 10, 2022 py notes & doc Oct 14, 2022 rs notes & doc Oct 14, 2022 testdata error msg change Oct 3, 2022 x doc fix Sep 24, 2022 .gitignore started new better multi-lang regression.py Sep 16, 2022 IDEAS.md notes & doc Oct 14, 2022 LICENSE started Uniform eXchange Format (uxf) Apr 9, 2022 MANIFEST renaming & improved exception Oct 13, 2022 README.md notes & doc Oct 14, 2022 regression.py regression.py now defaults to concurrent execution if not verbose Oct 7, 2022 st.sh notes & doc Oct 14, 2022 uxf.svg improved slides.py & added to slides.uxf Apr 28, 2022 uxf.vim tiny improvement Sep 19, 2022 View code [ ] UXF Overview Datatypes Terminology Minimal empty UXF Built-in Types Custom Types Formatting Bytes Strings Examples Minimal UXFs JSON CSV TOML Database Additional Examples Libraries Implementation Notes Imports BNF Supplementary Vim Support UXF Logo README.md UXF Overview Uniform eXchange Format (UXF) is a plain text human readable optionally typed storage format that supports custom types. UXF is designed to make life easier for software developers and data designers. It directly competes with csv, ini, json, toml, and yaml formats. A key advantage of UXF is its support for custom (i.e., user-defined) types. This can result in more compact, more readable, and easier to parse data. And in some contexts it may prove to be a convenient alternative to sqlite or xml. * Datatypes + Table of Built-in Types + Terminology + Minimal empty UXF + Built-in Types + Custom Types + Formatting * Examples + JSON + CSV + TOML + Database * Libraries [Python] [Rust] + Implementation Notes * Imports * BNF * Supplementary + Vim Support + UXF Logo Datatypes UXF supports the following eleven built-in datatypes. Type Example(s) Notes null ? ? is the UXF null type's literal representation. bool no yes Use no for false and yes for true. There must be an even number of bytes (:20AC 65 66 48:) case-insensitive hex digits; whitespace (spaces, newlines, etc.) optional. date 2022-04-01 Basic ISO8601 YYYY-MM-DD format. ISO8601 YYYY-MM-DDTHH[:MM[:SS]] format; datetime 2022-04-01T16:11:51 1-sec resolution no timezone support (see also Custom Types). int -192 +234 7891409 Standard integers with optional sign. real 0.15 0.7e-9 2245.389 Standard and scientific notation. str , use &, <, > include newlines> respectively. list [value1 value2 ... A list of values of any type. valueN] list [vtype value1 value2 A list of values of type vtype. ... valueN] {key1 value1 key2 A map with keys of any valid key type map value2 ... keyN and values of any type. valueN} {ktype key1 value1 A map with keys of type ktype and map key2 value2 ... keyN values of any type. valueN} {ktype vtype key1 A map with keys of type ktype and map value1 key2 value2 values of type vtype. ... keyN valueN} (ttype A table of values. Each value's type table ... ... must be of the corresponding type ... specified in the ttype, or any value ) type where no type has been specified. Note that it is also possible to represent Custom Types. Terminology * A map key-value is collectively called an item. * A "single" valued type (bool, bytes, date, datetime, int, str), is called a scalar. * A "multi-" valued type (list, map, table) is called a collection. * A list, map, or table which contains only scalar values is called a scalar list, scalar map, or scalar table, respectively. * A ttype is the name of a user-defined table type. Minimal empty UXF uxf 1 [] Every UXF file consists of a single header line (starting uxf 1, optionally followed by custom text). This may be followed by an optional file-level comment, then any ttype (table type) imports, then any ttype definitions. After this comes the data in the form of a single list, map, or table in which all the values are stored. The data must be present even if it is merely an empty list (as here), an empty map (e.g., {}), or an empty table. Since lists, maps, and tables can be nested inside each other, the UXF format is extremely flexible. Built-in Types Map keys (i.e., ktype) may only be of types bytes, date, datetime, int, and str and may not be null (?). List, map, and table values may be of any type (including nested maps, lists, and tables), unless constrained to a specific type. If constrained to a specific vtype, the vtype may be any built-in type (as listed above, except null), or any user-defined ttype, and the corresponding value or values must be any valid value for the specified type, or ? (null). Lists and tables preserve the order in which values appear. So the first value is at index/row 0, the second at index/row 1, etc. Maps are key-ordered. In particular when two keys are of different types they are ordered bytes < date < datetime < int < str, and when two keys have the same types they are ordered using < except for strs which use case-insensitive <. A table starts with a ttype. Next comes the table's values. The number of values in any given row is equal to the number of field names in the ttype. Lists, maps, tables, and ttype definitions may begin with a comment. And lists, maps, and tables may optionally by typed as indicated above. (See also the examples below and the BNF near the end). Strings may not include &, < or >, so if they are needed, they must be replaced by the XML/HTML escapes &, <, and > respectively. Strings respect any whitespace they contain, including newlines. Where whitespace is allowed (or required) it may consist of one or more spaces, tabs, or newlines in any combination. If you don't want to be committed to a particular UXF type, just use a str and do whatever conversion you want, or use a Custom Type. Custom Types There are two common approaches to handling custom types in UXF. Both allow for UXFs to remain round-trip readable and writeable even by UXF processors that aren't aware of the use of custom types as such. Here, we'll look at both approaches for three different custom types, a point and some constants which we'll treat as enumerations. uxf 1 [ { [1.4 9.8]} { [-0.7 3.0]} { [2.1 -6.3]} ] This first approach shows three points, each represented by a map with a str indicating the custom type ("Point"), and using lists of two reals for the x and y coordinates. The example also shows traffic light constants each represented by a str. uxf 1 [ { [1.4 9.8 -0.7 3.0 2.1 -6.3]} ] Since we have multiple points we've changed to a single map with a list of point values. This is more compact but assumes that the reading application knows that points come in pairs. A UXF processor has no knowledge of these representations of points or constants (or constants used as enumerations), but will handle both seamlessly since they are both represented in terms of built-in UXF types. Nonetheless, an application that reads such UXF data can recognize and convert to and from these representations to and from the actual types. uxf 1 =Point x:real y:real =TrafficLightGreen =TrafficLightAmber =TrafficLightRed [ (Point 1.4 9.8 -0.7 3.0 2.1 -6.3) (TrafficLightGreen) (TrafficLightAmber) (TrafficLightRed) ] This second approach uses four ttypes (custom table types). For the Point we specify it as having two real fields (so the processor now knows that Points have two real values). And for the enumeration we used three separate fieldless tables, i.e., three constants. Using tables has the advantage that we can represent any number of values of a particular ttype in a single table (including just one, or even none), thus cutting down on repetitive text. Here, the Point table has three Points (rows). And some UXF processor libraries will be able to return table values as custom types. (For example, the Python UXF library would return these as custom class instances--as "editable tuples".) If many applications need to use the same ttypes, it may make sense to create some shared ttype definitions. See Imports for how to do this. Formatting A UXF file's header must always occupy its own line (i.e., end with a newline). The rest of the file could in theory be a single line no matter how long. In practice and for human readability it is normal to limit the width of lines, for example, to 76, 80, or the UXF default of 96 characters. A UXF processor is expected to provide formatting options for pretty printing UXF files with user defined indentation, wrap width, and real number formatting. UXF bytes and strs can be of any length, but nonetheless they can be width-limited without changing their semantics. Bytes Any bytes value may be written with any amount of whitespace including newlines--with all the whitespace ignored. For example: (:AB DE 01 57:) [?] (:ABDE0157:) This makes it is easy to convert a bytes that is too long into chunks, e.g., (:20 AC 40 41 ... lots more ... FF FE:) to, say: (:20 AC 40 41 ... some more ... ... some more ... FF FE:) Strings Because UXF strings respect any whitespace they contain they cannot be split into chunks like bytes. However, UXF supports a string concatenation operator such that: [?] & & Which means, of course, that given a long string that might not contain newlines or whose lines are too long, we can easily split it into chunks, e.g., to, say: & & & Comments work the same way, but note that the comment marker must only precede the first fragment. # [?] # & & & < strings.> Examples Minimal UXFs uxf 1 {} We saw earlier an example of a minimal UXF file with an empty list; here we have one with an empty map. uxf 1 =Pair first second (Pair) Here is a UXF with a ttype specifying a Pair that has two fields each of which can hold any UXF value (including nested collections). In this case the data is a single empty Pair table. uxf 1 =Pair first second (Pair (Pair 1 2) (Pair 3 (Pair 4 5))) And here is a UXF with a single Pair table that contains two nested Pair tables, the second of which itself contains a nested pair. JSON JSON is a very widely used format, but unlike UXF it lacks user-defined types. Here's an example of GeoJSON data from Wikipedia: { "type": "FeatureCollection", "features": [ { "type": "Feature", "geometry": { "type": "Point", "coordinates": [102.0, 0.5] }, "properties": { "prop0": "value0" } }, { "type": "Feature", "geometry": { "type": "LineString", "coordinates": [ [102.0, 0.0], [103.0, 1.0], [104.0, 0.0], [105.0, 1.0] ] }, "properties": { "prop0": "value0", "prop1": 0.0 } }, { "type": "Feature", "geometry": { "type": "Polygon", "coordinates": [ [ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ] ] }, "properties": { "prop0": "value0", "prop1": { "this": "that" } } } ] } It would be easy to "translate" this directly into UXF: uxf 1 { : , : [ { : , : { : , : [102.0, 0.5] }, : { : } ... Naturally this works, but doesn't take advantage of any of UXF's benefits. Here's a more realistic possible UXF alternative: uxf 1 =Feature geometry properties:map =LineString x:real y:real =Point x:real y:real =Polygon x:real y:real (Feature (Point 102.0 0.5) { } (LineString 102.0 0.0 103.0 1.0 104.0 0.0 105.0 1.0) { 0.0} (Polygon 100.0 0.0 101.0 0.0 101.0 1.0 100.0 1.0 100.0 0.0) { { }} ) We don't need a FeatureCollection because UXF tables can accept zero or more values, so a Feature table is sufficient. Here's a last JSON alternative, this time avoiding the duplication of x:real and y:real: uxf 1 =Feature geometry properties:map =LineString points:Point =Point x:real y:real =Polygon points:Point (Feature (Point 102.0 0.5) { } (LineString (Point 102.0 0.0 103.0 1.0 104.0 0.0 105.0 1.0)) { 0.0} (Polygon (Point 100.0 0.0 101.0 0.0 101.0 1.0 100.0 1.0 100.0 0.0)) { { }} ) This seems like the clearest solution. CSV Although widely used, the CSV format is not standardized and has a number of problems. UXF is a standardized alternative that can distinguish fieldnames from data rows, can handle multiline text (including text with commas and quotes) without formality, and can store one--or more--tables in a single UXF file. Here's a simple CSV file: Date,Price,Quantity,ID,Description "2022-09-21",3.99,2,"CH1-A2","Chisels (pair), 1in & 11/4in" "2022-10-02",4.49,1,"HV2-K9","Hammer, 2lb" "2022-10-02",5.89,1,"SX4-D1","Eversure Sealant, 13-floz" Like with JSON we could simply "translate" this directly into UXF as a list of lists. But doing so would leave us with the same problem as .csv files: is the first row data values or column titles? (For software this isn't always obvious, for example, if all the values are strings.) Even so, this is still an improvement, since unlike the .csv representation, every value would have a concrete type (all strs for the first row, and date, real, int, str, str, for the subsequent rows). The most appropriate UXF equivalent is to use a UXF table: uxf 1 =PriceList Date Price Quantity ID Description (PriceList 2022-09-21 3.99 2 2022-10-02 4.49 1 2022-10-02 5.89 1 ) When one or more tables are used each one's ttype (table type) must be defined at the start of the .uxf file. A ttype definition begins with an = sign followed by the ttype (i.e., the table name), followed by zero or more fields. A field consists of a name optionally followed by a : and then a type (here only names are given). Both table and field names are user chosen and consist of 1-60 letters, digits, or underscores, starting with a letter or underscore. No table or field name may be the same as any built-in type name, so no table or field can be called bool, bytes, date, datetime, int, list, map, null, real, str, or table. (But Date, DateTime, and Real or real_ are fine, since names are case-sensitive and none of the built-in types contains an underscore or uses uppercase letters.) If whitespace is wanted one convention is to use underscores in their place. Once we have defined a ttype we can use it. Here, we've created a single table whose ttype is "PriceList". There's no need to group rows into lines as we've done here (although doing so is common and easier for human readability), since the UXF processor knows how many values go into each row based on the number of field names. In this example, the UXF processor will treat every five values as a single record (row) since the ttype has five fields. This is already an improvement on .csv--we know the table's name and field names, and could easily store two or more tables (as we'll see later). Although the UXF processor will correctly determine the field types, what if we want to constrain each field's value to a particular type? uxf 1 Price List =PriceList Date:date Price:real Quantity:int ID:str Description:str (PriceList 2022-09-21 3.99 2 2022-10-02 4.49 1 2022-10-02 5.89 1 ) Here we've added a custom file description in the header, and we've also added field types to the ttype definition. When types are specified, the UXF processor is expected to be able to check that each value is of the correct type. Omit the type altogether (as in the earliler examples) to indicate any valid table type. TOML Here is a TOML example from the TOML website and Wikipedia: # This is a TOML document. title = "TOML Example" [owner] name = "Tom Preston-Werner" dob = 1979-05-27T07:32:00-08:00 # First class dates [database] server = "192.168.1.1" ports = [ 8000, 8001, 8002 ] connection_max = 5000 enabled = true [servers] # Indentation (tabs and/or spaces) is allowed but not required [servers.alpha] ip = "10.0.0.1" dc = "eqdc10" [servers.beta] ip = "10.0.0.2" dc = "eqdc10" [clients] data = [ ["gamma", "delta"], [1, 2] ] # Line breaks are OK when inside arrays hosts = [ "alpha", "omega" ] And here's a possible UXF alternative: uxf 1 # =Clients a b =Database server:str ports:list connection_max:int enabled:bool =DateTime when:datetime tz:str =Owner name:str dob:DateTime =Server name:str ip:str dc:str =Hosts name:str [ (Owner (DateTime 1979-05-27T07:32:00 <-08:00>)) (Database <192.168.1.1> [8000 8001 8002] 5000 yes) (Server <10.0.0.1> <10.0.0.2> ) (Clients 1 2) (Hosts ) ] The main differences from .toml are that UXF quotes strings using <> s, and uses yes and no for bools. UXF doesn't require the use of indentation, but UXF processors default to using it for pretty printing. Unlike TOML, UXF doesn't natively support timezones, so we've created a DateTime ttype which has a when datetime and a timezone offset. For Clients the data will come in pairs because we've specified two fields. Although written compactly, we could have newlines wherever whitespace is required--or optional. There are many similar formats, including .conf, .ini, and .yaml, all of which can easily be advantageously translated into UXF. Database Database files aren't normally human readable and usually require specialized tools to read and modify their contents. Yet many databases are relatively small (both in size and number of tables), and would be more convenient to work with if human readable. For these, UXF format provides a viable alternative. A UXF equivalent to a database of tables can easily be created using a list of tables: uxf 1 MyApp Data =Customers CID Company Address Contact Email =Invoices INUM CID Raised_Date Due_Date Paid Description =Items IID INUM Delivery_Date Unit_Price Quantity Description [# (Customers 50 <123 Somewhere> 19 ? ) (Invoices 152 50 2022-01-17 2022-02-17 no 153 19 2022-01-19 2022-02-19 yes <> ) (Items 1839 152 2022-01-16 29.99 2 1840 152 2022-01-16 5.98 3 1620 153 2022-01-19 11.5 1 ) ] Here we have a list of tables representing three database tables. The list begins with a comment. Notice that the second customer has a null (?) address and the second invoice has an empty description. uxf 1 MyApp Data # =Customers CID:int Company:str Address:str Contact:str Email:str =Invoices INUM:int CID:int Raised_Date:date Due_Date:date Paid:bool Description:str =Items IID:int INUM:int Delivery_Date:date Unit_Price:real Quantity:int Description:str [# (Customers 50 <123 Somewhere> 19 ? ) (Invoices 152 50 2022-01-17 2022-02-17 no 153 19 2022-01-19 2022-02-19 yes <> ) (Items 1839 152 2022-01-16 29.99 2 1840 152 2022-01-16 5.98 3 1620 153 2022-01-19 11.5 1 ) ] Here, we've added types to each table's ttype. It is conventional in a database to have IDs and foreign keys. But these can often be avoided by using hierarchical data. For example: uxf 1 MyApp Data # =Database customers:Customers invoices:Invoices =Customers CID:int Company:str Address:str Contact:str Email:str =Invoices INUM:int CID:int Raised_Date:date Due_Date:date Paid:bool Description:str Items:Items =Items IID:int Delivery_Date:date Unit_Price:real Quantity:int Description:str (Database (Customers 50 <123 Somewhere> 19 ? ) (Invoices 152 50 2022-01-17 2022-02-17 no (Items 1839 2022-01-16 29.99 2 1840 2022-01-16 5.98 3 ) 153 19 2022-01-19 2022-02-19 yes <> (Items 1620 2022-01-19 11.5 1 ) ) ) Notice that Items no longer need an INUM to identify the Invoice they belong to because they are nested inside their Invoice. However, the relational approach has been retained for Customers since more than one Invoice could be for the same Customer. In addition, rather than using a simple list of tables, we've created a "Database" ttype and specified it as containing two tables. What if we wanted to add some extra configuration data to the database? One solution would be to add a third field to the "Database" ttype (e.g., =Database customers:Customers invoices:Invoices config:map). Or we could go further and specify a "Config" ttype and specify the third field as config:Config. Additional Examples See the testdata folder for more examples of .uxf files (some with other suffixes). See also the t and eg folders in each language-specific library (e.g., py/t and py/eg) for additional examples. Libraries Implementations in additional languages are planned. Library Language Notes uxf Python 3 See the Python UXF library. uxf Rust See the Rust UXF library. Implementation Notes If you create a UXF library please let us know so that we can add a link here (providing your library passes the regression tests!). Implmenting a UXF pretty printer whould be doable by a CS major as a final year project. Implementing a UXF parser--without support for imports, string concatenation, or aliases--should be doable by a CS major as a big final year project. Imports UXF files are normally completely self-contained. However, in some cases it may be desirable to share a set of ttype definitions amongst many UXF files. The disadvantages of doing this are: first, that the relevant UXF files become dependent on one or more external dependencies; second, it is possible to have import conflicts (i.e., two ttypes with the same name but different definitions; and third, if URL imports are used, load times will be affected by network availability and latency. (However, the first and third disadvantages don't apply if all the dependencies are provided by the UXF processor itself, i.e., are system imports.) The advantage of importing ttype definitions is that for UXF's that have lots of ttypes, only the import(s) and the data need be in the file, without having to repeat all the ttype definitions. Imports go at the start of the file after the header and after any file-level comment, and before any ttype definitions. Each import must be on its own line and may not span lines, nor have comments. If a filename import has no path or a relative path, the import attempt will be made relative to the importing .uxf file, and failing that, relative to the current folder, and failing those, relative to each path in the UXF_PATH environment variable (if it exists and is nonempty). Any ttype definition that follows an import will redefine any imported defintion of the same name. Import Notes ! complex System import of ttype Complex ! fraction System import of ttype Fraction ! numeric System import of ttypes Complex and Fraction Import the ttypes from mydefs.uxi in the importing ! mydefs.uxi .uxf file's folder, or from the current folder, or from a folder in the UXF_PATH ! /path/to/ Import the ttypes from the given file shared.uxf ! http:// www.qtrac.eu/ Import from the given URL ttype-eg.uxf Imports with no suffix (e.g., complex, fraction, numeric), are provided by the UXF processor itself. The imported file must be a valid UXF file. It need not have a .uxf suffix (e.g., you might prefer .uxt or .uxi), but must have a suffix (to distinguish it from a system import), and must have a .gz suffix if gzip compressed. Any custom string, comments, or data the imported file may contain are ignored: only the ttype definitions are used. uxf 1 !complex !fraction [(Complex 5.1 7.2 8e-2 -9.1e6 0.1 -11.2) (Fraction 22 7 355 113)] Here we've used the official system complex's Complex and fraction's Fraction ttypes without having to specify them explicitly. The data represented is a list consisting of three Complex numbers each holding two reals each, a str, and two Fractions holding two ints each. uxf 1 !numeric [(Complex 5.1 7.2 8e-2 -9.1e6 0.1 -11.2) (Fraction 22 7 355 113)] This is the same as the previous example, but using the system convenience numeric import to pull in both the Complex and Fraction ttypes. If you choose to use imports we recommed that UXF files intended for import either contain a single ttype definition or two or more imports. We recommend avoiding imports and using stand-alone UXF files wherever possible. Some UXF processors can do UXF to UXF conversions that will replace imports with (actually used) ttype definitions. (For example, the Python UXF library's uxf.py module can do this.) BNF A UXF file consists of a mandatory header followed by an optional file-level comment, optional imports, optional ttype definitions, and then a single mandatory list, map, or table (which may be empty). UXF ::= 'uxf' RWS VERSION CUSTOM? '\n' CONTENT VERSION ::= /\d{1,3}/ CUSTOM ::= RWS [^\n]+ # user-defined data e.g. filetype and version CONTENT ::= COMMENT? IMPORT* TTYPEDEF* (MAP | LIST | TABLE) IMPORT ::= '!' /\s*/ IMPORT_FILE '\n' # See below for IMPORT_FILE TTYPEDEF ::= '=' COMMENT? OWS IDENFIFIER (RWS FIELD)* # IDENFIFIER is the ttype (i.e., the table name) FIELD ::= IDENFIFIER (OWS ':' OWS VALUETYPE)? # IDENFIFIER is the field name (see note below) MAP ::= '{' COMMENT? MAPTYPES? OWS (KEY RWS VALUE)? (RWS KEY RWS VALUE)* OWS '}' MAPTYPES ::= OWS KEYTYPE (RWS VALUETYPE)? KEYTYPE ::= 'bytes' | 'date' | 'datetime' | 'int' | 'str' VALUETYPE ::= KEYTYPE | 'bool' | 'real' | 'list' | 'map' | 'table' | IDENFIFIER # IDENFIFIER is table name LIST ::= '[' COMMENT? LISTTYPE? OWS VALUE? (RWS VALUE)* OWS ']' LISTTYPE ::= OWS VALUETYPE TABLE ::= '(' COMMENT? OWS IDENFIFIER (RWS VALUE)* ')' # IDENFIFIER is the ttype (i.e., the table name) COMMENT ::= OWS '#' STR KEY ::= BYTES | DATE | DATETIME | INT | STR VALUE ::= KEY | NULL | BOOL | REAL | LIST | MAP | TABLE NULL ::= '?' BOOL ::= 'no' | 'yes' INT ::= /[-+]?\d+/ REAL ::= # standard or scientific notation DATE ::= /\d\d\d\d-\d\d-\d\d/ # basic ISO8601 YYYY-MM-DD format DATETIME ::= /\d\d\d\d-\d\d-\d\dT\d\d(:\d\d(:\d\d)?)?/ # see note below STR ::= STR_FRAGMENT (OWS '&' OWS STR_FRAGMENT)* STR_FRAGMENT ::= /[<][^<>]*?[>]/ # newlines allowed, and & < > supported i.e., XML BYTES ::= '(:' (OWS [A-Fa-f0-9]{2})* OWS ':)' IDENFIFIER ::= /[_\p{L}]\w{0,31}/ # Must start with a letter or underscore; may not be a built-in typename or constant OWS ::= /[\s\n]*/ RWS ::= /[\s\n]+/ # in some cases RWS is actually optional Note that a UXF file must contain a single list, map, or table, even if it is empty. An IMPORT_FILE may be a filename which does not have a file suffix, in which case it is assumed to be a "system" UXF provided by the UXF processor itself. (Currently there are just three system UXFs: complex, fraction, and numeric.) Or it may be a filename with an absolute or relative path. In the latter case the import is searched for in the importing .uxf file's folder, or the current folder, or a folder in the UXF_PATH until it is found--or not). Or it may be a URL referring to an external UXF file. (See Imports.) To indicate any type valid for the context, simply omit the type name. As the BNF shows, list, map, and table values may be of any type including nested lists, maps, and tables. For a table, after the optional comment, there must be an identifier which is the table's ttype. This is followed by the table's values. There's no need to distinguish between one row and the next (although it is common to start new rows on new lines) since the number of fields indicate how many values each row has. It is possible to create tables that have no fields; these might be used for representing constants (or enumerations or states). Note that for any given table each field name must be unique. If a list value, map key, or table value's type is specified, then the UXF processor is expected to be able to check for (and if requested and possible, correct) any mistyped values. UXF writers are expected output collections--list values and table records (and values within records) in order. Similarly map items should be output in key-order: when two keys are of different types they should be ordered bytes < date < datetime < int < str, and when two keys have the same types they should be ordered using < except for strs which should use case-insensitive <. For datetime's, only 1-second resolution is supported and no timezones. If microsecond resolution or timezones are required, consider using custom ttypes, e.g., =Timestamp when:datetime microseconds:real =DateTime when:datetime tz:str Alternatively, if all the datetimes in a UXF have the same timezone, one approach would be to to just set it once, and then use plain datetimes throughout e.g., =Timezone tz:str [(Timezone <+01:00>) ... 1990-01-15T13:05 ...] Note that a UXF reader (writer) must be able to read (write) a plain text .uxf file containing UTF-8 encoded text, and ought to be able to read and write gzipped plain text .uxf.gz files. Note also that UXF readers and writers should not care about the actual file extension (apart from the .gz needed for gzipped files), since users are free to use their own. For example, data.myapp and data.myapp.gz. Supplementary Vim Support If you use the vim editor, simple color syntax highlighting is available. Copy uxf.vim into your $VIM/syntax/ folder and add these lines (or similar) to your .vimrc or .gvimrc file: au BufRead,BufNewFile,BufEnter * if getline(1) =~ '^uxf ' | setlocal ft=uxf | endif au BufRead,BufNewFile,BufEnter *.uxf set ft=uxf|set expandtab|set tabstop=2|set softtabstop=2|set shiftwidth=2 UXF Logo uxf logo --------------------------------------------------------------------- About Uniform eXchange Format (uxf) is a plain text human readable optionally typed storage format that supports custom types. It may serve as a convenient alternative to csv, ini, json, sqlite, toml, xml, or yaml. Topics yaml toml parser json data storage-engine xml sqlite ini pretty-printer Resources Readme License GPL-3.0 license Stars 64 stars Watchers 1 watching Forks 1 fork Releases No releases published Packages 0 No packages published Used by 3 * @mark-summerfield @mark-summerfield / amp-rs * @mark-summerfield @mark-summerfield / tlm * @mark-summerfield @mark-summerfield / uxf Contributors 2 * @mark-summerfield mark-summerfield Mark * @thewchan thewchan Matt Chan Languages * Python 52.1% * Rust 45.3% * C++ 1.1% * Scheme 1.0% * Vim Script 0.3% * Shell 0.2% Footer (c) 2022 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.