[HN Gopher] Show HN: Xq - command-line XML and HTML beautifier a...
___________________________________________________________________
Show HN: Xq - command-line XML and HTML beautifier and content
extractor
Author : sibprogrammer
Score : 57 points
Date : 2022-11-12 14:07 UTC (8 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| sam_lowry_ wrote:
| xmllint --format -
|
| xsltproc
| samwillis wrote:
| Neat! Like Jq but for XML and HTML.
|
| Have you considered adding css sectors as an alternative to
| xpath? For many simple things a css selector is easer to write
| and more people already know them.
|
| I believe it's possible to translate css selectors to xpath so it
| wouldn't need another selection engine.
| sibprogrammer wrote:
| Thank you for your support! I didn't consider CSS selectors as
| an alternative to Xpath, but really like the idea. Absolutely
| agree CSS selectors will be more convenient :)
| esprehn wrote:
| The issue with CSS selectors is they can only express element
| selection. There's no concept of picking the text of the
| element or the value of an attribute. There's been proposals
| over the years to support pseudo elements for that, so you
| could make something up like ::attr(name) and ::text, but
| nothing in the standards.
|
| htmlq does this by having various command line options like
| --attribute and --text.
|
| https://github.com/mgdm/htmlq#examples
| samwillis wrote:
| Quite a few scraping (such as Scrapy) or browser automated
| testing frameworks (like Playwright) have standardised on
| ::text. I would take a look around and if other tools are
| using the same none standard selectors for these features,
| it would be perfect for Xq. It's not like you are complying
| with standards for browser compatibility.
| mk12 wrote:
| htmlq does that.
| kristopolous wrote:
| There's an ancient formatter I've been using for years (gasp,
| probably well over a decade) https://xml-
| coreutils.sourceforge.net/ ... xml-fmt (https://xml-
| coreutils.sourceforge.net/xml-fmt_man.html)
|
| It'll be nice to try something new. swiss army style command line
| XML tools has been pretty neglected.
| xg15 wrote:
| There is also yq [1], which attempts the same for yaml, toml and
| xml. (And confusingly _also_ contains a binary named "xq" for
| the xml part - however, it uses jq for querying instead of xpath)
|
| [1] https://github.com/kislyuk/yq
| encryptluks2 wrote:
| This is awesome. Hugo has been missing a tidy feature for a long
| time. I hope they use this to implement it finally or that the
| author might even consider creating a PR.
| password4321 wrote:
| related discussion, orthogonally:
|
| _JC - JSONifies the output of many CLI tools_
|
| https://news.ycombinator.com/item?id=33448204 2022-11-03
| its-summertime wrote:
| doesn't seem to work great with https://ap-play
| erservices.streamtheworld.com/api/livestream?station=NOVA_919&ver
| sion=1.9
|
| so xqilla and learning xlst et al still seems like my to-go for
| complex documents
| manbart wrote:
| Hadn't heard of XQilla, but looks like it's limited to XPath 2.
|
| You should try an XPath 3 based implementation. XQuery 3.1 has
| some nice features not found in XPath 2 implementations e.g.
| native JSON support, the arrow operator (piping), group by
| clause in FWLORs. XSLT 3.0 has some nice new stuff too, but I
| use that less
| mdaniel wrote:
| Shoutout to my go-to: https://github.com/EricChiang/pup#readme
| (also golang) and my 2nd favorite
| https://xmlstar.sourceforge.net/
| tmsbrg wrote:
| Definitely seconding pup, love it and it's easy to use with css
| selectors. Often use it to parse HTML tables of data on random
| websites into usable CSVs / etc.
|
| xq looks interesting for pure XML, will add it to my notes.
|
| Regarding xmlstarlet, I don't use it much but here's some stuff
| from my notes:
|
| XML filter on xpath: xmlstarlet sel -t -c
| configuration/appender logback.xml; echo
|
| XML editing: xmlstarlet ed -d configuration/root -d
| configuration/logger -s configuration -t elem -n woof -v
| 'Woof!' logback.xml
|
| The last one deletes elements given by xpath
| "configuration/root" and "configuration/logger", then adds a
| new element with the contents <woof>Woof!</woof> under the
| xpath "configuration".
| mdaniel wrote:
| > Regarding xmlstarlet, I don't use it much but here's some
| stuff from my notes:
|
| It is also handy doing at some PoC work, and then one can
| feed it "-C" to output the backing xslt for use elsewhere
| $ echo '<a><b>hello</b></a>' | xmlstarlet sel -t -m /a -c
| 'count(*)' --nl 1 $ echo
| '<a><b>hello</b></a>' | xmlstarlet sel -C -t -m /a -c
| 'count(*)' --nl <?xml version="1.0"?>
| <xsl:stylesheet
| xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
| version="1.0"> <xsl:output omit-xml-
| declaration="yes" indent="no"/> <xsl:template
| match="/"> <xsl:for-each select="/a">
| <xsl:copy-of select="count(*)"/> <xsl:value-of
| select="' '"/> </xsl:for-each>
| </xsl:template> </xsl:stylesheet>
| mikmoila wrote:
| Looks good! Does the xpath expression support XML namespaces?
| mdaniel wrote:
| No, and arguably worse: it completely ignores them
| $ echo '<a xmlns="http://www.w3.org/2005/Atom"><b
| xmlns="urn:beta">hello</b></a>' | xq --xpath '/a/b/text()'
| hello
|
| Only partial support for functions, too, it would appear
| $ echo '<a xmlns="http://www.w3.org/2005/Atom"><b
| xmlns="urn:beta">hello</b></a>' | xq --xpath 'count(/a/*)'
| $
|
| Some A++ error handling, too, as it just calls panic when angry
| mikmoila wrote:
| Ah ok; Namespace support before XPath 3.0 with
| Q{nsname}localname would make cli tool like this a bit too
| verbose. Of course you could store ns-prefix to ns mappings
| to conf file or environment variables...
| mdaniel wrote:
| I actually tried the lxml syntax "/{urn:beta}b" and that's
| how I learned about the panic-based error handling, but I
| didn't know about that Q syntax, so thank you for bringing
| it to my attention $ echo '<a
| xmlns="urn:alpha">hello</a>' | xq --xpath
| '/Q{urn:alpha}a/text()' panic:
| /Q{urn:alpha}a/text() has an invalid token.
| goroutine 18 [running]:
| github.com/antchfx/xmlquery.Find(...)
| github.com/antchfx/xmlquery@v1.3.8/query.go:76 ...
|
| It seems to be a problem with that backing xmlquery
| library, and I did a quick GitHub topics search and found
| https://github.com/zzossig/rabbit which claimed to be XPath
| 3, but then https://github.com/zzossig/rabbit#what-is-not-
| supported says "lol, namespaces, wat?" so :-(
| mikmoila wrote:
| There aren't that many libraries available with full
| XPath 3.x conformance. In order to give some constructive
| feedback, perhaps an optional .conf file with ns-
| prefix->ns-name mappings could be read and passed to the
| XPath engine as ns-context? Even w/o the XPath functions
| support and only XPath selector expression this would
| make the tool to cover most of the usecases eg. simply
| finding elements.
| marban wrote:
| What would be the differences to lxml/beautifulsoup/etc.?
___________________________________________________________________
(page generated 2022-11-12 23:01 UTC)