Post B7pdfawtDRyknOaldI by rl_dane@polymaths.social
(DIR) More posts by rl_dane@polymaths.social
(DIR) Post #B7pSGNIliflXjbb66i by rl_dane@polymaths.social
0 likes, 0 repeats
#Unix / #sed / #awk / #shell ninjas,What's the easiest way to suppress anything more than 2-3 blank lines at a time? I don't want to remove all blank lines, just excessive blank lines.I'm coming up with a simple shell function to strip html tags, because I don't like the way terminal browsers re-flow/wrap the text when you use the -dump|--dump option.The way I'd normally do it is something like:sed 's/$/{lineend}/' |tr -d '\n' |sed -E 's/(\{lineend\}){3,}/\1\1\1/g; s/\{lineend\}/\n/g'…but I figure there's probably a more elegant way?
(DIR) Post #B7pSGNyxBpVfqRKmrQ by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_dane cat -s removes all but one empty line in between (which is my standard)Otherwise, something with perl -0ne ….Your second sed uses a GNU extension (-E) and is ill-formed (the input does not end with a newline).
(DIR) Post #B7pdNLx8ET9V4d0UUK by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_daneI'm coming up with a simple shell function to strip html tagslike… <foo.htm xmlstarlet sel -T -t -m . | cat -s?
(DIR) Post #B7pdfakpwGL0C0x8qW by sotolf@polymaths.social
0 likes, 0 repeats
@rl_dane Can't you just run with uniq? I think that only takes away repeated duplicates, as in only really being unique if you sort the file.
(DIR) Post #B7pdfawtDRyknOaldI by rl_dane@polymaths.social
0 likes, 0 repeats
@sotolfThat's not a bad idea, but it would repress some formatting, like paragraph breaks. That's why I was thinking to supress anything more than three empty lines.
(DIR) Post #B7pdfbB4MjJzVNE5jc by sotolf@polymaths.social
0 likes, 0 repeats
@rl_daneso this is not what you want?
(DIR) Post #B7pdfbKzlpGG09s1Cq by rl_dane@polymaths.social
0 likes, 0 repeats
@sotolfHmm... honestly? Not half bad. 😄Just tried it. Yeah, it actually works better than I thought, since some of the blank lines actually have varying amounts of white space in them, so the varying breaks are pretty well preserved, and the annoying blank regions are pretty well removed:sed 's|<[^/>]*>[^<]*</[^>]*>||g; s|<[^>]*>||g; s/&quot;/"/g' index.html |uniq |less
(DIR) Post #B7pdfbZAv6bUi8VLJA by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_dane @sotolf <img alt="Bobby>breaking RL Dane’s script, not tables" />Repeat after me:We do not use regexen to parse HTML.
(DIR) Post #B7pduuqLWzDaqCpxKK by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_dane @gotosocial … what is happening here?… and, I cannot even reply to that toot, gives me a 410 error.A regression in the new release?
(DIR) Post #B7pe4HN195yhJUaEm8 by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_dane @gotosocial it’s still in the database: id | created_at | uri | url | content | attachments | tags | mentions | emojis | account_id | account_uri | in_reply_to_id | in_reply_to_uri | in_reply_to_account_id | boost_of_id | boost_of_account_id | content_warning | language | created_with_application_id | activity_streams_type | text | fetched_at | poll_id | interaction_policy | approved_by_uri | edits | visibility | edited_at | content_type | content_warning_text | thread_id | flags ----------------------------+-------------------------------+-----------------------------------------------------------------------------+------------------------------------------------------------------------+---------+-------------+------+------------------------------+--------+----------------------------+-----------------------------------------+----------------------------+----------------------------------------------------------------------------+----------------------------+-------------+---------------------+-----------------+----------+-----------------------------+-----------------------+------+------------------------------+---------+--------------------+-----------------+-------+------------+-----------+--------------+----------------------+----------------------------+------- 01KW9YBEW059TDRP9HYCSQEQ50 | 2026-06-29 15:00:50.718253+00 | https://toot.mirbsd.org/users/mirabilos/statuses/01KW9YBEW059TDRP9HYCSQEQ50 | https://toot.mirbsd.org/@mirabilos/statuses/01KW9YBEW059TDRP9HYCSQEQ50 | | | | {01KW9YBEWSF5T8PEZ8GMV6FAHN} | | 01GS55SW7VWYN3BFRYCX5NW526 | https://toot.mirbsd.org/users/mirabilos | 01KW9S6BZ8X183YA3NPQHVWYNP | https://polymaths.social/users/rl_dane/statuses/01KW9S6C418MWMWZW61FN07TBT | 01JF074TAG0BFZ3B2702J113N3 | | | | | | Note | | 2026-06-29 15:01:12.73502+00 | | | | | 3 | | | | 01KW9S6BZ8MS8BSJH79RRPJYDQ | 26
(DIR) Post #B7pe7DE70YznKZohA8 by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_dane @gotosocial … coule be a post where I used the “Delete and redraft” feature from Pinafore (since it cannot edit posts yet)…
(DIR) Post #B7sVPW5jw2bejtlncm by rl_dane@polymaths.social
0 likes, 0 repeats
@mirabilosMan, you'd think that in three+ decades of using UNIX, I'd read the man page for cat. 😄sed -E isn't GNU-only, though.rld@Intrepid:~$ uname; man sed |grep -c -- -EFreeBSD6rld@Intrepid:~$ ssh $tildepink sh -c 'uname; man sed |grep -c -- -E'NetBSDman: Formatting manual page...4rld@Intrepid:~$
(DIR) Post #B7sVPWN6tSV7bltfhQ by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_dane doesn’t change that it’s a GNU extension
(DIR) Post #B7sWHQETw1MCCnYyEi by rl_dane@polymaths.social
0 likes, 0 repeats
@mirabilosYeah, never messed with xmlstarlet before, didn't even know it could handle html
(DIR) Post #B7sWHQZ2hZntEZBOHg by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_dane only well-formed XML, but that’s nothing a call to tidy cannot fix.* looks up what he did in the pastFor normal HTML, install tidy and pipe through tidy -q -asxhtml -w 0 -utf8 --quote-nbsp no 2>/dev/null first.For HTML5, install python-html5lib python-lxml, acquire html5tidy, and pipe through python "$me/html5tidy/html5tidy"… oh, that was in Python 2.x times 😹 maybe it’s packaged now.(You can probably also use html5tidy on normal HTML, but a bit of twiddling may be required to get useful output without too many undesired changes.)More, general, notes from other projects:pipe through tidy -asxhtml -utf8 --newline LF --output-bom no --drop-empty-elements no --drop-empty-paras no --quote-nbsp no --wrap 0might need to escape unescaped ampersands in hrefs manually first if producer is known to create thempiping through sed 's!<html xmlns="http://www.w3.org/1999/xhtml"!<html!' after a first xmlstarlet fo -e UTF-8 - step (to format it) might be needed, because handling the default namespace (not xmlns:anything="…") in xmlstarlet is ickywhen outputting XML, xmlstarlet will not put spaces before /> and will not use the expanded form for not-EMPTY elements like textarea, though LD_PRELOADing mircvs://contrib/hosted/tg/forceXHTML.c usually fixes that
(DIR) Post #B7te9ZDfdCy6YJuPWi by rl_dane@polymaths.social
0 likes, 0 repeats
@mirabiloseregex itself, or just sed's eregex mode?
(DIR) Post #B7te9ZTGhDRfKhCrq4 by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_dane ERE are old, but sed has no -E flag traditionally. (It was added to POSIX 2024 (so, too new to rely on), but only because the GNU addition had begun to spread sufficiently.)
(DIR) Post #B7tuEdXyZGNugkqVfM by JdeBP@tty0.social
0 likes, 0 repeats
@rl_dane When you view manual pages on a video terminal, the -s ('squeeze') option to the final $PAGER in the pipeline does this. You'll be surprised at how many blank lines there really are in manuals.less, (BSD) more, most, and (my) console-tty37-viewer all have this option.Given that, it's an of course moment, with a nod to the people who think that this should no more be a part of cat than vis(1) is, that cat has a squeeze option too.#Unix #TextProcessing #nosh