[HN Gopher] The Array Cast - A podcast about the array programmi...
___________________________________________________________________
The Array Cast - A podcast about the array programming languages
Author : srpeck
Score : 147 points
Date : 2021-05-19 14:26 UTC (1 days ago)
(HTM) web link (www.arraycast.com)
(TXT) w3m dump (www.arraycast.com)
| nulldata wrote:
| Great first episode. I'd love to hear Aaron Hsu on this!
| mjburgess wrote:
| Surprising highly production values -- programming podcasts are
| typically extremely poor auido with very little prep going into
| to them.
| slver wrote:
| These days having good video and audio quality doesn't require
| a lot of money. It just requires couple of hours educating
| yourself online what you need and how to use it.
| [deleted]
| skipcave wrote:
| Boxing in J sep=:10#.^:_1] NB. Separate digits
| ea=:&.> NB Perform on each sep ea 23 7534 322 7 24756
|
| |2 3|7 5 3 4|3 2 2|7|2 4 7 5 6| #ea n NB. How
| many?
|
| |2|4|3|1|5| +/ea n NB. Sum
|
| |5|19|7|7|24| */ea n NB. Product
|
| |6|420|12|7|1680| /:~ ea n NB. Sort
|
| |2 3|3 4 5 7|2 2 3|7|2 4 5 6 7|
| not_knuth wrote:
| This looks interesting. Can't wait to give it a listen! Has
| anyone already done so? Reviews?
| rscho wrote:
| It's good. They've got representatives of all the major array
| language communities among the hosts (APL, J, K/Q) plus a C++
| guy.
| __vim__ wrote:
| No fortran guy?
| rscho wrote:
| I think the goal in bringing in a C++ programmer was to
| provide an outsider view, but no, no Fortran guy. I don't
| think Fortran is an array language in that arrays are not
| the only datastructures in Fortran, though?
| Banana699 wrote:
| Fortran isn't an array language at all, really. Maybe
| some features of the array languages style are super-
| imposed atop it with parallizing extensions, dialects, or
| whatnot; but it's style has always been and still is to
| write explicit loops and mutate state left and right.
| John Backus in his famous 1977 turing award speech
| explicitly named it a representative of the "fat weak"
| languages he talked of and said it was a thin veneer over
| assembly, he based his fictional FP language on APL
| instead.
|
| Maybe grandparent meant it has the same _application_ as
| array languages, in that its only surviving kingdom is
| scientific computing where devouring gargantuan arrays of
| numerics is the only thing that matters, unlike C or C++
| that are much more widely used. Maybe that's why it has a
| long history of being parallized with various tools and
| runtimes _despite_ its inherent imperativeness. (I don't
| remember where I heard this, but somebody wrote an
| analyzer to analyze some Fortran programmes in the 90s
| and found that over 80%/90% of Fortran programmes are
| spent doing variations of map, filter and reduce. So it's
| an extremely imperative language against its intended
| use. Maybe I will post a link in an edit when I find the
| source of the claim)
| Bostonian wrote:
| Fortran has had built-in array operations since the
| Fortran 90 standard. If X and Y are scalars or arrays of
| the same shape, you can X+Y, X*Y, exp(X), sin(X) etc. You
| can define your own elemental functions that act on both
| scalars and arrays of any rank. I still write loops when
| programming in Fortran 95 but less often than in Fortran
| 77. So I think modern Fortran is an array language.
| slver wrote:
| Is an array language a language where you only have
| arrays? That honestly sounds a bit odd.
| klibertp wrote:
| No, an array language is one in which all the built-in
| operations are applicable to arrays. Take addition as an
| example: 4 + 4 8
|
| In array languages the same operator can be used for
| arrays; or equivalently you can say that the example
| above sums two arrays of length 1. In J, you can do this
| and expect it to work: 4 4 4 + 2 2 2
| 6 6 6
|
| This is true for all the built-ins, and many user defined
| operations (as long as you don't fiddle with so called
| _rank_ of the verb you 're defining).
|
| Numpy is close to that, but there's still a distinction
| between arrays and scalars, while in array langauges that
| distinction is often blurred: 4 + 1
| 2 3 5 6 7
|
| Edit: in J, you can have atoms, or scalars, but you need
| to box them: (3;4;5) +-+-+-+
| |3|4|5| +-+-+-+
|
| but then you can't do anything with them until you unbox
| them again: (3;4;5) + 3
| |domain error | (3;4;5) +3
| (+&4) each (3;4;5) +-+-+-+ |7|8|9|
| +-+-+-+
|
| (Examples straight from J REPL)
| slver wrote:
| It seems almost as if it'd be more useful to not
| explicitly expose operators as applicable to arrays, but
| implement SIMD optimization for operator expressions in
| .map(function) and something like .binaryMap(right,
| function) ex. [1, 2, 3].binaryMap([10,
| 10, 10], (a, b) => a * b) // Produces [10, 20, 30]
|
| This would be easier to optimize when compiled because
| the expressions can be simplified and mapped to the right
| SIMD instructions.
| dzaima wrote:
| That just removes a tiny tiny bit of dynamic dispatch
| overhead. Which is still needed anyways, as array
| languages can often dynamically switch between 1-bit,
| 8-bit, 16-bit, 32-bit integer (and 64-bit float) arrays,
| depending on the elements, completely transparently to
| the user.
| slver wrote:
| Most compilers can inline statically defined closures in
| these contexts. And tracing JITs do this even when the
| closure is not define statically (but is stable).
|
| It's more about allowing the SIMD goodness without the
| ambiguity and restrictions of "scalar operators work on
| arrays" implemented naively.
| dzaima wrote:
| A compiler could inline SIMD with or without an operation
| having an explicit map, the dynamic type checks needed
| are gonna be the same. (and, since this is an array
| language, the tiny extra overhead of completely dynamic
| dispatch is gonna be small compared to the executed SIMD
| afterwards anyways)
|
| You lose a lot of simplicity & brevity by requiring
| explicit mapping, and gain close to nothing.
| rscho wrote:
| Traditional array languages (APL,J) are not amenable to
| static compilation due to other dynamic issues, though. I
| think Dyalog APL is experimenting with a bytecode
| interpreter, but no JITs in sight either (that I know
| of). The very dynamic aspect of those languages makes
| that difficult. I'd love a compileable similar language,
| though. To replace R/Python for statistics that are
| sooooo annoying when exploring data due to verbosity.
| klibertp wrote:
| > without the ambiguity
|
| There's no ambiguity! In J, all (with some exceptions)
| operations work on arrays, period. `1 + 1` is not an
| addition of two scalars, but instead of two arrays of
| length 1. As I mentioned, you can have scalars, but a)
| they still live in arrays and b) you can't do anything
| with them without unboxing. So there's no ambiguity, as
| far as I can tell.
|
| Also, APL and J are implemented as intepreters, but that
| doesn't preclude using SIMD instructions when executing
| expressions. I'm 100% sure the operations are not
| "implemented naively" :)
| rscho wrote:
| > `1 + 1` is not an addition of two scalars, but instead
| of two arrays of length 1
|
| I think that's true of (Dyalog) APL, but not of J [1].
| APL follows the nested array model (that you describe),
| while J follows the flat array model that does have true
| scalars.
|
| [1]
| https://aplwiki.com/wiki/Array_model#Flat_array_theory
| dzaima wrote:
| AFAIK, J considers `1` to be an array - `L. 1` and `L. 1
| 2` both give 0 (`L. (1;2)` gives 1).
|
| APL's simple scalar numbers are much more like regular
| numbers in non-array languages.
| klibertp wrote:
| Unfortunately, I have no idea about the implementation
| and what optimizations are done in J or APL for which
| operations :( I know that there are hardcoded "fast path"
| expressions (particular combinations of operations) which
| have much better performance than more general
| expressions doing the same thing, so it might be that the
| optimization happens at that level.
|
| OTOH, your example is very verbose when compared to J's
| version: 1 2 3 * 10 10 20 30
|
| Plus, in J it generalizes to higher dimensional arrays:
| i. 3 3 0 1 2 3 4 5 6 7 8
| (1 + i. 3 3) * 10 10 20 30 40 50 60
| 70 80 90
| rscho wrote:
| even less verbose:
|
| (>: i. 3 3) * 10
| rscho wrote:
| ...or
|
| 10 * >: i. 3 3
|
| FWIW
| slver wrote:
| My example is verbose in order to clearly communicate the
| principle. I can trivially shorten it to
| [1, 2, 3].map(a => a * 10)
|
| if I wanted to literally carry out that task alone.
| [[0, 1, 2], [3, 4, 5], [6, 7,
| 8]].flatMap(a => a * 10)
| klibertp wrote:
| I think this is pseudo-code? `flatMap` and `=>` for
| lambdas look like Scala, but there are no array literals
| using `[]` there. Assuming you mean Scala-like semantics,
| your second example wouldn't work at all:
| scala> Array(Array(1,2),
| Array(4,5)).flatMap(a => a * 10)
| ^ error: value * is not a member of
| Array[Int]
|
| You would need to write it like this:
| scala> Array(Array(1,2),
| Array(4,5)).flatMap(a => a.map(x => x * 10))
|
| But then you'd get a flattened array of ints, not array
| of arrays of ints: res6: Array[Int] =
| Array(10, 20, 40, 50)
|
| So to get the same result as (i. 2 2) *
| 10
|
| You'd need to write: scala>
| Array(Array(1,2), Array(4,5)).map(a
| => a.map(x => x * 10)) res7: Array[Array[Int]] =
| Array(Array(10, 20), Array(40, 50))
|
| ...which _is_ more verbose, no? :) EDIT: obviously, I
| mean "map in map" part, not the Array initialization!
|
| The problem with list comprehensions and `map`, `filter`
| and friends is that they work very well for flat lists,
| or lists of lists in some specific circumstances (ie.
| when you can use `flatMap`). 2D arrays in the general
| case, and arrays with higher dimensions are really hard
| to work with using these primitives, unless you have some
| clever overloads, like what Clojure does. I think Haskell
| also has a solution for this, but I don't know it enough
| to comment further, unfortunately :)
| slver wrote:
| What I can't understand is what would J do if I tell it
| to sum these two arrays: 1 2 3 4 5
| 6 7 8
|
| I.e. 5 elements and 3 elements
| klibertp wrote:
| 1 2 3 + 4 5 6 7 |length error | 1 2 3
| +4 5 6 7
|
| In general - depends on the operation. Sometimes the verb
| checks that both operands have the same rank and
| dimensions, sometimes the shorter/smaller side gets
| applied column-wise (or cell-wise/page-wise):
| (2 2 $ 1 2 3 4) NB. $ means "reshape" 1 2
| 3 4 (2 2 $ 1 2 3 4) + 1 2 2 3
| 5 6
|
| Sometimes the shorter side is repeated as much as needed
| to get the correct length, and sometimes the shorter side
| gets extended with 0s, and sometimes the longer side gets
| truncated: (2 2 $ 1 2 3 4 5 6)
| 1 2 3 4 (2 4 $ 1 2 3 4 5 6) 1
| 2 3 4 5 6 1 2
|
| To be perfectly honest: it's very unintuitive to me and I
| have to check the docs pretty often to see how the given
| verb behaves outside of the simplest case (ie. when the
| rank and dimensions match). But, I learned J as a hobby
| and never invested enough time into it to _really learn_
| all the built-ins, nor did I try using J outside of toy
| examples, so _maybe_ this behavior becomes convenient
| once you internalized it?
|
| EDIT: forgot to mention, you can also control the
| effective rank of the verb, like this:
| <"0 (2 2 $ 1 2 3 4) +-+-+ |1|2|
| +-+-+ |3|4| +-+-+ <"1 (2 2 $ 1
| 2 3 4) +---+---+ |1 2|3 4|
| +---+---+ <"2 (2 2 $ 1 2 3 4) +---+
| |1 2| |3 4| +---+
|
| the `<` verb means "box", without the `"n` part you'd get
| the last value, which is a 1-dimensional array of boxes
| of length 1 (with 2x2 array inside the box). By selecting
| rank of the verb, you can decide on which level the verb
| should be applied: with `"0` it's applied to the smallest
| possible chunks of the array (cells - you get 2x2 array
| of boxes), with `"1` you apply the verb to bigger chunks
| (rows), and so on, for input arrays of any dimensions (so
| if you have a 3x2x2 array, `"2` will apply the verb to
| the 3 2x2 arrays).
| slver wrote:
| Gotta say, "depends on the operation" was the most
| unfortunate possible answer to my question, alas :-)
| Maybe one day we'll figure out how to make a consistent
| array language ;-)
| klibertp wrote:
| Well, for what it's worth - and as mentioned by a sibling
| poster - most verbs, and certainly all arithmetic
| operators, obey a few simple rules for determining which
| behavior you'll get. I should have noted this, before I
| started whining about all the _other_ verbs, which don 't
| :D
| ithkuil wrote:
| Give this a try: https://t3x.org/klong/klong-ref.txt.html
| dzaima wrote:
| I don't think $ is of worthy note here. It's literally a
| builtin for cycling data, that's what it does.
|
| All simple arithmetic follows leading axis agreement[1]
| (so erroring on mismatched length, and mapping over cells
| for high rank). Don't know much about other builtins,
| but, unless J does some weird stuff I don't know about (I
| don't know much of J), it shouldn't be too illogical.
|
| [1] https://aplwiki.com/wiki/Leading_axis_agreement
| klibertp wrote:
| > I don't think $
|
| You're right, although cycling instead of erroring out
| was surprising to me. Another verb I had trouble with is
| roll/deal:
| https://code.jsoftware.com/wiki/Vocabulary/query
|
| > it shouldn't be too illogical.
|
| I'm not saying it's illogical. I'm pretty sure there are
| good reasons behind the design of each word, especially
| since there are so few. As I mentioned, I suspect it's
| just a matter of me not getting enough exposure to the
| various use-cases of many standard words.
| mjaniczek wrote:
| You could consider Pandas / numpy an array language, even
| though it's really a library for Python. The exposed API
| is IMHO what's important.
| rklmno wrote:
| Mostly yes.
|
| Numbers (and character) are implemented as arrays with 0
| dimensions. Text would be an array of characters with 1
| dimension (the number of characters), and generally
| speaking the dimension of an array is a one dimensional
| list of non-negative integers. Many array languages also
| include an array type which is approximately the same as
| a C pointer to an array, with a bit of jargon thrown in
| to distinguish a reference to an array from the array
| itself.
|
| Something like an SQL table in an array language would be
| implemented as a list of columns (rather than as a list
| of rows) and a corresponding list of column labels. This
| has some interesting benefits.
|
| That said, functions in array language are typically not
| arrays (though they presumably would have a textual
| representation). So... not everything is an array.
| rscho wrote:
| From the J docs:
| https://www.jsoftware.com/help/dictionary/dx005.htm
| nub=: (i.@# = i.~) # ] 5!:2 <'nub'
| +-------------------+-+-+ |+--------+-+------+|#|]|
| ||+--+-+-+|=|+--+-+|| | | |||i.|@|#|| ||i.|~||| | |
| ||+--+-+-+| |+--+-+|| | | |+--------+-+------+| | |
| +-------------------+-+-+
|
| ... so J functions have an array representation, at
| least.
| mlochbaum wrote:
| That's the case in APL and J. K uses nested lists to
| represent arrays, and has non-lists (atoms). But the
| convention is that an n-times nested list is considered
| an n-dimensional array so even an atom is an array, with
| 0 dimensions.
|
| There's a page on the various approaches to arrays in the
| APL family at https://aplwiki.com/wiki/Array_model .
| kieckerjan wrote:
| Nice podcast. Definitely made me want to check out APL. Again.
| (Disclaimer: I go through this phase every couple of years.)
| RojerGS wrote:
| I listened to the first episode and it was really interesting
| to listen to really experienced programmers share the things
| they like the most about array programming and the array-
| oriented languages they are most familiar with.
| BlanketLogic wrote:
| An interesting episode. Recommended.
|
| One of the presenters is a novice with c++ background and the
| rest all are experts in their respective array programming
| languages.
| zanethomas wrote:
| https://github.com/abrudz/ngn-apl
| an1sotropy wrote:
| I had thought of APL as something from computing pre-history,
| with its bizarro custom keyboard, but I learned that APL and
| other array languages are apparently alive and well. Will
| subscribe to the podcast.
|
| Two quotes the hosts brought up stuck with me:
|
| (at 15:05) "A language that doesn't change the way you think is
| not a language worth learning". From Alan Perlis [1], and his
| Epigrams in Programming (#19) [2]
|
| (at 16:49) "it is a privilege to learn a language/ a journey into
| the immediate". From poet Marilyn Hacker [3]; totally captivating
| idea, even if not not about programming languages [4]
|
| [1] https://en.wikipedia.org/wiki/Alan_Perlis [2]
| https://cpsc.yale.edu/epigrams-programming [3]
| https://poets.org/academy-american-poets/winner/prizes/james...
| [4] https://www.enotes.com/topics/marilyn-hacker/critical-essays
| bidirectional wrote:
| k (and the closely related q) is the main language used in
| industry, particularly at investment banks and hedge funds. It
| can be a bit of a shock to realise there are people in London
| earning in excess of PS1000/day (pretty good for London)
| working in a language where well-written code looks like
| this[1]:
| us:{$[#i:&{(y~*K)&"*"~\*x}':x;@[x;i;:[;,"_"]];x]}
|
| It's like discovering a whole different world of software
| development. Also I don't use that example to disparage k, I
| have come to appreciate the array language way-of-working. It
| just looks very alien.
|
| [1] Real example found in a random script on https://nsl.com/:
| http://nsl.com/k9/sql.k
| jiggawatts wrote:
| What the actual f?
|
| It's like someone threw up the noise that modems make during
| initial connection onto an electric typewriter from the
| 1960s, and then explained their intention using quotes from a
| Lovecraft novel.
| userbinator wrote:
| I bet Chinese and Japanese look like that to someone who
| knows only English too.
| Avshalom wrote:
| As some one who only knows English. No that is not what
| Chinese and Japanese look like to me.
|
| That's why I prefer APL and its special symbols. It's
| still utterly inscrutable if you don't know it but at it
| _looks intentional_. Those symbols shift your mindset or
| reframe what you 're looking at.
| geocar wrote:
| I am going to tell you something fantastic, but first, I
| want to explain some things about this:
| us:{$[#i:&{(y~*K)&"*"~*x}':x;@[x;i;:[;,"_"]];x]}
|
| The first is that there's a typo in what bidirectional
| wrote. The above is correct. The second, is what it is.
| Once I have explained that, I can tell you the fantastic
| thing.
|
| k syntax is very simple. There's just a few forms you need
| to be aware of: f x
|
| which applies x to f. a f b
|
| which is apply f to the two arguments a and b, and:
| f[a;b;c]
|
| which allows you to do three arguments. You can write the
| first one as f[x] and the second as f[a;b] if you like even
| more consistency. f can be an "operator" -- that is a
| symbol. The symbols ' / and \ are special and called
| adverbs. These adverbs have a special form if followed by a
| colon, so ': is different than ' and has nothing to do with
| : or '. I think Arthur just ran out of keys on the
| keyboard. Once you have those, parenthesis () and braces {}
| have some special syntax, just like double-quotes " do.
|
| With the syntax explained, let us try to understand what we
| are looking at.
|
| us: is how we start assignment. You can say "us gets" if
| you like (the colon can be pronounced). {} braces surround
| a lambda, this one takes a single argument "x" (the first
| argument). $[a;b;c] is cond like in lisp; if a then b else
| c. # means count. i: is another assignment.
|
| & means where -- the argument to which is going to be a
| bitmap like 000100b or 01101b or something like that, and
| where returns the indices of the set bits; the former
| example being the list 3, the latter example being the
| three-element list 1 2 4.
|
| Another lambda comes next: We can see it takes two
| arguments because there's an x and a y in there (y is the
| second argument). We can get a clue as to what it expects
| because the following adverb ': means each-prior. This
| tells us "x" is going to be a list of things, and this
| lambda is going to consume them pairwise. If given the list
| {(x;y)}':"iliketacos" we get the result:
| {(x;y)}':"iliketacos" i li il
| ki ek te at ca oc
| so
|
| y is the "previous" value, and "x" is the current value.
| The "where" before it tells us we want to know the indices
| where the condition inside is true. Let's try and
| understand that condition.
|
| y~*K is in parenthesis. Parenthesis group, so we execute
| them first (just like in other languages). We're looking
| for a situation where the previous value is the first
| (that's what asterisk means here) of K. What is K?
| K:("select";"distinct";"partition";"from";"where";"group";"
| having";"order";"limit")
|
| So we're looking for a value (x) whose previous (y) is the
| first of K which is "select". The "&" that follows here is
| "and" - Arthur likes to overload operators since there
| aren't many symbols on the keyboard and this is something
| you get used to.
|
| So you can read {(y~*K)&"*"~*x}':x as simply trying to find
| the sequences "select star" -- given a list ("select"; "*";
| "from"; "potato") you get 0100b and from ("select"; "*";
| "from"; "("; "select"; "*"; "from"; "potato; ")") you get
| 01000100b. I think the attempt is to disambiguate the
| asterisks in the sql: select * from tacos
| where cat=4*42
|
| but sql is a strange and irregular language, so this kind
| of thing is necessary. Back to our query:
| us:{$[#i:&{(y~*K)&"*"~*x}':x;@[x;i;:[;,"_"]];x]}
|
| i is going to be the locations of the asterisks following
| select. If the count of that is nonzero; we're going to do
| the @-part, and if not, we're just going to return x.
| @[x;i;f]
|
| is called amend. It returns x, but at indices i, we apply
| them to f, so it's x[i]:f[x[i]] which is pretty cool. f in
| this case is a projection, of "gets" (the function colon)
| with the second-argument bound to an underscore. That is:
| :[;,"_"]
|
| is just a function. That's how @[x;i;:[;,"_"] replaces all
| of the asterisks that follow select with an a "_"
|
| Almost. It's actually a list of length one, rather than the
| scalar "_". I haven't read everything in sql.k but this is
| probably important elsewhere.
|
| Ok. Now that I have explained what this is and what it
| does, I am ready to tell you something fantastic. I read
| this:
| us:{$[#i:&{(y~*K)&"*"~*x}':x;@[x;i;:[;,"_"]];x]}
|
| as:
|
| "us gets a function, that finds the indices of asterisk
| following the first element of K, and then replaces the
| things at those indices with underscores"
|
| Literally. From left, to right. Just that fast. And I only
| program in k part-time. That's not the fantastic thing. The
| fantastic thing is that by learning to read k, I am _almost
| miraculously_ able to read other languages faster. This:
| copied=False for i in range(1,len(a)): if
| a[i] == "*" and a[i-1] == "select": if not
| copied: copied = True a = a[:]
| a[i] = "_";
|
| gives me some grief for being so irregular and gross, and I
| have to look up range/xrange and len and memorise a much
| more complex set of rules for syntax, and I have to track
| the order of things carefully and so on, but I have places
| in my brain, made by k, for those things, and so I am able
| to absorb code in other languages faster.
|
| If that does not amaze you, I do not think you have
| considered the ramifications of what I said. I can suggest
| maybe reading it again (or maybe _actually_ reading what I
| wrote instead of skipping to the punchline), but if after
| two or three tries you are still lost, maybe you can ask a
| question and I can try to answer it.
| specialist wrote:
| Wow. That's some explanation. Thanks.
|
| Do you use "rainbow brackets"?
|
| This example is first hit I found: https://kristofferc.gi
| thub.io/OhMyREPL.jl/latest/features/ra...
|
| Such an obvious idea once you see it. Wish I had syntax
| coloring and rainbow brackets when I coded LISP for hire.
| geocar wrote:
| > Wow. That's some explanation. Thanks.
|
| Happy to help.
|
| > Do you use "rainbow brackets"?
|
| No. I flash brackets, but I tend to turn off other forms
| of syntax highlighting. I find it extremely distracting
| when the syntax highlighter "decides" wrong, and I've
| become convinced comments at the end of lines like //} to
| "fix" the highlighter deal with complicated stuff is
| hurting more than it's helping.
| kazinator wrote:
| > _I have to look up range /xrange and len and memorise_
|
| All you're communicating here is that you don't regularly
| work with Python.
|
| The claim that you have to look up _len_ seems
| disingenuous; I might believe it if you didn 't look like
| a speaker of English.
|
| > & means where
|
| And that's something any engineer would know, unlike
| having to look up what _len_ means?
| geocar wrote:
| > The claim that you have to look up len seems
| disingenuous
|
| In what way?
|
| I have to look up "how do I get the indices of a list" to
| get range(1,len(x)) -- I think I could have also used
| enumerate() and a bunch of other things, but this seemed
| the shortest.
|
| > All you're communicating here is that you don't
| regularly work with Python.
|
| I hope I'm communicating more than that because I put a
| lot of effort into my comment. I don't regularly work
| with k either.
|
| What exactly do you think you are communicating?
| kazinator wrote:
| What I mean by that is that len is an obvious
| abbreviation for length that crops up in numerous
| languages. Not to mention code bases. It's simply not
| plausible you could forget what len means in Python after
| confirming one time that it is exactly what it looks
| like. (Unless you're a struggling non-native speaker of
| English forgetting the word _length_ itself.)
| kazinator wrote:
| TXR Lisp: 1> (defun rewrite (fun list)
| (build (while* list (let
| ((nlist [fun list])) (if (eq list nlist)
| (if list (add (pop list))) (set list
| nlist)))))) rewrite 2> (defmacro rewrite-case
| (sym list . cases) ^(rewrite (lambda (,sym)
| (match-case ,sym ,*cases))
| ,list)) rewrite-case 3> (rewrite-case x '(foo
| bar * select * fox select * bravo) ((select *
| . @rest) ^(select _ . ,rest)) (@else else))
| (foo bar * select _ fox select _ bravo)
|
| _rewrite-case_ and _rewrite_ appear in the TXR Lisp
| internals; they are used in the compiler for scanning
| instruction sequences for patterns and rewriting them.
|
| E.g. a function early-peephole looks for one particular
| four instruction pattern (six items, when the labels are
| included). Rewriting it to a different form helps it
| disappear later on. (defun early-peephole
| (code) (rewrite-case insns code (((mov
| (t @t1) (d @d1)) (jmp @lab2)
| @(symbolp @lab1) (mov (t @t1) (t 0))
| @lab2 (ifq (t @t1) (t 0) @lab3) .
| @rest) ^((mov (t ,t1) (d ,d1)) (jmp
| ,lab3) ,lab1 (mov (t ,t1) (t 0))
| ,lab2 ,*rest)) (@else else)))
|
| This is much more general and powerful than a hack which
| just looks at successive pairs for an ad-hoc match.
| rewrite-case can have multiple clauses, of different
| lengths, and arbitrary matching complexity.
| kazinator wrote:
| The original requirements should be addressed. The thing
| being matched is not just _select_ , but actually any one
| of a set of symbols that appear in K.
|
| We can stick that data into the pattern matching syntax
| using the or operator: 3> (rewrite-case x
| '(foo bar * select * fox where * bravo) ((@(or
| select distinct partition from where
| group having order limit) * . @rest) ^(,(car x) _ .
| ,rest)) (@else else)) (foo bar * select
| _ fox where _ bravo)
|
| Or put it into a variable: 4> (defvarl K
| '(select distinct partition from where group having order
| limit)) K 5> (rewrite-case x '(foo bar *
| select * fox where * bravo) ((@(member @sym K)
| * . @rest) ^(,sym _ . ,rest)) (@else else))
| (foo bar * select _ fox where _ bravo)
|
| "If an object _sym_ which is a member of K is followed by
| * and some remaining material, replace that by _sym_ ,
| underscore and that remaining material."
|
| Hash table: 6> (set K (hash-list '(select
| distinct partition from where group having order limit)))
| #H(() (select select) (where where) (limit limit) (order
| order) (having having) (distinct distinct)
| (partition partition) (group group) (from from))
| 7> (rewrite-case x '(foo bar * select * fox where *
| bravo) ((@[K @sym] * . @rest) ^(,sym _ .
| ,rest)) (@else else)) (foo bar * select
| _ fox where _ bravo)
|
| This code golfs moderately well:
| https://news.ycombinator.com/item?id=27227276
| leephillips wrote:
| Is the open source version (Kona) any good? Or do we have
| to go proprietary to try this out?
| ithkuil wrote:
| I had much fun learning the klong language:
| https://t3x.org/klong/
|
| I even bought the well written book, which was a pleasure
| to read.
| RodgerTheGreat wrote:
| ngn-k is a different dialect from Kona, but actively-
| maintained and open source: https://codeberg.org/ngn/k
|
| oK (which is also k6) can be tried here:
| http://johnearnest.github.io/ok/index.html
| kazinator wrote:
| Let's do this using the same approach, "find indices and
| assign over them in a copy of the sequence":
| This is the TXR Lisp interactive listener of TXR 259.
| Quit with :quit or Ctrl-D on an empty line. Ctrl-X ? for
| cheatsheet. Do not operate heavy equipment or motor
| vehicles while using TXR. 1> (defun subst-select-*
| (list) (let ((indices (where (op starts-with
| '(select *)) (cons nil
| (conses list))))) (if indices
| (let ((list (copy list))) (set [list
| indices] (repeat '(_))) list)
| list))) subst-select-* 2> (subst-select-*
| '(foo bar * select * fox select * bravo)) (foo bar
| * select _ fox select _ bravo)
|
| (conses list) gives us a list of the list's conses: e.g
| in (1 2 3) the conses are (1 2 3), (2 3) and (3), so the
| list of them is ((1 2 3) (2 3) (3)).
|
| _where_ applies a function to a sequence, and returns
| the 0-based indices of where the function yields true.
|
| (op starts-with '(select *)) yields a lambda which tests
| whether its argument starts with (select *). No brainer.
|
| If we naively applied that to the conses, we would get
| thew rong indices: the indices of the select symbols, not
| of the asterisks.
|
| The workaround for that is (cons nil (conses list)): we
| cons an extra dummy nil element to shift the positions,
| and process the resulting list.
|
| Once we have the indices list, if it isn't empty, we copy
| the original input, and assign underscore symbols into
| the indicated positions. To do that we generate an
| infinite lazy list of underscores; the assignment takes
| elements from this list and puts them into the specified
| index positions.
| geocar wrote:
| If this were me, and I needed a function like us, I would
| have written this:
| us:{$[x~(z;y);,"_";y]}[(*K;,"*")]':
|
| I would be interested in seeing _anything_ that was
| shorter[1] and faster than that _in any language_ , and I
| would be very curious to learn from anyone who could also
| do that faster than me.
|
| But I'm not a fetishist: I didn't learn k because it was
| cute, and I don't wake up every day looking for ways to
| rewrite other people's code so that it is slower and
| bigger. Do you? Or is today special?
|
| [1]: Measured in source-code bytes.
| kazinator wrote:
| > _I would be interested in seeing anything that was
| shorter_
|
| One way to do that would be to pose that as a problem on
| the Code Golf Stackexchange.
|
| My rewrite-case solution condenses (by removal of all
| non-essential spaces) to 69 bytes if the symbols are in a
| hash table K, and the input list is in a variable y,
| rather than a big literal: (rewrite-case
| x y((@[K @sym]* . @rest)^(,sym _ . ,rest))(@else else))
|
| A one-letter name could be chosen for the macro.
| Furthermore, one-letter names could be chosen for sym,
| rest and else: 20> (r x y((@[K @s] * .
| @r)^(,s _ . ,r))(@e e)) (foo bar * select _ fox
| where _ bravo)
|
| Still working! Now down to 43 bytes.
|
| (It's not because I _cannot_ that I _do not_ do this with
| all my code.)
|
| Doh, why use . ,r in the backquote if we are golfing?
| That should be ,*r: using the splice operator, like
| Common Lisp's or Scheme's ,@, getting us to 42 bytes:
| 20> (r x y((@[K @s] * . @r)^(,s _ ,*r))(@e e)) (foo
| bar * select _ fox where _ bravo)
|
| I suspect Code Golf Stackechange regulars could get it
| down to way in one of the dedicated golfing languages
| like Retina or what have you.*
|
| What else can we do? The @[K @s] predicate syntax could
| be replaced by a custom operator defined by _defmatch_ ,
| looking like @(K s). 21> (defmatch k
| (sym) ^@[K (sys:var ,sym)]) k 22> (r x y((@(k
| s) * . @r)^(,s _ ,*r))(@e e)) ;; 41 (foo bar *
| select _ fox where _ bravo)
|
| Just noticed the ) * is not fully golfed:
| 23> (r x y((@(k s)* . @r)^(,s _ ,*r))(@e e)) ;; 40
| (foo bar * select _ fox where _ bravo)
|
| The k pattern operator macro could be non-hygienic: it
| could implicitly bind a variable called _s_ :
| 24> (defmatch k () ^@[K @s]) k 25> (r x
| y((@(k)* . @r)^(,s _ ,*r))(@e e)) ;; 38 (foo bar *
| select _ fox where _ bravo)
|
| These last few feel like cheating because they are too
| special purpose. Defining anything you can only possibly
| use just once isn't making the overall program smaller.
| rscho wrote:
| you'll never get J/K brevity though. Because in
| opposition to Lisp where control flow is explicit in the
| syntax, most of the control flow in array languages is
| implicit. I love lisp too, but for some (rare) use cases
| it's not the best.
| kazinator wrote:
| You will never get J/K brevity by writing a condensed for
| of normal Lisp because spaces are often required to
| separate tokens.
|
| You can get implicit control flow via macros.
|
| Furthermore, all control flow _is_ implicit is some way.
|
| Take basic old progn: (progn (foo) (bar)) evaluates (foo)
| and then control implicitly passes to (bar) because
| that's the next thing.
|
| The only control flow which is not implicit is that of a
| state machine described by a table or graph, indicating
| the next state for every input and current state.
|
| (How can you say you can't get implicit control flow in
| Lisp, when I'm using pattern matching to express if you
| see this prefix, replace it with that?)
|
| Speaking of macros, you could always resort to a macro
| like this: (j"... line noise in character
| string ...")
|
| The j macro expands the string to code at compile time.
| The code could be a _bona fide_ J implementation, or
| something like it. That 's still not down to J, because
| of the (j"") wrapping chewing up five characters. In a
| Lisp with a programmable read table, that could be
| reduced to two character framing or something.
| rscho wrote:
| > Do you? Or is today special?
|
| Today is the usual. He does that in all threads related
| to array languages.
| rscho wrote:
| it's extremely readable when used to it, though. I use J
| for exploratory data analysis. It's really, really good at
| that kind of thing.
| jiggawatts wrote:
| I've lost count of the number of times I've heard some
| theoretical mathematician say that about impenetrable
| gibberish.
| rscho wrote:
| Then you have a different use case. Doesn't mean
| theoretical mathematicians are wrong for their use case.
| I do epidemiology, not pure maths though.
| dang wrote:
| " _Please don 't post shallow dismissals, especially of
| other people's work. A good critical comment teaches us
| something._"
|
| Dismissing what you don't understand because it is
| unfamiliar is the essence of a shallow dismissal, no? And
| you did it twice in this thread. The first was a blessing
| in disguise because of geocar's excellent reply, but now
| you're just repeating it, with added name-calling. Please
| don't do that here.
|
| https://news.ycombinator.com/newsguidelines.html
| secondcoming wrote:
| What part of finance uses this language? Is it in widespread
| use, or is like Goldman Sachs' proprietary language (I forget
| the name)
| bidirectional wrote:
| Various parts, particularly in markets, from pricing quants
| to high-frequency traders. It is fairly widespread.
| Barclays, JPM, UBS, Morgan Stanley, HSBC are some of the
| big names, then you have loads of smaller firms.
| mlang23 wrote:
| The latest language which fits quote 1 for me was Haskell. Even
| though I already had some functional background (Lisp), it took
| me seemingly forever to actually grok purely functional
| programming. But once it clicked, it felt like stepping up on a
| ladder. My perspective on other languages changed as well.
| an1sotropy wrote:
| For me it was Prolog. I came to a class, which used Prolog,
| with a bad attitude of "whatever I can think of, I can
| program in C". Luckily for me I was schooled.
| geocar wrote:
| I have a common-lisp background, and I learned Haskell at the
| insistence of a former colleague. I also know k, and have
| since learned some APL and j. I would like to try and suggest
| to you my perspective:
|
| The jump from Python to Haskell - or really anything _along
| that way_ is like talking about a ladder of computing. You
| start at one end, and you are climbing upwards. And every
| step you take, you can look down and see all of the things
| you knew before, but with greater perspective.
|
| And Haskell? Well, it's definitely pretty far up the ladder.
| If you get Haskell, you feel like you _really_ understand
| what 's going on. I know pg was talking about lisp when he
| was thinking blub, but in some blubish respects, Haskell is a
| better lisp than lisp.
|
| But see, going from Haskell (or really anything) to Iverson
| is like, listen: Forget the ladder, because a ladder only
| goes up and down. Iverson is _sideways_. It is in this way,
| like adding depth to flatland, that Arrays are an even bigger
| deal than you can possibly imagine until you go there.
| mamcx wrote:
| For my was Rust. I done like 12(?) Langs before, including F#
| that also was a change of mind, but the first 2 or 3 months
| of Rust I fell like an _idiot_ looking _intensely_ to a wall.
| I start to think my 20 years programming were a big fat lie.
|
| I can't believe why it feels so hard? !I already know pascal
| and obj-c and F#!, kind of similar, no?
|
| Now I feel rust so easy (as python easy!) that is weeeeeeird.
| (btw: I think is months now where I never think I have meet
| an error or situation that truly confuse me).
| FemmeAndroid wrote:
| This is very exciting. I've been really inspired by the passion
| Conor Hoekstra has for APL and J. His other podcast (Algorithms +
| Data Structures = Programming) is a lot of fun, and his YouTube
| videos are very educational, but I've really wanted something
| like this where he can interact with other experts outside of the
| C++ world.
|
| Instant subscribe from me.
| chewxy wrote:
| this seems like it's very very niche (and very specific to the
| niche that I am interested in!)
| bidirectional wrote:
| True, but I also feel like array programming is seen as less
| prevalent in industry than it really is due to a lack of online
| community (e.g. Haskell as a language probably has an order of
| magnitude more content online, let alone the functional
| paradigm).
|
| In any financial centre there are hundreds of (often very well-
| paying) jobs using these languages (well mainly k and its ilk).
| enriquto wrote:
| A bit sad that this interesting content is not available for
| audio/video impaired readers.
| tfsh wrote:
| Chrome now provides on-device powered live captions (which
| hooks into any chrome originating audio) -
| chrome://settings/accessibility -> toggle "Live Captions"[1]
| which could help alleviate some of the limitations for audio
| impaired viewers
|
| 1: https://support.google.com/chrome/answer/10538231?hl=en
| enriquto wrote:
| > on-device powered live captions
|
| I hate this. What were they thinking about? Why not a damn
| text file that people can grep?
| uryga wrote:
| probably because that's a very niche usecase and most
| people just want some video captions :)
|
| (don't get me wrong, what you describe would be cool and
| useful! but i can't imagine a lot of people would use it)
| Y_Y wrote:
| Samsung's bastard version of Android had a similar "Automated
| Subtitles" feature. It's decent for watching videos with the
| phone on silent, but it's pretty crap when there are lots of
| proper nouns and unusual jargon, as I imagine this podcast
| has.
| iudqnolq wrote:
| So does stock Android, at least the second (?) latest
| version. (I can never keep track, but I think my phone was
| eol'ed before the latest version ...)
| jasode wrote:
| _> Chrome now provides on-device powered live captions [...]
| which could help alleviate some of the limitations for audio
| impaired viewers_
|
| That's a great feature! But it also highlights the limited
| accuracy of the AI machine learning algorithm for _technical
| topics with jargon_. E.g., at 27m00s, the caption algorithm
| incorrectly transcribes it as as _" APL is joked about as a
| right only language"_ -- but we know the speaker actually
| said, _" APL is joked about as a write-only language"_. And
| the algorithm incorrectly transcribes _" oversonian
| languages"_ when it's actually _" Iversonian languages"_.
|
| The algorithm also doesn't differentiate multiple speakers
| and the generated text is just continuously concatenated even
| as the voices change. Therefore, an audio-impaired wouldn't
| know which person said a particular string of words.
|
| This is why podcasters still have to _pay humans_ (sometimes
| with domain knowledge) to carefully listen to the audio and
| accurately transcribe it.
| geocar wrote:
| I know the joke is that APL is a write-only language, but
| it somehow seems more true to say it is a right-only
| language.
|
| I am nonplussed about AI/ML in general but this accidental
| wisdom is worth meditating on even if it didn't come from a
| human.
| rak1507 wrote:
| I have just finished transcribing it (quite roughly)
| https://gist.github.com/rak1507/3aec8c0b720e6d8a9ef121fc14e4...
| enriquto wrote:
| wow! much appreciated, many thanks!
| rak1507 wrote:
| No problem
| slver wrote:
| Or patience impaired. It's a whole hour.
| jasode wrote:
| Fyi if you weren't aware ... most podcasts don't have text of
| the audio because high-quality (accurate) transcription of
| podcasts _costs money_. Example rates:
| https://www.google.com/search?q=podcast+transcription+servic...
|
| So this thread's podcast of 52 minutes of a complex technical
| topic with multiple speakers could cost ~$200. A programming-
| related podcast is already a niche topic with a tiny audience
| and an Array Languages podcast is an even tinier subset of that
| so the cost might not be justified.
|
| I suppose podcasts could be uploaded to Youtube and let their
| speech-to-text algorithm do an auto-transcribe. However, the
| A.I. algorithm is not good at tech topics with industry
| jargon/acronyms and the resultant transcription will be
| inaccurate.
| kwhitefoot wrote:
| Presumably there was a script or at least a summary. Why not
| publish that as well as any slides used?
| bidirectional wrote:
| Why would there be? It's a recorded conversation between a
| group of people. Some of them may have some rough notes but
| maybe not even that.
| abrudz wrote:
| There was no script or summary, nor any slides. It was a
| completely organic conversation.
| anchpop wrote:
| I make transcripts of all my work using Descript. It uses
| Google's speech-to-text algo (same as the one in youtube
| presumably) and gives you a transcript you can then edit. It
| costs $15/month I believe, and you have to spend some time
| editing the transcript that realistically won't be read by
| many, but it works pretty well ime (no affiliation besides
| being a happy customer)
| abnercoimbre wrote:
| Right. A high-quality podcast is already lots of pre- and
| post-production work on just the audio. I use Rev which
| hires captioners on my behalf [0] but it's also expensive.
| I use it sparingly.
|
| [0] https://www.rev.com/
| dave84 wrote:
| Thanks for bringing Descript to my attention. Do you use
| any of the production aspects of it?
___________________________________________________________________
(page generated 2021-05-20 23:03 UTC)