[HN Gopher] Wild claims about K performance
___________________________________________________________________
Wild claims about K performance
Author : tosh
Score : 89 points
Date : 2021-08-31 09:25 UTC (13 hours ago)
(HTM) web link (mlochbaum.github.io)
(TXT) w3m dump (mlochbaum.github.io)
| anonu wrote:
| So the author is incredulous but provides no proof or stats, just
| a long rant.
|
| Another overlooked "performance" feature of terse languages is
| that of the developer. Vector based, concise languages give you
| more in one line of code. And I don't find that readability
| suffers. Yes you need to stare at that one line longer, but
| there's more going on.
| eatonphil wrote:
| > So the author is incredulous but provides no proof or stats,
| just a long rant.
|
| I thought the same thing but on the other hand, what can you
| do? If you're not allowed to publish benchmarks you only can
| rant like this even if it sounds petty.
|
| By getting attention here on HN it helps put pressure on
| commercial vendors to stop sharing unconfirmable benchmarks
| otherwise they will lose credibility since they've been called
| out.
|
| The author does make some good points about the fundamental
| nature of interpreters too. Although it's hard to confirm there
| too what these languages do without seeing their source.
|
| I mean that if K or some language is actually interpreting
| without a virtual machine at some stage, that is indeed going
| to be slower than any language with code running on a virtual
| machine.
|
| And also that any code that runs on a virtual machine must be
| slower than an equivalently intelligent compiled code since
| it's running on another layer of abstraction that compiled code
| isn't. ("equivalently intelligent" because yes a virtual
| machine could be faster than terrible compiled code.)
| dagw wrote:
| Here are some (micro) benchmarks of q vs python that they did
| publish: https://kx.com/blog/a-comparison-of-python-and-q-
| for-data-pr...
|
| tl;dr, while q is orders of magnitude faster that naive
| python, it is, at best, 5 times faster than optimal python
| using numpy. Based on those numbers I would be very surprised
| if an optimal C program wouldn't be significantly faster than
| both.
| mlochbaum wrote:
| Almost! I can't quite replicate these tests because it
| doesn't give the type (integer or float, and width) of the
| database columns. There are many possibilities[0] so I
| tested a few. Fortunately the testing CPU (Intel Xeon
| E5-2650 v4 @ 2.20 GHz) is comparable to mine (Intel Core
| i5-6200U CPU @ 2.30GHz). Turbo (one-core speed) is 3.00 GHz
| versus 2.80 GHz with advantage to the Xeon, but my laptop
| CPU is a year newer and on a smaller process so could have
| better IPC. Both have AVX2, and the server CPU has a much
| better cache, but I don't think Dyalog is bandwidth
| limited.
|
| I adapted the vectorized Dotsch solution, which gave the
| fastest times and is easy to write, to Dyalog APL. Pre-
| release 18.0 from when I worked there because I can't be
| bothered to do a real install, but I doubt the arithmetic
| code's been changed since 17.1. Here are results from a
| Dyalog session giving times in seconds for 8-byte floats,
| 4-byte integers, and 1-byte integers. From the article, Q
| takes 13E-3 seconds. )copy dfns
| cmpx (sA sB pA pB)-?{1e6[?] 0}"[?]4 [?] cmpx
| '(sAxpA>=pB) + sBxpA<=pB' 9.3E-3 (sA
| sB pA pB)-?{1e6[?]1e9}"[?]4 [?] cmpx '(sAxpA>=pB) +
| sBxpA<=pB' 4.6E-3 (sA sB pA
| pB)-?{1e6[?]120}"[?]4 [?] cmpx '(sAxpA>=pB) + sBxpA<=pB'
| 1.3E-3
|
| If the table consists of floats then Dyalog appears
| substantially faster, although this could plausibly be due
| to better IPC on my CPU. It could also be a real increase.
| Dyalog uses bit booleans for the comparison results which
| allows it to make smaller reads and writes, and code for
| packing results to bit booleans and multiplying floats by
| booleans does have to be written with vector intrinsics
| ([1] indicates Q doesn't pack booleans). If the benchmark
| uses 8-byte ints then they should be comparable to floats,
| and if it's using smaller ints then Dyalog is clearly much
| better.
|
| Will see if I can get numpy benchmarks running.
|
| [0] https://code.kx.com/q/basics/datatypes/
|
| [1] https://code.kx.com/q4m3/3_Lists/#323-simple-binary-
| lists
| geocar wrote:
| They're 8-byte ints, and the reported times are sum of
| 100 runs. This is what my 2019 i5 1.6ghz Macbook Air
| does: q)`sA`sB`pA`pB set'4
| 0N#1000000?100 `sA`sB`pA`pB q)\t:100
| (sA*pA>=pB)+sB*pA<=pB 60
| mlochbaum wrote:
| Where do you get this information? Do you have some
| connection to the article?
|
| It appears this benchmark uses vectors of a quarter-
| million, not a million, elements? My understanding is
| that 4 0N#a redistributes elements of a into four vectors
| without repeating them, and this is what ngn/k does. The
| article says a "sample table of size one million", refers
| to "one million rows" elsewhere, and later gives timings
| that scale linearly with the number of columns, so I
| don't think that's what is meant.
|
| EDIT: Oh, there's generation code near the top. In an
| image for some reason so the Q version is transcribed
| below. Definitely a million rows. Seems both Numpy's
| random.randint and Q default to 8-byte ints? Dyalog would
| use 1-byte integers if the data fits, as it does in this
| example. N: 1000 * 1000 t:
| ([] pA: N?5; pB: N?5 sA: N?100; sB: N?100)
| mlochbaum wrote:
| Ran the following numpy program, which resulted in a time
| of 0.01116s, faster than the article's benchmark but
| slightly slower than my Dyalog timing of 0.0093s.
| Assuming relative times are consistent across CPUs, this
| would put Dyalog on 8-byte floats just ahead of Q on
| 8-byte ints. Floating-point vector instructions take
| longer than integer ones; while overflow checking could
| turn things the other way, K and Q famously don't do it.
| It's not consistent with a sum of 100 runs as geocar
| suggests. import numpy as np
| import time N = 1000 * 1000 pA =
| np.random.randint(0, 5, N) pB =
| np.random.randint(0, 5, N) sA =
| np.random.randint(0, 100, N) sB =
| np.random.randint(0, 100, N) start =
| time.perf_counter() runs = 100 for i in
| range(runs): bS = sA * (pA >= pB) + sB * (pA
| <= pB) end = time.perf_counter()
| print((end - start)/runs)
| lmm wrote:
| > I mean that if K or some language is actually interpreting
| without a virtual machine at some stage, that is indeed going
| to be slower than any language with code running on a virtual
| machine.
|
| Not necessarily. The VM itself will introduce overhead and
| potentially blow out cache etc.. You can argue, as the
| article does, that this is a small proportion of execution
| time if you're executing a lot of loops, but that's
| inherently workload-dependent. If the language being
| interpreted is denser than the VM bytecode - something that's
| believable for a language like K - then interpreting directly
| makes more efficient use of memory bandwidth than
| interpreting via bytecode.
| eatonphil wrote:
| That's an interesting idea, I missed that.
|
| Has anyone published a PoC of this (not necessarily using
| any of these languages) that would give some quantitative
| credence?
| sz4kerto wrote:
| "And also that any language with a virtual machine must be
| slower than an equivalently intelligent compiled code since
| it's running on another layer of abstraction that compiled
| code isn't. "
|
| I think this is not correct. A VM has access to the data that
| the code is operating on, while an AOT compiler doesn't. It's
| possible that a VM (eg. JVM) produces faster code than a
| normal compiler.
| icen wrote:
| Sadly it also has to consider the time it takes to compile
| code like that - we grumble, but not too loudly, at the
| time LLVM takes to compile things. If the JVM took that
| same time whilst running live that wouldn't be acceptable.
| eatonphil wrote:
| I'm talking about where the code is run. If it's run on a
| VM it's run on a VM. If it's run on the cpu by JIT-ed code
| it's not running on a VM.
|
| Edit: But fair I did say "language with a VM" which would
| include Java JIT-ed or not.
|
| Edit, edit: Modified the original comment to say "code on a
| VM" not "language with a VM".
| geocar wrote:
| > If you're not allowed to publish benchmarks you only can
| rant like this even if it sounds petty.
|
| You can repeat whole-system benchmarks like STAC[1] or that
| some bloggers do[2], which specify the hardware and software,
| inputs and outputs for a problem. You can _always_ do that.
|
| [1]: https://stacresearch.com/m3
|
| [2]: https://tech.marksblogg.com/billion-nyc-taxi-kdb.html
| eatonphil wrote:
| Interesting, but I don't understand how this is allowed
| under the license?
|
| Edit: it's not that these licenses says you cannot ever
| publish benchmarks it's that they must be approved. So I
| assume these benchmarks were approved.
| theelous3 wrote:
| I'm not familiar with k, and I'm not understanding this whole
| "no benchmarks" thing.
|
| So it's a proprietary lang? What stops benchmarks being run?
| Just how ridiculous are the licenses? Why has seemingly
| nobody run them anyway?
| tsimionescu wrote:
| Not sure about K, but I know for example that MS didn't
| allow publishing performance benchmarks of .NET (not sure
| of this changed with .NET core).
| eatonphil wrote:
| Both Oracle and MS SQL disallow this without approval.
|
| https://stackoverflow.com/a/12116865/1507139
| geocar wrote:
| > I'm not understanding this whole "no benchmarks" thing.
|
| Back in the 1980s, there was a paper published[1] that
| compared a number of database systems, and Oracle felt this
| paper harmed them. _Perhaps if the authors knew Oracle
| better, they could have made more favourable benchmarks_ ,
| they would argue, and whilst perhaps DeWitt et al could
| agree to that, they may point out they had not _intended_
| to harm Oracle, and being researchers and not marketers
| were given no notice or expectation that such a review
| _could_ harm Oracle.
|
| And so for the avoidance of doubt, Oracle, Microsoft, and
| many others (probably including KX, although I've never
| checked) insist in their licensing agreements that they be
| able to approve benchmarks, ostensibly so that they have an
| opportunity to correct any misunderstandings the author
| might have about the software before.
|
| [1]: http://pages.cs.wisc.edu/~dewitt/includes/benchmarking
| /vldb8...
| q-big wrote:
| > Back in the 1980s, there was a paper published[1] that
| compared a number of database systems, and Oracle felt
| this paper harmed them.
|
| This is rather an argument why the benchmark code should
| be made public so that other people can check it and
| vendors can post modified versions if they feel treated
| unfairly.
| eatonphil wrote:
| I think the article answers all your questions.
| tosh wrote:
| Concise languages are also more fun to use in a REPL (another
| aspect of "performance")
| icen wrote:
| You might enjoy the language created by the author of the
| article, then. BQN is a language that favours terseness, even
| above and beyond k. It is in many ways an improvement on APL
| and J, importantly through a more convenient set of operators
| than found in either.
| rak1507 wrote:
| Personally from what I've seen I wouldn't say BQN values
| terseness more than k, probably less than most other array
| langs.
|
| (Edit: if you downvoted this: why?)
| icen wrote:
| I agree; I don't think BQN _values_ terseness more than k.
| I think k values terseness a lot.
|
| I think it may be more convenient to write terse code in
| BQN over k, because there is greater facility to control
| function calls in a terse way. Things like under and over
| are quite common patterns that BQN finds easier (and
| shorter) than k.
|
| I've not spent a great deal of time yet with BQN, so this
| could be wrong.
| rak1507 wrote:
| Yeah, under and over are great (wish k had more
| 'combinators' in general), but k does manage to cram a
| lot of stuff in with overloads. I feel like k is terser
| in general but I don't really have much evidence to back
| it up, just from playing a bit with both of them.
| tomp wrote:
| The author explicitly states that K licence _prevents_ him from
| providing stats.
| gd1 wrote:
| There seem to be benchmarks available in other places:
|
| https://tech.marksblogg.com/benchmarks.html
| linspace wrote:
| I'm surprised nevertheless that this is enforceable. Is there
| a precedent of someone being sued and losing?
| eatonphil wrote:
| If you're reliant on a vendor for a service why risk
| getting on their bad side?
| linspace wrote:
| What makes you think everyone doing benchmarks is
| reliant? it could be research, it could be curiosity, you
| actually could be competing...
|
| Edit: and nevertheless is quite depressing that as a
| paying customer you should be afraid
| mlochbaum wrote:
| I wrote this article to link in response to evidence-free
| claims about K, and have in fact used it in this way twice in
| the two months since I wrote it. Didn't expect HN in general to
| take an interest. So I need to stress again that K is a really
| cool language and I like it a lot!
|
| If the article can provoke clear benchmarks against current
| Dyalog or J then that would be an ideal outcome, even if it
| proves me wrong. I already see some benchmarks linked here and
| will try to analyze these.
| IshKebab wrote:
| Yeah the fact that they forbid benchmarks pretty much tells you
| all you need to know. Anyway once you get to something that
| autovectorises array operations there's not really anywhere to go
| is there?
|
| I seriously doubt it is really any faster than Matlab, Fortran,
| Eigen, etc. etc.
| eatonphil wrote:
| > Yeah the fact that they forbid benchmarks pretty much tells
| you all you need to know.
|
| I don't think this is the case. Oracle and MS SQL have the same
| clause and it's pretty hard to believe they aren't competitive
| with postgres and mysql given the resources behind the former.
|
| More so it's just annoying of them to have this restriction.
| athrowaway3z wrote:
| Ha.
|
| This is extremely easy to believe.
| cameronh90 wrote:
| I was on a team that migrated an app from SQL Server to
| Postgres a some years ago, and the performance dropped by
| around 30%.
|
| Of course, due to the benchmark rules, I'm not allowed to
| post about it... ;)
|
| That being said, in our application it came down to three
| main issues:
|
| - Postgres doesn't allow optimizer hints, and the optimizer
| generally produced worse plans than Microsoft's (e.g. CTE
| were an optimization fence in Postgres).
|
| - Postgres at the time did not support parallel queries.
|
| - Postgres connection handling is terrible. Pgbouncer helps
| but introduces its own problems.
|
| - Database bloat / vacuum issues
|
| Generally I found SQL Server would perform very well with
| no tuning and minimal maintenance. And generally, if it
| doesn't do the right thing, a query hint will fix it.
| Postgres required a lot more configuration and testing to
| obtain acceptable performance.
|
| We still switched to Postgres. It has lots of other
| benefits - and none of the licensing.
| darksaints wrote:
| The first two are already fixed, the third is fixed in
| v14.
|
| The fourth will likely never be fixed as it is a
| byproduct of the extremely fine grained transactional
| capabilities in Postgres (which brings the advantage of
| fewer deadlocks). However, if you are disinclined to make
| that tradeoff, I believe that there is a backend in
| development that is designed to reduce bloat, using the
| new pluggable storage engine API.
| cameronh90 wrote:
| I'm aware some of these things have changed, but equally
| SQL server has likely developed since then too.
|
| That said for #1 are you saying PG now allows query hints
| or just the specific example is fixed? There are a lot of
| cases still where the optimiser does the wrong thing and
| with PG you're powerless to fix it.
|
| With #3, do you have any more information about that? I
| can't find any on a brief Google.
| darksaints wrote:
| I was specifically referring to the optimization fence on
| CTEs. PG still doesn't have query hints, and it seems
| like the core team has an ideological opposition to
| building query hint capabilities. Their stance has been
| that query hints often lead to suboptimal behavior
| because queries often don't change over time, but data
| sets do, so relying on statistics allows your query plans
| to change with different data. A query hint, on the other
| hand, may lead to an optimal query execution today, but a
| terrible one tomorrow.
|
| Over the years Postgres has steadily chipped away at
| almost all of the reasons why I would ever want query
| hints. Only two frustrations remain:
|
| * the optimizer is not aware of TOAST storage. This means
| that for tables with lots of large objects (geospatial in
| my case), the optimizer only sees the costs of the table
| with a pointer to the large object, as opposed to the
| cost of pulling table _with_ the large object.
|
| * The optimizer only has singular statically configured
| costs for functions and aggregates, but costs can
| actually dramatically vary based on the size of the data
| inputs, the size of the aggregate "window", etc.
|
| Overall, these are rarely a big enough deal for me to not
| choose Postgres. I see the default planner behavior as
| better than the default planners for both Oracle and SQL
| Server, but it is still possible that the lack of query
| hints could be a deal breaker. If you get a chance, give
| it another try...I think you might be surprised at how
| far Postgres has come since the 9.X days.
|
| As far as number 3, I guess I should clarify that they've
| dramatically reduced the connection overhead, but I
| shouldn't have said "fixed". It may or may not be reduced
| enough for your requirements. Here is a blog post with
| some links to various commits:
| https://www.depesz.com/2020/08/25/waiting-for-
| postgresql-14-...
|
| Some rudimentary benchmark comparisons here: https://gith
| ub.com/digoal/blog/blob/master/202008/20200817_0...
| eatonphil wrote:
| Where can I read more about improved connection handling
| in v14?
| darksaints wrote:
| https://www.depesz.com/2020/08/25/waiting-for-
| postgresql-14-...
| defaultname wrote:
| No one who has done serious work across the database
| platforms believes they restrict benchmarks because they're
| behind or uncompetitive. They both make very capable,
| competitive database systems.
|
| The problem, as cited in the linked page, is that "It takes
| much more work to refute bad benchmarks than to produce
| them". We see on HN with regularity where people post
| egregiously flawed benchmarks, usually to demonstrate some
| preconceived notion or other. And while that can usually be
| ascribed to ignorance or sloppiness, it's trivial for
| vendors to contrive benchmarks that are purpose-suited to
| make their own product look good and the competition look
| bad, however completely artificial and unrealistic the
| scenario is.
|
| DeWitt clauses are an abomination. They should not exist.
| But I _get_ why they exist and are threatened, even if they
| are basically never actually enforced (seriously though --
| are they _ever_ actually enforced?).
| nullc wrote:
| How is that hard to believe? Clearly Oracle itself doesn't
| believe it. :P
| jpf0 wrote:
| Without general theories, the only reasonable question is
| "Faster for What"?
|
| What are generalizable theories of 'fast'? Fewer bits moving
| less with fewer instructions, and using the most suitable,
| specialized hardware available.
|
| Even once you have a reasonably well-defined goal, if it's
| complex enough you may start hitting subsidiary questions, like
| numerical precision and stability.
| apricot wrote:
| "This car is the fastest one on the planet, but we legally forbid
| you from talking about how fast it can go."
|
| Sure.
| [deleted]
| tosh wrote:
| The notes by mlochbaum were also discussed briefly in the most
| recent ArrayCast episode on k
|
| https://www.arraycast.com/episodes/episode-08-attila-vrabecz...
___________________________________________________________________
(page generated 2021-08-31 23:02 UTC)