[HN Gopher] Intel Hyperscan is a high-performance multiple regex...
___________________________________________________________________
Intel Hyperscan is a high-performance multiple regex matching
library
Author : wslh
Score : 29 points
Date : 2022-09-12 20:40 UTC (2 hours ago)
(HTM) web link (intel.github.io)
(TXT) w3m dump (intel.github.io)
| vinkelhake wrote:
| There's also Vectorscan[1], which is a fork of Hyperscan that
| runs on more platforms.
|
| [1] https://github.com/VectorCamp/vectorscan
| smoldesu wrote:
| I'd be really curious to see how it optimizes for ARM. x86 is
| great for kitchen-sink optimizations like this, but with ARM
| you have a much smaller extension set to target. Hell, you'd
| probably be stuck with NEON unless you figured out a way to
| GPU-accelerate the process.
| kissgyorgy wrote:
| In our firmware extraction suite, we started searching patterns
| with Yara, but it was so slow we switched to Hyperscan:
| https://github.com/onekey-sec/unblob/blob/main/unblob/finder...
|
| We are quite happy with the huge performance boost, never looking
| back :) Yara was even segfaulted when we tried to extract an
| Ubuntu ISO image with it, never had problems like that with
| Hyperscan.
|
| The only problem with Hyperscan is that it only supports Intel
| CPUs (it has some hand-crafted assembly), so it doesn't work on
| Apple M1 Macs, but there is a fork called VectorScan, which is
| working on ARM: https://github.com/VectorCamp/vectorscan
|
| We implemented a couple of small DSL classes in Python to be able
| to define YARA-like rules, it might be useful for you too, check
| it out here: https://github.com/onekey-
| sec/unblob/blob/cdd7a46667ffdfdfae...
| twhitmore wrote:
| Capturing is not supported. That eliminates many normal regex
| usecases, where you want to be able to capture & interpret the
| data.
| dang wrote:
| Related:
|
| _Hyperscan: High-performance multiple regex matching library
| from Intel_ - https://news.ycombinator.com/item?id=21873557 - Dec
| 2019 (53 comments)
|
| _Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs_
| - https://news.ycombinator.com/item?id=19270199 - Feb 2019 (36
| comments)
|
| _Multiple regex performance shootout: RE2 vs. Intel 's
| Hyperscan_ - https://news.ycombinator.com/item?id=14608663 - June
| 2017 (55 comments)
|
| _Hyperscan, a high-performance multiple regex matching library_
| - https://news.ycombinator.com/item?id=10420295 - Oct 2015 (19
| comments)
| gnfargbl wrote:
| One of the Hyperscan developers commented on HN slightly over a
| year ago [1] that Intel had basically abandoned Hyperscan due to
| lack of sufficiently vocal customer interest. I hope that has
| changed, or is in the process of changing, because Hyperscan
| remains amazing.
|
| [1] https://news.ycombinator.com/item?id=27421665
| jandrese wrote:
| The official repo[1] is looking pretty dead.
|
| Edit: This is a surprisingly heavy build. Over 15 minutes on my
| somewhat older i5. I suggest using make -j instead of cmake
| -build to speed it up. Decent number of compiler warnings too.
|
| [1] https://github.com/intel/hyperscan
| Cu3PO42 wrote:
| In sufficiently new versions of cmake you can use cmake
| --build -j as well. It will be translated to whatever
| argument is required for the underlying build system to use
| parallel compilation, e.g. -j for make and -m for msbuild.
| jandrese wrote:
| It just seemed excessive for a regular expression library.
| For comparison the most recent version of PCRE built in
| almost exactly 26 seconds.
|
| Sadly the hyperscan build crapped out for me in the linking
| phase, failing to find a bunch of coreX and avx optimized
| memset functions.
| KerrAvon wrote:
| That post and the ensuing discussion are somewhat depressing. I
| am curious as to how the team's progress was measured; did the
| team's manager not know they needed to solicit user
| testimonials? Someone had to be responsible for the existence
| of the team and their productivity. What was the disconnect?
| jiggawatts wrote:
| Not enough three letter agencies in the US to fund big brother
| at scale?
|
| (One if the use cases of this library is scanning through bulk
| intercept data streams for a set of key words.)
| [deleted]
| Lammy wrote:
| Best throw away all your CPUs then too:
|
| https://groups.google.com/g/comp.arch/c/UXEi7G6WHuU/m/Z2z7fC.
| ..
|
| https://vaibhavsagar.com/blog/2019/09/08/popcount/
|
| https://arxiv.org/abs/1611.07612
| fred123 wrote:
| Somewhat related: I built a library to speed up matching many
| regexes with mostly mismatches by adding non-regex pre matchers.
| https://github.com/Quantco/multiregex
| cl0ckt0wer wrote:
| C# has also recently put a lot of work into speeding up regex. MS
| put up a post about it a while ago going into the specifics. I
| expected the vectorization. I did not expect source code
| generation.
|
| https://devblogs.microsoft.com/dotnet/regular-expression-imp...
___________________________________________________________________
(page generated 2022-09-12 23:00 UTC)