[HN Gopher] C++ String Conversion: Exploring std:from_chars in C...
___________________________________________________________________
C++ String Conversion: Exploring std:from_chars in C++17 to C++26
Author : jandeboevrie
Score : 64 points
Date : 2024-10-14 02:43 UTC (20 hours ago)
(HTM) web link (www.cppstories.com)
(TXT) w3m dump (www.cppstories.com)
| lynx23 wrote:
| Whoever introduced the rule to automatically delete :: in titles
| on a hacker site should be made to rethink their decisions. Its a
| silly rule. It should go.
| MathMonkeyMan wrote:
| It would be an interesting piece to learn why.
|
| Maybe it was for ::reasons::.
| lynx23 wrote:
| I am sure the intern who wrote a rule for the Apple VoiceOver
| speech synthesizer to special case the number 18 being
| pronounced english while the synth voice is set to german
| imagined to have a good reason at the time as well. However,
| that desn't make ther decision less stupid. "Vierzehn Uhr",
| "Funfzehn Uhr", "Sechzehn Uhr", "Siebzehn Uhr", "Eighteen
| Uhr".
| pjmlp wrote:
| Most of those silly rules can be overriden after submission,
| there is a timeout to enforce the desired title instead by
| editing it.
| dgrunwald wrote:
| Caution with these functions: in most cases you need to check not
| only the error code, but also the `ptr` in the result. Otherwise
| you end up with `to_int("0x1234") == 0` instead of the expected
| `std::nullopt`, because these functions return success if they
| matched a number at the beginning of the string.
| pilif wrote:
| how can this be the ~5th iteration of a very wide-spread use-
| case and still contain a footgun?
|
| The API looks like it's following best-practice, with a
| specific result type that also contains specific error
| information and yet, that's not enough and you still end up
| with edge-cases where things look like they're fine when they
| aren't.
| amelius wrote:
| I suppose the reason is that sometimes you want to parse more
| than just the number. For example, numbers separated by
| commas. In that case you will have to call the function
| repeatedly from your own parsing routine and advance the
| pointer.
| gpderetta wrote:
| Yes, if you are parsing in-place numbers in an existing
| string you do not necessarily know a-priori where the
| number end. You could have extra logic to look ahead to the
| end of the number before passing it from_chars, but a) it
| would require an additional pass and b) you could end up
| replicating part of the implementation of from chars.
|
| from_chars is supposed to be a low lever routine, it must
| be possible to use it correctly, but it is not necessarily
| the most ergonomic.
| khwrht wrote:
| That is a good question. The C++ stdlib has some truly
| bizarre APIs. I wonder if they should freeze std and work on
| std2.
| otabdeveloper4 wrote:
| from_chars is the correct API here. When you're parsing
| decimal numbers you want to do it with streaming semantics.
| mort96 wrote:
| Hm but there's nothing streaming about it? You need the
| entire numeric string in memory
| Leherenn wrote:
| I think they meant the other way around: you can have a
| single string in memory containing multiple numbers, and
| process that string as a stream.
| jdbdndj wrote:
| I don't see a foot gun, you just need to check if you have
| consumed the whole input. Which is the norm in nearly any
| streaming api
| mort96 wrote:
| In what way is it streaming? It takes at least the entire
| numeric string as a string in memory, you can't stream in
| more data as needed
| randomNumber7 wrote:
| I didn't see a foot gun either, but somehow my foot went
| missing.
| pilif wrote:
| I get that this is a super low level API, but yet, my
| expectation about an API that parses a buffer with length
| to a number and which has a specific enum for error cases
| as its return type would be that when asked to parse "9not
| a number" that I would get an error response and not a "the
| number you asked me to parse is 9 and everything went well"
| as the result.
|
| The whole reason for returning a result enum would be so I
| don't forget to also check somewhere else whether something
| could possibly have gone wrong.
|
| Especially when there is a specific result type.
| gpderetta wrote:
| But now you need a different API to compute where a
| number starts and end and that API must use exactly the
| same definition of number.
| pilif wrote:
| There could be an enum value for "read a partial number"
| einpoklum wrote:
| Note that this won't work (AFAICT) with Unicode strings and non-
| western-arabic digits, e.g.: std::u8string_view
| chars{ u8"1234" }; int value; enum { digit_base =
| 10 }; auto [ptr, ec] = std::from_chars(
| chars.data(), chars.data() + chars.size(), value, digit_base);
| return (ec == std::errc{}) ? value : -1;
|
| will fail to compile due to pointer incompatibility.
| cout wrote:
| What would you suggest instead?
| jcelerier wrote:
| std::from_chars / std::to_chars are explicitly made to only
| operate in the C locale, so basically ASCII texts. It's not
| meant for parsing user-input strings but rather text protocols
| with maximal efficiency (and locale support prevents that).
|
| E.g. "1234" isn't as far as I know a valid number to put in a
| json file, http content-length or CSS property. Maybe it'd be
| ok in a CSV but realistically have you ever seen a big data csv
| encoded in non-ascii numbers?
| gpderetta wrote:
| This way your spreadsheet could interpret the data it loads
| from a CVS differently depending on on the locale of the
| machine it is running on.
|
| Of course nobody would design an application like that.
| einpoklum wrote:
| > have you ever seen a big data csv encoded in non-ascii
| numbers?
|
| Let me ask you this: How much data have you processed which
| comes from human user input in Arabic-speaking countries?
| cherryteastain wrote:
| Wish they returned std::expected<T, std::errc> instead of the
| weird from_chars_result struct
| Longhanks wrote:
| std::expected is new in C++23, std::from_chars was introduces
| in C++17. Obviously, 2023 features were not available in 2017.
| Changing the return type now would break everybody's code, with
| very little benefit, you can easily wrap std::from_chars.
| criddell wrote:
| If returning std::expected makes more sense, why not make it
| the primary signature and create a wrapper to maintain
| compatibility with old code?
| Thorrez wrote:
| Then you would have to use 2 names: the primary name and
| the wrapper name. What would they be? Using 2 names wastes
| more of the namespace, and will confuse people. If the
| wrapper name isn't from_chars, then people's code will
| break when upgrading.
| criddell wrote:
| Oh right. A different return type isn't enough to
| differentiate one function from another.
| cherryteastain wrote:
| They could add an overload like std::expected<T, std::errc>
| from_chars(std::string_view). That way, since the input
| arguments are different, there'd be no issues about overload
| resolution.
| gpderetta wrote:
| But overloads providing slightly different interfaces are a
| bane to generic programming.
|
| Overloads are already confusing, if they can't be used
| generically there is really no point in reusing the name.
| jlarocco wrote:
| Having an inconsistent, special case way of doing something
| in the name of backwards compatibility is the defining
| characteristic of C++.
| eMSF wrote:
| from_chars_result closely matches the strtol line of functions
| we've had for decades. Not returning the end position would be
| weird here!
| userbinator wrote:
| _Wasn't the old stuff good enough? Why do we need new methods? In
| short: because from_chars is low-level, and offers the best
| possible performance._
|
| That sounds like marketing BS, especially when most likely these
| functions just call into or are implemented nearly identically to
| the old C functions which are already going to "offers the best
| possible performance".
|
| _I did some benchmarks, and the new routines are blazing
| fast![...]around 4.5x faster than stoi, 2.2x faster than atoi and
| almost 50x faster than istringstream_
|
| Are you sure that wasn't because the compiler decided to optimise
| away the function directly? I can believe it being faster than
| istringstream, since that has a ton of additional overhead.
|
| After all, the source is here if you want to look into the
| horse's mouth:
|
| https://raw.githubusercontent.com/gcc-mirror/gcc/master/libs...
|
| Not surprisingly, under all those layers of abstraction-hell,
| there's just a regular accumulation loop.
| j16sdiz wrote:
| Enabling new static optimization is a good, no?
| OvbiousError wrote:
| I agree with some of this, and the author could've made a
| better case for from/to_chars:
|
| - Afaik stoi and friends depend on the locale, so it's not hard
| to believe this introduced additional overhead. The implicit
| locale dependency is also often very surprising.
|
| - std::stoi only accepts std::string as input, so you're forced
| to allocate a string to use it. std::from_chars does not.
|
| - from/to_chars don't throw. As far as I know this won't affect
| performance if it doesn't happen, it does mean you can use
| these functions in environments where exceptions are disabled.
| eptcyka wrote:
| Locale env stuff is inherently thread unsafe, which is the
| main reason to never rely on it.
| deeringc wrote:
| There's also the new Ryu algorithm that is being used, which
| is probably the biggest speed up.
|
| https://github.com/ulfjack/ryu
| badmintonbaseba wrote:
| AFAIK the state of the art now is "dragonbox":
|
| https://github.com/jk-jeon/dragonbox
| blux wrote:
| Did you verify their claims or are you just calling BS and
| that's it? The new functions are in fact much faster than their
| C equivalent (and yes, I _did_ verify that).
| userbinator wrote:
| Care to explain and show the details?
|
| "Extraordinary claims require extraordinary evidence."
| j16sdiz wrote:
| How not?
|
| It compiles from sources, can be better in-lined, benefits
| from dead code elimination when you don't use unusual
| radix. It also don't do locale based things.
| nmeofthestate wrote:
| Your original claim "I've not checked but this guy, and by
| extension the C++ standards committee who worked on this
| new API, are probably full of shit" was pretty
| extraordinary.
| userbinator wrote:
| Look at the compiler-generated instructions yourself if
| you don't believe the source that I linked; in the cases
| I've seen all the extra new stuff just adds another layer
| on top of existing functions and if the former are faster
| the latter must necessarily also be.
|
| The standards committee's purpose is to justify their own
| existence by coming up with new stuff all the time. Of
| course they're going to try to spin it as better in some
| way.
| blux wrote:
| I wrote this library once; https://github.com/ton/fast_int.
|
| Removed `std::atoi` from the benchmarks since it was
| performing so poorly; not a contender. Should be easy to
| verify.
|
| Rough results (last column is #iterations):
| BM_fast_int<std::int64_t>/10 1961 ns
| 1958 ns 355081 BM_fast_int<std::int64_t>/100
| 2973 ns 2969 ns 233953
| BM_fast_int<std::int64_t>/1000 3636 ns
| 3631 ns 186585 BM_fast_int<std::int64_t>/10000
| 4314 ns 4309 ns 161831
| BM_fast_int<std::int64_t>/100000 5184 ns
| 5179 ns 136308
| BM_fast_int<std::int64_t>/1000000 5867 ns
| 5859 ns 119398
| BM_fast_int_swar<std::int64_t>/10 2235 ns
| 2232 ns 316949
| BM_fast_int_swar<std::int64_t>/100 3446 ns
| 3441 ns 206437
| BM_fast_int_swar<std::int64_t>/1000 3561 ns
| 3556 ns 197795
| BM_fast_int_swar<std::int64_t>/10000 3650 ns
| 3646 ns 188613
| BM_fast_int_swar<std::int64_t>/100000 4248 ns
| 4243 ns 165313
| BM_fast_int_swar<std::int64_t>/1000000 4979 ns
| 4973 ns 140722 BM_atoi<std::int64_t>/10
| 10248 ns 10234 ns 69021
| BM_atoi<std::int64_t>/100 10996 ns
| 10985 ns 63810 BM_atoi<std::int64_t>/1000
| 12238 ns 12225 ns 56556
| BM_atoi<std::int64_t>/10000 13606 ns
| 13589 ns 51645 BM_atoi<std::int64_t>/100000
| 14984 ns 14964 ns 47046
| BM_atoi<std::int64_t>/1000000 16226 ns
| 16206 ns 43279 BM_from_chars<std::int64_t>/10
| 2162 ns 2160 ns 302880
| BM_from_chars<std::int64_t>/100 2410 ns
| 2407 ns 282778 BM_from_chars<std::int64_t>/1000
| 3309 ns 3306 ns 208070
| BM_from_chars<std::int64_t>/10000 5034 ns
| 5028 ns 100000
| BM_from_chars<std::int64_t>/100000 6282 ns
| 6275 ns 107023
| BM_from_chars<std::int64_t>/1000000 7267 ns
| 7259 ns 96114 BM_fast_float<std::int64_t>/10
| 2670 ns 2666 ns 262721
| BM_fast_float<std::int64_t>/100 3547 ns
| 3542 ns 196704 BM_fast_float<std::int64_t>/1000
| 4643 ns 4638 ns 154391
| BM_fast_float<std::int64_t>/10000 5056 ns
| 5050 ns 132722
| BM_fast_float<std::int64_t>/100000 6207 ns
| 6200 ns 111565
| BM_fast_float<std::int64_t>/1000000 7113 ns
| 7105 ns 98847
| majoe wrote:
| A few months ago I optimized the parsing of a file and did some
| micro benchmarks. I observed a similar speed-up compared to
| stoi and atoi (didn't bother to look at stringstream). Others
| already commented, that it's probably due to not supporting
| locales.
| ftrobro wrote:
| Integers are simple to parse, but from_chars is a great
| improvement when parsing floats. It's more standardized on
| different platforms than the old solutions (no need to worry
| about the locale, for example whether to use comma or dot as
| decimals separator) but also has more reliable performance in
| different compilers. The most advanced approaches to parsing
| floats can be surprisingly much faster than intermediately
| advanced approaches. The library used by GCC since version 12
| (and also used by Chrome) claims to be 4 - 10 times faster than
| old strtod implementations:
|
| https://github.com/fastfloat/fast_float
|
| For more historical context:
|
| https://lemire.me/blog/2020/03/10/fast-float-parsing-in-prac...
| deeringc wrote:
| You might want to watch this releavnt video from Stephan T.
| Lavavej (the Microsoft STL maintainer):
| https://www.youtube.com/watch?v=4P_kbF0EbZM
| userbinator wrote:
| I don't need to listen to what someone says if I can look at
| the source myself.
| secondcoming wrote:
| You profiled the code in your head?
| deeringc wrote:
| I believe the impl you link to is not fully standards
| compliant, and has an approximate soln.
|
| MSFT's one is totally standards compliant and it is a very
| different beast:
| https://github.com/microsoft/STL/blob/main/stl/inc/charconv
|
| Apart from various nuts and bolts optimizations (eg not
| using locales, better cache friendless, etc...) it also
| uses a novel algorithm which is an order of magnitude
| quicker for many floating points tasks
| (https://github.com/ulfjack/ryu).
|
| If you actually want to learn about this, then watch the
| video I linked earlier.
| flqn wrote:
| They're locale independent, which the C stol, stof, etc
| functions are not.
| a1369209993 wrote:
| Yes, exactly. Which means that, while the speed gains _are_
| real, they only apply in cases where your libc is dangerously
| defective.
| halayli wrote:
| youre answer shows dunning-kruger is full effect.
| dexen wrote:
| For sake of example: a "locale-aware" number conversion routine
| would be the worst possible choice for parsing incoming network
| traffic. Beyond the performance concerns, there's the
| significant semantic difference in number formatting across
| cultures. Different conventions of decimal or thousands coma
| easily leading to subtle data errors or even security concerns.
|
| Lastly, having a simple and narrowly specified conversion
| routines allows one to create a small sub-set of C++ standard
| library fit for constrained environments like embedded systems.
| doug_durham wrote:
| I get that. However then they should name the function and
| put highly visible disclaimers in the documentation.
| Something like "from_ascii" instead of "from_chars". Also the
| documentation, including this blog post should be very clear
| that this function is only suitable for parsing machine to
| machine communications and should never be used for human
| input data. There is clearly a place for this type of
| function, however this blog post miscommunicates this in a
| potentially harmful way. When I read the post I presumed that
| this was a replacement for atoi() even though it had a
| confusing "non-locale" bullet point.
| adev_ wrote:
| > Not surprisingly, under all those layers of abstraction-hell,
| there's just a regular accumulation loop.
|
| Your dismissive answer sounds so much like the one of a typical
| old-C style programmer that underestimate by 2 order of
| magnitude what compiler inlining can do.
|
| Abstraction, genericity and inlining on a function like
| from_chars is currently exactly what you want.
| userbinator wrote:
| It's my experience that says inlining only looks great in
| microbenchmarks but is absolutely horrible for cache usage
| and causes other things to become slower.
| captainmuon wrote:
| I wonder why it is called `from_chars` and not `to_number` or
| similar. It's obvious what you are converting from, because you
| have to write down the argument `from_chars(someString)`, but you
| don't see what is coming out.
| Someone wrote:
| As you indicate, you _do_ see what you're putting in, and that
| includes an argument holding a reference to the result of the
| conversion.
|
| What's coming out is a _std::from_chars_result_ : a status code
| plus an indicator how much of the data was consumed.
|
| What to name this depends on how you see this function. As part
| of a function on number types, _from_chars_ is a good names. As
| part of a function on strings, _to_int_ / _to_long_ /etc are
| good names. As freestanding functions, _chars_to_int_ (ugly,
| IMO), _parse_int_ (with _parse_ sort-of implying taking a
| string) are options.
|
| I can see why they went for _from_chars_. Implementations will
| be more dependent on the output type than on character
| pointers, it's more likely there will be more integral types in
| the future than that there will be a different way to specify
| character sequences, and it means adding a single function
| name.
| pavlov wrote:
| Maybe "number" is too ambiguous because they'd have to define
| that "in this context a number means a float or integer type
| only." The C++ standard also includes support for complex
| numbers.
| beyondCritics wrote:
| As a matter of fact, this overloads the name and hence gives
| less hassle for generic coding.
| criddell wrote:
| The author lists sprintf as one of the ways you can convert a
| string to a numbers. How would that work?
| murderfs wrote:
| It's not a list of ways to convert strings to numbers, it's a
| list of string conversion functions (i.e. including the other
| direction). to_string is also listed there.
| randomNumber7 wrote:
| Did you mean sscanf? You could for example parse a number into
| a double variable by using "sscanf(str, "%d", &x)"
|
| You can even parse a whole line of a csv file with multiple
| numbers in one call "sscanf(str, "%d;%d;%d\n, &d1, &d2, &d3)"
| criddell wrote:
| No, the article lists sprintf/snprintf.
|
| Another person already correctly pointed out that the author
| was listing a bunch of functions for number and string
| conversions in general.
| vaylian wrote:
| Example from the website: const std::string str {
| "12345678901234" }; int value = 0;
| std::from_chars(str.data(),str.data() + str.size(), value);
|
| On the third line: Why can I just pass in `value` like this?
| Shouldn't I use `&value` to pass in the output variable as a
| reference?
| hardlianotion wrote:
| In the function signature, value is a reference. You can think
| of reference as being a pointer that points to a fixed location
| but with value semantics.
|
| So you can dip value into the function call and the function
| can assign to it, as it could to data pointed to by a pointer.
| _gabe_ wrote:
| I saw another commenter explain that it's passed by reference,
| but I agree with you. The C++ Core Guidelines even mention that
| it's better to use raw pointers (or pass by value and return a
| value) in cases like this to make the intent clear.
|
| https://isocpp.org/wiki/faq/references#call-by-reference
| cornstalks wrote:
| nit: I don't think the Core Guidelines actually suggests it's
| better. It's just "one style." There are pros and cons to
| both styles.
| nmeofthestate wrote:
| A pointer parameter can be null and it doesn't make sense for
| this parameter to be null, so IMO a reference is the better
| choice here.
|
| A non-const reference is just as clear a signal that the
| parameter may be modified as a non-const pointer. If there's
| no modification const ref should be used.
| jdashg wrote:
| It's about clarity of intent at the call site. Passing by
| mutable ref looks like `foo`, same as passing by value, but
| passing mutability of a value by pointer is textually
| readably different: `&foo`. That's the purpose of the pass
| by pointer style.
|
| You could choose to textually "tag" passing by mutable ref
| by passing ` _& foo` but this can rub people the wrong way,
| just like chaining pointer outvars with `&_out_foo`.
| gpderetta wrote:
| In theory a from_char with an optional output parameter
| could be useful to validate that the next field is a number
| and/or discard it without needing to parse it; it might
| even be worth optimizing for that case.
| trealira wrote:
| &value would be a pointer to that integer. Instead, it's using
| references, which also use the & character. References use
| different syntax, but they're like pointers that can't be
| reassigned. Examples: int x = 0; int &r =
| x; r += 1; assert(x == 1); int y; r =
| y; // won't compile void inc(int &r) { r += 1; }
| int x = 0; inc(x); assert(x == 1);
|
| The equivalent using pointers, like in C: int x
| = 0; int *p = &x; *p += 1; assert(x == 1);
| int y; p = &y; void inc(int *p) { *p += 1; }
| int x = 0; inc(&x); assert(x == 1);
| trealira wrote:
| Mistake: r = y; // won't compile
|
| This will compile. It will be effectively the same as "x =
| y". The pointer equivalent is *p = y".
|
| My apologies, as it's been a while since I've used C++.
| Dwedit wrote:
| What if you try to convert a French floating point number that
| uses a comma instead of a dot?
| jeffbee wrote:
| Then the programmer has made a mistake, because the behavior is
| the same as `strtod` in C locale, i.e. it stops parsing at the
| first comma.
|
| You should think of `from_chars` as a function that accepts the
| outputs of `to_chars`, not as a general text understander.
___________________________________________________________________
(page generated 2024-10-14 23:02 UTC)