[HN Gopher] ---libcurl
___________________________________________________________________
---libcurl
Author : Tomte
Score : 611 points
Date : 2024-01-29 13:19 UTC (9 hours ago)
(HTM) web link (everything.curl.dev)
(TXT) w3m dump (everything.curl.dev)
| ukuina wrote:
| Wait, this option would give you a compilable C program that
| would replace the need for scripting around the original curl
| call!?
| LoganDark wrote:
| The idea is that you can copy-paste the C code into an existing
| program or at least use it as a reference to know exactly which
| libcurl API calls are needed to replicate the curl call.
| notRobot wrote:
| Yes, it has always been possible with libcurl but you had to
| write the boilerplate code yourself.
| EasyMark wrote:
| you'd probably have to modify it with a few parameters to make
| it useful for generic web pages and such, I would think of it
| as more of a base to build on so you don't have to dig through
| documents for hours
| oleg_antonyan wrote:
| I wish something like this was possible with ffmpeg
| bufo wrote:
| Seriously!!
| d3m0t3p wrote:
| That would be awesome
| r0ckarong wrote:
| The compiler has been thinking about the answer for a few
| decades ...
| donatj wrote:
| Before I found out that statically compiled ffmpeg was a
| thing, I was trying to compile ffmpeg on a 2010 Mac.
|
| It was a multi-day process and every time like two days in
| I'd hit some snag.
| a3w wrote:
| Not our problem, we just need to write the feature request ;)
|
| But yes, perhaps as a follow-up, we would need to ask for
|
| 1. a speed-up of the process or
|
| 2. find a way to look up and redistribute existing conversion
| flows
| ape4 wrote:
| Useful idea
| Waterluvian wrote:
| I use the browser equivalent of this all the time to generate
| javascript code for requests. It's very cool to see this for C
| and hopefully other languages, too.
| zokier wrote:
| One convenient thing in browser web developer tools is the
| ability to copy requests from network tab as either curl commands
| or even as javascript code. I love to see more this sort of
| things!
| dewey wrote:
| And was curl is such a "standard" to represent a request
| there's also many tools converting that curl output into native
| code (Like Go) already, which makes it very fast to reproduce
| something without manually having to set all the flags. I'm
| always happy this feature exists without even needing third
| party extensions in the browser.
| Joker_vD wrote:
| _In an enthusiastic tone of an AI enthusiast_ : Thankfully, now
| that we have ChatGPT, this feature is obsolete and the curl
| executable doesn't have to contain half-baked quines in it
| anymore!
| ijustlovemath wrote:
| It's not a quine though, as it's not producing curl itself
| tomtomtom777 wrote:
| One could argue it's a half-baked quine.
| kevindamm wrote:
| mm parbaked quine, just pop it in the compiler add a main
| driver and serve with some gengetopt for a delicious
| webcrawler!
| throwaway_08932 wrote:
| In Italian, it's common to append 'ino' or 'ina' to something
| when you want to imply it's a smaller or cuter version of
| itself.
|
| So 'quinino' might work here.
| bayindirh wrote:
| If you want something with more rhyme, poco-quine can also
| work, I guess, pq for short.
| neuromanser wrote:
| libpq is taken by PostgreSQL though ;)
| Biganon wrote:
| Reading this comment gave me a fever... I wish a plant
| could help me get better...
| bayindirh wrote:
| I'd never change --libcurl and gengetopt[0] with some output
| from some artificial thingy which babbles semi-truths which
| doesn't understand what it's doing.
|
| They are deterministic tools which does what you want in a
| battle tested way, and will let me sleep well at night, which
| is an underappreciated feature of mature programs.
|
| [0]: https://www.gnu.org/software/gengetopt/gengetopt.html
| jcul wrote:
| First I've heard of gengetopt, thanks!
| avgcorrection wrote:
| Hurray for deterministic tools! Yes, there is no reason to
| risk using an LLM when you have a code generator that you can
| trust.
| smy1es wrote:
| This kind of thing was one of the reasons Visual Basic macros for
| Microsoft Office was so successful. You can perform actions in
| Word, Excel, watch the macros they produce then customise
| themselves to your needs afterwards in code. It is a simple and
| powerful concept, so good to see it in curl.
| actionfromafar wrote:
| I was going to say, there has to be a generic pattern for
| things like this. Also made me think of AREXX on the Amiga.
| sph wrote:
| There is, but we need to throw away our outdated current
| programming model. Think Lisp or Smalltalk. There should not
| be a separation between program written in some language,
| operating system and shell [1].
|
| You'd simply run: CURL url:
| "https://example.com" method: 'post
|
| in an interactive system, that can either represent your
| shell, or your application code. We need --libcurl because
| UNIX is not an interactive environment, so there is an
| enormous abyss between runtime and compile time.
|
| (Syntax in this example from a Smalltalk-like environment I
| have been designing, should be understandable enough)
|
| ---
|
| 1: _" An operating system is a collection of things that
| don't fit in a language. There shouldn't be one."_ -- Dan
| Ingalls, 1981
| supriyo-biswas wrote:
| Given that we already have the concepts of exported entry
| points and extern in C, this should already be possible to
| a certain extent. The only other thing which needs to
| happen is that ELFs should have the concept of exported
| data structures, so while "URL" might not be a defined
| structure in the OS, something like curl can provide it.
|
| Too bad we're stuck with UNIX/POSIX model - you could even
| take this idea of exported data structures and have the
| terminal represent data in the preferred users format
| instead of having tools like jq.
| sph wrote:
| We have dlopen and we can list exported symbols, but we
| have no information about a function's arguments, ABI and
| calling convention, so it's pretty much impossible to
| turn UNIX into a fully late-bound and interactive REPL.
| Same issue with syscalls.
|
| The only way is starting from scratch, with a novel
| approach and programming model.
| overboard2 wrote:
| Or you could add some more sections to object files. Just
| make clang and/or gcc export function signatures,
| structs, enums, and typedefs
| actionfromafar wrote:
| I wonder if one of the existing interpreted languages
| (python/javascript/ruby/whatever) could maintain a patch
| for llmv/gcc which did exactly that and in the process
| make the most incredible seamless integration ever
| between itself and C (and also C++ now with its ABI
| stabilizing!)
| supriyo-biswas wrote:
| Why can't you add additional data about the data types of
| parameters in a separate ELF section? It would be only
| used by programs that look for it like a specially
| designed shell.
|
| The calling convention can be assumed to be the same as
| used by the OS/arch combination.
| sph wrote:
| You could, but you then have to recompile the world with
| this new information stored as a ELF header or something,
| and good luck if the library is not written in C (so it
| has its own conventions, ABI, memory model and binary
| format)
|
| I'm talking about the status quo today, not how you can
| improve in a perfect world where everybody adopts a
| better way of doing things.
|
| Implementing an half-baked Smalltalk layer on top of UNIX
| will not turn UNIX into a Smalltalk environment.
| Tyr42 wrote:
| PowerShell?
| xnorswap wrote:
| What you're looking for exists, it's powershell.
| Invoke-WebRequest -Method 'Get' -Uri 'https://example.com'
|
| And the response is a typed object:
|
| That you can pipe properly by piping objects not strings,
| e.g. : Invoke-WebRequest -Method 'Get'
| -Uri 'https://example.com' | Select-Object -ExpandProperty
| Headers
|
| Outputs: Key Value ---
| ----- Age 438810 Vary
| Accept-Encoding X-Cache HIT Content-
| Length 1256 Cache-Control max-age=604800
| Content-Type text/html; charset=UTF-8 Date
| Mon, 29 Jan 2024 15:08:42 GMT Expires Mon,
| 05 Feb 2024 15:08:42 GMT ETag
| "3147526947+ident" Last-Modified Thu, 17 Oct 2019
| 07:18:26 GMT Server ECS (nyb/1D15)
| freedomben wrote:
| Yes yes, power shell is powerful and really good. I
| wanted to hate it because it seems too verbose and I
| don't like the mix of capitals and hashes in names. but
| the APIs they make available from .Net are pretty
| phenomenal. Extreme verbosity aside, done day I'm have to
| seriously learn it
| actionfromafar wrote:
| Can't hate it, but I can't love it either. I wish some
| other more "normal" languages (Ruby? Python?) had a
| better "shell" story and dotnet integration.
| shirogane86x wrote:
| For simple interactive usage aliases go a long way, and
| with tab completion it gets really fast to write. For
| example, Invoke-WebRequest is iwr, Select-Object I think
| is just select. Some others are ForEach-Object which is
| %, and Where-Object can be ? Or where. Also, since
| PowerShell is case insensitive, you don't really need to
| use all those capitals - the worst you'll get is a
| warning from your editor, if it has PowerShell
| integration (around 99% sure about this)
| sph wrote:
| Can you copy and paste that snippet as is into a C#
| program? This is what I meant.
|
| Powershell has a high level object model but it doesn't
| make Windows itself anymore programmable and interactive,
| as a whole than zsh does for Linux. It is no Lisp
| Machine.
| philzook wrote:
| You might find this work and talks on liballocs by Stephen
| Kell interesting. The pitch is how to enable smalltalk like
| reflection for unix as it exists today.
| https://www.humprog.org/~stephen/#works-in-progress
| cm2187 wrote:
| Or the excel side by side xaml/UI of the wpf editor in visual
| studio.
| jasomill wrote:
| While it doesn't appear to have been updated in many years,
| Microsoft built a similarly useful tool[1] that lets you browse
| the structure of a given Office document and see C# code that
| generates various components of it.
|
| [1] https://github.com/dotnet/Open-XML-SDK/releases/tag/v2.5
| jari_mustonen wrote:
| Please also add support for different so we can write
|
| ---libcurl example.py
| boffinAudio wrote:
| Its quite effective to use ChatGPT to convert C code to Python
| ..
| a3w wrote:
| ... or "curl calls to python webscraping". Although having to
| look for the right version of libraries with different number
| names and therefore different feature sets was tedious to do
| manually, AI might just guess fast and sometimes even right
| as to which import to use.
| justinclift wrote:
| You can probably get pretty far with the generated C code, then
| look up the same curl options in the Python bindings:
|
| https://github.com/pycurl/pycurl
| 0l wrote:
| Libcurl is a c/c++ library... Just use Requests
| ok123456 wrote:
| Tools already convert a curl command line to a Python requests
| script.
|
| [1] https://github.com/spulec/uncurl
| npteljes wrote:
| Try https://curlconverter.com/ - "Convert curl commands to
| Python, JavaScript and more"
| hhthrowaway1230 wrote:
| I love this! Very nice transition for people to move from
| commandline to the library when they need to!
| xwowsersx wrote:
| Having a flag in the command line interface that spits out the
| source code of the program doing the same stuff as your command
| is pretty cool. It's like lifting the hood and showing you what's
| going on. This not only helps you get a better grip on how things
| work but also lets you make changes to fit your needs. You can
| tweak or add stuff based on what you want, making the whole
| experience user-friendly. It's all about giving users the power
| to customize things their way.
| xuhu wrote:
| I wish this also worked for Gnome settings, network settings,
| firewall config GUIs, and anything that can do things using CLI
| commands.
| INTPenis wrote:
| With dbus we're moving in that direction.
| OskarS wrote:
| It's also just great documentation for a programming library.
| Like, if you're using libcurl and realize you need to do a
| range request (or whatever), or "copy as curl" from browser
| network tab, you can just do it on the command line and add
| `--libcurl` and find out exactly how to do that with the C
| library. It's the bee's knees.
| xwowsersx wrote:
| 100%
| jraph wrote:
| Nice. Daniel Stenberg is really careful to details and dev / user
| experience. This level of polishing is astonishing.
| pbaam wrote:
| I remember this option was mentioned in a 3 hour video[1] where
| Daniel Stenberg himself went through most of the curl command
| line options.
|
| [1] https://www.youtube.com/watch?v=V5vZWHP-RqU
| ahmedfromtunis wrote:
| Can someone give an example use case for this utility? I'm really
| curious to learn how would you use this?
| Daviey wrote:
| This is really nice, but a feature that some applications
| implement that I wish was available in the library is some way of
| outputting curl command line reference which is equivalent of the
| requests being made. In-fact, I often find myself, MITM'ing
| myself using burp-suite and "copy as cURL command" feature for
| exactly this.
| p4bl0 wrote:
| What an awesome idea! That's kind of like a library documentation
| but _alive_. Very cool.
| nindalf wrote:
| One pattern I really like is opening the networks tab of the
| browser, finding the request I'm interested in and "copying as
| curl" - the browser generates the equivalent command in curl.
|
| Then I'd use something like https://curlconverter.com/ to convert
| it into request code of the language I'm using.
|
| In a way curl is like the "intermediate representation" that we
| can use to translate into anything.
| Mavvie wrote:
| This is pretty interesting. It's not like HTTP needs an
| intermediate representation, but since cURL is so ubiquitous,
| it ends up functioning as one. cURL is popular so people write
| tools that can export requests as cURL, and it's popular so
| people write tools that can import it.
| tracker1 wrote:
| I've used a similar tool as part of API logging, filtering
| out the signature on the bearer token... It's useful with
| request id for dealing with error reporting.
| everforward wrote:
| The benefit of curl over raw HTTP is the ability to strip
| away things that dont matter.
|
| Eg an HTTP request should have a client header, but they're
| typically not relevant to what you're trying to do.
|
| curl is like an HTTP request that specifies only the parts
| you care about. It's brief, and having access to bash makes
| it easy to express something like "set this field to a
| timestamp".
| incorrecthorse wrote:
| HTTP requests don't need to have any specific headers, and,
| if anything, curl will only add ones for you.
| ricardo81 wrote:
| But specific HTTP requests might. Like cookies, an accept
| header, or anything.
| everforward wrote:
| > if anything, curl will only add ones for you.
|
| That's kind of what I mean. E.g. I believe curl will add
| a Content-Length header, which is good to have, but I
| don't need every example HTTP call to show me that.
|
| To me a curl call is kind of shorthand for "these are the
| parts unique to this request, do the appropriate thing
| for general-use headers". If I see a raw HTTP request
| missing a Content-Length header (assuming it could use
| one), I don't know whether to assume that I do the normal
| thing, or whether the server ignores Content-Length, or
| perhaps if the server specifically errors out when it's
| set.
|
| Vice-versa, if a raw HTTP request does have a Content-
| Length header, I'm not sure if that means it's required
| or just supported.
|
| If I see a curl call specifying Content-Length, it sets
| off the "something weird is going on" bells in my head.
| Nobody specifies that in curl, so it's presence is odd
| and worth looking at.
| 1vuio0pswjnm7 wrote:
| Actually "Copy as cURL" adds much that is not required. In
| some cases this can be useful. However if all one cares
| about is what is actually needed to succesfully request a
| resource using HTTP, then "Copy as cURL" always includes
| too much. It includes "things that dont matter".
|
| HTTP is more flexible than "Copy as cURL". There are things
| that can be done with HTTP that cannot be done with cURL.
| elaus wrote:
| curlconverter.com looks amazing, instant bookmark - thanks!
|
| I also use the browser's 'copy as curl' function quite
| frequently as it's so convenient, having all auth and encoding
| headers set to _definitely_ work (instead of messing around
| with handmade, multi-line curl command lines)
| vdfs wrote:
| Be aware that online service like this one might log your
| request which could have sensitive data, I'm not saying it
| does, but those websites give me the creep
| judge2020 wrote:
| I agree that it's possible, and that the majority of
| utility websites do use a backend to provide their utility,
| but it seems curlconverter.com doesn't make any requests to
| a website to convert and instead does so in javascript.
|
| It would be nice if more sites offered themselves as PWAs
| that worked when you set "throttling" to "offline" in the
| dev menu, so that you could ensure that no data is leaving
| the browser when working with sensitive info.
| adolph wrote:
| Maybe that would be a nice browser plugin. Something that
| blocks any further requests. I guess it would work
| similarly to ad blockers, only once enabled blocks
| everything.
| dotancohen wrote:
| Just unplug the cord, or disable the WiFi, for a few
| seconds. As we are presumably discussing sensitive data,
| nothing is as certain as the end of the cord in your
| palm.
| judge2020 wrote:
| Although an attack vector is it saving all that info in a
| buffer, continuously trying to beam it up until it gets a
| success response from the server.
| diegoperini wrote:
| True. Luckily it's open source. You can do `npx
| curlconverter --language go example.com` behind a firewall
| after downloading the npm module.
| spdustin wrote:
| I watched the network logs, and it doesn't seem to transmit
| anything. Additionally, their privacy info clearly states:
|
| > We do not transmit or record the curl commands you enter
| or what they're converted to. This is a static website
| (hosted on GitHub Pages) and the conversion happens
| entirely in your browser using JavaScript.
| lp0_on_fire wrote:
| Privacy policies on most websites mean jack when they can
| be changed at any time for any reason, imo.
|
| Not to mention the pattern nowadays is: offer a free
| service, pay a little lip service to privacy concerns,
| then the enshittification train comes rolling down the
| tracks a few years/ months later.
|
| Not saying this site is going to go down that path but
| IMO giving the benefit of the doubt with regard to
| privacy on the internet is bad practice for 2024.
| cloudwalk9 wrote:
| I kinda wish the address bar in any browser had an "advanced"
| popout menu that's basically a curl frontend with all of its
| bells and whistles. Basically move that functionality from
| the dev tools.
| elaus wrote:
| Sadly most major browsers go the opposite direction,
| removing more and more "advanced" functionality and
| information from the address bar (e.g. stripping the
| protocol, 'https://', from the URL)
| LtdJorge wrote:
| Yeah, I hate that. Firefox also removed http when editing
| in the address bar
| tracker1 wrote:
| Yeah, that has made my life so much easier when troubleshooting
| an API endpoint. I can tweak the request params to run against
| a local instance as well as pipe through jq for formatting etc.
| npteljes wrote:
| Postman can also import in "curl format", so yeah, the
| representation works.
| gullywhumper wrote:
| For R users, the httr2 package (a recent rewrite of httr) has a
| function that translates the copy as curl command to a request:
|
| https://httr2.r-lib.org/reference/curl_translate.html
| ekianjo wrote:
| oh WOW! this is super useful. Thanks for the pointer!
| appplication wrote:
| This is such an interesting and true observation. Anytime
| something isn't working with an endpoint, first step is "can
| you get it to work with curl".
| ricardo81 wrote:
| I've used 'copy as curl' a bunch. Often find I have to append
| --compressed so the command line will provide the uncompressed
| output.
| rpozarickij wrote:
| In curlconverter.com clicking on "C" redirects you to the
| --libcurl option documentation page instead of generating a C
| snippet. Wouldn't a more user-friendly way be to still generate
| a C snippet, but to mention that it can be done with the
| --libcurl option too?
| xnx wrote:
| Great Chrome feature! For those who haven't seen it, it also
| includes copy as PowerShell, fetch, and Node.js commands.
| spdustin wrote:
| https://curlconverter.com/ is a great example of intelligent
| UX. Whatever browser you're using is shown in the instructions
| for "Copy as cURL". Very clever.
|
| See:
| https://github.com/curlconverter/curlconverter.github.io/blo...
| oaiey wrote:
| Not sure if I like patching more and more parameters into the
| executable for gimmicks like this. Would be also work when you
| exchange the executable name instead of addding a parameter. Like
| ... curl-as-librucl https://www.example.com
| eichin wrote:
| > Added in 7.16.1. > Jan 29 2007
|
| so this isn't a _new_ trend...
| rafaelmn wrote:
| code generator that isn't LLM based - how quaint :)
| fullspectrumdev wrote:
| Tried this out, it doesn't seem to work at all with a good few
| flags - notably --upload-file
| Too wrote:
| As if the world needed more unsafe C code connected to the
| internet.
|
| Rust-evangelism aside, I guess one can run the program under
| ltrace to achieve almost the same result.
| ricardo81 wrote:
| libcurl is used in billions of situations.
|
| Fair point about memory allocations in C, but often alt
| languages rely on other people's code which you'd implicitly
| trust to do the same thing. So then it becomes an argument of
| testing and trust. All the same, you trust strangers code or
| you write your own.
| ranger_danger wrote:
| and often "your own" turns out much buggier.
| ricardo81 wrote:
| turtles all the way down, just pick your first turtle.
| EasyMark wrote:
| I suspect OP has already made up their mind on using C at all
| :)
| ricardo81 wrote:
| :)
|
| Stabilisers for some, but definitely not others.
| avgcorrection wrote:
| > As if the world needed more unsafe C code connected to the
| internet.
|
| Assuming (as usual) that the code generation is solid because
| of curl's reputation: why not trust it? It would be pretty bad
| if the generator could emit memory-unsafe code. (I don't know.)
| eichin wrote:
| For a trivial example, the code just calls curl_easy_init, a
| bunch of curl_easy_setopt, curl_easy_perform to do the work,
| and curl_easy_cleanup. (It leaves comments like
| "CURLOPT_WRITEDATA set to a objectpointer" in a comment block
| on params for which "You may select to either not use them or
| implement them yourself" - that's presumably where you are
| going to write your own unsafe code :-)
| avgcorrection wrote:
| Ah I see, thanks.
| pimlottc wrote:
| The submission title should be two hyphens (--), not an em-dash
| and a hyphen (---)
| altairprime wrote:
| Emailing the mods about this would enable them to see your
| comment and fix it; use the Contact link in the footer.
| pimlottc wrote:
| Reported, thanks!
| chfritz wrote:
| It says everything about C when you need a code generator for
| something like this. This is why development in python or node.js
| is so much faster.
| pimlottc wrote:
| Someone should buy libcurl.com and make it return the source code
| to generate a request to itself. Bonus points for setting the
| same headers and options as the triggering request.
| LatticeAnimal wrote:
| Sadly, it looks like godaddy is charging $5000 for the domain.
| Otherwise that would be a fun project.
| jicea wrote:
| Shameless promotion: Hurl [1] is an Open Source cli using libcurl
| to run to test HTTP requests with plain text!
|
| We use libcurl for the super reliability and top features (HTTP/3
| for instance) and we've added little features like:
|
| - requests chaining,
|
| - capturing and passing data from a response to another request,
|
| - response tests (JSONPath, XPath, etc...)
|
| There is nice syntax sugar for requesting REST/SOAP/GraphQL APIs
| but, at the core, it's just libcurl! Using verbose option, you
| can grep the curl commands for instance. (I'm one of the
| maintainers)
|
| [1]: https://hurl.dev
| cassepipe wrote:
| Used it and really liked it
|
| Thanks for your work
| fweimer wrote:
| I think I saw this first with the ASM bytecode toolkit:
| https://asm.ow2.io/javadoc/org/objectweb/asm/util/ASMifier.h...
|
| Possibly via the Eclipse extension:
| https://marketplace.eclipse.org/content/bytecode-outline
| undecisive wrote:
| To compile it you'll need to tell it to link to libcurl, e.g.
| with -lcurl on gcc: curl https://ifconfig.me
| --libcurl ip_fetcher.c # Output: your ip address, and a
| file ip_fetcher.c gcc -o ip_fetcher ip_fetcher.c
| -lcurl # Output: no errors, just a file ip_fetcher
| ./ip_fetcher # Output: your ip address
|
| (I'm sure most people are saying "no duh" right now, but I'm
| probably not the only one on here who doesn't write C code every
| day!)
| chungy wrote:
| It might be even easier this way: $ curl
| https://ifconfig.me --libcurl ip_fetcher.c $ make
| LDFLAGS=-lcurl ip_fetcher $ ./ip_fetcher
|
| (I just find make a heck of a lot easier than running direct cc
| commands)
| theanonymousone wrote:
| Is there any libcurl wrapping for Java?
| junon wrote:
| Of course there is.
| EasyMark wrote:
| dang.... i wish I knew that when I was working on my own little
| hacker news filter program just to practice some old school c >:(
___________________________________________________________________
(page generated 2024-01-29 23:01 UTC)