[HN Gopher] Ask HN: Can web scraping be the basis of a viable bu...
___________________________________________________________________
Ask HN: Can web scraping be the basis of a viable business model?
I'm a data engineer at heart, and I never did or enjoyed front-end
work. Having said that I always was happy to code and evolve
crawlers and web scrapers. Now I've taken some time off from work
and gigs and I'm working on a side-project I've been hacking for
some time. Without getting into the details yet: it aims to make
web data collection a little bit easier for non-devs. I'll soon
have an MVP and will start pitching to investors: aiming for an
open-source business model (after a few months of stealth
development) and eventually a typical SaaS offering for extra
functionality. At this point I'm trying to consolidate and counter
the steel-man counter-arguments I should expect from investors. The
most obvious one: as one can imagine, the product it's not magic
and, after a certain point it does require some manual work from
the customer, hence this is an aspect I should prepare for. I have
done some preliminary analysis of the space of potential
competitors (think import.io, Apify, Zyte/ScarpingHub, etc.) and
described opportunities for differentiation. What I'm afraid of is
getting sidetracked in a discussion of "um, this is web scraping
and it's hard to make a business on top of it". I understand that
there's not much context now and one could easily say "well yeah,
anything could be possible with a good team, product...", but I'm
reaching out to the HN community to gather some considerations,
mental models and pointers, I may not think of myself at this
point.
Author : fforflo
Score : 107 points
Date : 2021-12-18 15:25 UTC (7 hours ago)
| kccqzy wrote:
| Of course. See Yodlee and Plaid.
| tixocloud wrote:
| There are many data providers serving the financial
| services/capital markets sector that provide everything from raw
| data feeds to insights based off of web-scraped data. What will
| be important for you is to identify a good niche, understand
| which part of the data value chain your customer needs, and
| deliver the data in a way that fits their workflow.
|
| The value is in the decisions that can be made based on the data
| being sold rather than the method at which you extract it from.
| If you focus on the value of the data and who needs it, you'll
| likely find a viable business model.
| shtopointo wrote:
| And to answer your question, because my other post was a question
| itself:
|
| > after a certain point it does require some manual work from the
| customer
|
| Once you gain traction, you can become a platform, intermediary
| between customers and engineers that will fine tune scraping to
| what the customer needs. This could either be some sort of
| "Solution Engineer" that the company hires, or open it up to
| outside developers that get paid per integration (either by you
| or by the customer, or both). There's a solution to every
| problem.
|
| As far as the business itself, I think you could be on to
| something. Of course, ideas are cheap and it's the execution that
| counts, but here's how I'd think about it: with scraping, every
| website on the web has an API. Before, only 0.1% of websites had
| an API.
|
| And certainly wouldn't hurt to change the "scraping" word - such
| an ugly word.
| thruway516 wrote:
| Never mind business, a web-scraping command-line utility as
| comprehensive and easy to use as say curl would be something. I
| would even pay for that.
| adinosaur123 wrote:
| I don't wish to hijack this thread, but I've been pondering a
| similar question. I've been working on a product that requires a
| very large amount of data that, as far as I can tell, can only be
| gathered by scraping (real estate data - even data vendors like
| estated.com don't have stuff like sales data).
|
| Many, many websites contain legal language that forbids automatic
| data collection/scraping. How can a business be built in such a
| case?
|
| Perhaps OPs tool only scrapes a select few sites that _don 't_
| prohibit scraping, but that seems like the exception, not the
| norm.
| GhettoComputers wrote:
| Do it manually. I wonder what automatically means legally or
| scraping. It's pretty hard to enforce those requirements,
| because I assume it's being broken by search providers.
|
| If it's a derivative work like copilot, I wonder if there's a
| legal case to say you can't do it. I assume you're doing
| something like an RSS feed for pricing suggestions with
| commissions? I just looked this up and it seems like it's legal
| to do so but their information is copyrighted.
| https://law.stackexchange.com/questions/15556/is-scraping-re...
| nlh wrote:
| Read up on LinkedIn vs. HiQ. As long as that ruling holds (and
| it might not), the tl;dr is: If it's on the open web, you can
| scrape it. You might be violating some Terms of Service (that
| you never agreed to), but you're not violating (US) law.
|
| If it's NOT on the public web - e.g. it's behind a login, then
| you can be sued, as you'll have had to explicitly agree to
| Terms of Service during your account creation and you'll then
| be in explicit violation of that ToS.
| gwbas1c wrote:
| Are you falling in love with the code?
|
| I think you'll have a lot more luck finding 2-3 initial customers
| before you try to raise money. It's always easier to explain what
| your product does, and who the target market is, in terms of
| actual customers; instead of hypotheticals.
|
| Remember, the goal is to build a business. If you fall in love
| with the code, it's too easy to build something you enjoy working
| on, but has limited commercial value.
| fforflo wrote:
| Am I falling in love with the code? I'd say we're on the 2nd
| date .... I may even get lucky. But it's enough to make me hang
| up on recruiters (respectfully).
|
| Good point though. Bootstrapping is a viable option too, but
| fundraising has certain no non-financial benefits that can be
| appealing.
| rank0 wrote:
| Some major issues from my experience web scraping:
|
| 1. Changes in data structures. If some site randomly decides to
| alter the format of their json/xml objects for their frontend api
| it may brake your scraper and anything that relies on that
| scraper's output.
|
| 2. Security controls like rate limiting, captcha, ip
| blacklisting, auth systems.
|
| 3. Html which is rendered via complicated client side JavaScript
| blobs or web sockets. You'll need a Headless browser engine like
| selenium and some site-specific parsing logic.
|
| 4. Legal issues.
| sonofhans wrote:
| I worked at AboutUs.org for a while, and that's what we did. Good
| news: it was fun and rewarding. In many ways it felt like a
| satisfying old-skool problem: scrape, find edge case, patch it,
| scrape again. We were scraping 100 million domains once a week
| with a team of six engineers, one UX (me), and Ward Fucking
| Cunningham as wiki expert. Ward in particular was great at
| prototyping solutions.
|
| It is an arms race, since many people don't want you to scrape.
| We tried hard to respect robots.txt, but we still got angry
| cease-and-desist emails from people who'd malformed or
| misconfigured the file.
|
| You will have a scale problem: it's a lot of data. You'll have
| parsing problems: live HTML is about the dirtiest data set I've
| ever seen. Refresh rate can be a major competitive advantage: how
| often can you scrape, store, diff, and report? These days you'll
| need first-class JavaScript execution to catch dynamic content.
|
| But the biggest problem isn't the scraping tech, it's the use
| case -- what uses cases are you going to afford your early users?
| You don't mention this in your post, and it will non-trivially
| affect what you scrape and how you report it. I'd encourage you
| to find users who have business problems that can be solved by
| paying money for scraping. Otherwise you'll be another
| interesting open source tool that no one's figured out how to
| monetize. Do this _before_ you talk to investors or take their
| money.
| eatonphil wrote:
| People make their own blogs and sites with React and other
| JavaScript frameworks all the time now... It won't even be
| actually dynamic content before you need to be able to execute
| JavaScript.
| max002 wrote:
| From what i know thats how skyscanner started, not sure if ita
| very popular in US, but it is in Europe. Now they're
| paying/having deqls for data, but they started with scraping the
| hell out of airline sites.
| ethor wrote:
| In my opinion you should start with defining your value
| proposition and target market, i.e. how do I create value for
| customers, and who are my customers?
| martini333 wrote:
| Yes.
| test0account wrote:
| Is Google not just one massive web scraper?
| edmundsauto wrote:
| Yes, although I would encourage you to think about something
| higher on the value chain than raw data feeds. Those exist and
| have become an increasingly difficult market to compete in. You
| can buy a custom feed for like $250/mo.
|
| Instead, think about what people want to do with the data. For
| example, if you are going to scrape diamond prices, don't try to
| sell that feed. Set up a website with a UI so people can research
| diamond prices, and get alerts when specific thresholds are met
| or items come in stock. Monetize with ads.
| stef25 wrote:
| Like camelcamelcamel.com I guess?
| bitshaker wrote:
| It is for openexchangerates.org
|
| Makes the founder a hefty sum.
| ChikkaChiChi wrote:
| Scraping services on their own are a viable business product, but
| the power to assign metadata and contextualization is where the
| unicorn lies.
|
| The service itself will always be in flux because of how freeform
| hypertext is as a schema. So many other comments here reflect
| that better than I could.
|
| The fact is that any chunk of data you're handing over to clients
| still needs to be handled by their team and in my experience
| reality often falls short of expectations. If you can somehow
| deliver them something cleaner (or even something that can help
| them reach conclusions faster), then you have a product with a
| high value prop.
| nojs wrote:
| I cofounded a company based on scraping academic publications. We
| ended up getting a lot of traffic (millions of pageviews per
| month) because we had good SEO, but it's not necessarily a
| defensible business model by itself.
|
| You'll likely have to do more manual data cleaning than you
| expect, and get some amount of pushback from the sources you're
| crawling (depending how commercially valuable the data is).
| bjourne wrote:
| Web scraping is a legal gray area in many or most jurisdictions.
| In some jurisdictions, depending on the tos of the web site
| itself, scraping it might be illegal. In others republishing the
| scraped information in any form might be illegal. In others still
| you might not be allowed to use the scraped data for any
| commercial purpose.
|
| "But what about Google?" Google is worth 100 billion dollars and
| can play by completely different rules than scraping startups.
| stef25 wrote:
| Google also plays nice. I manage a few sites that get hammered
| by scrapers to the extent that it causes big spikes in CPU,
| something that doesn't happen with G.
| that_guy_iain wrote:
| So many people don't even think about number of requests and
| often want the job to end as quickly as possible. For me,
| that is thinking short term, I've seen requests per second as
| low as 4 cause performance issues for a company. If you want
| to scrape long term you need to think requests per minute.
| That is unless you're dealing with a mega site then it's
| keeping a low request per second.
| tempestn wrote:
| Yeah, I was going to say something like this. And one step
| better than Google, but still challenging, there are also a lot
| of sites out there that prohibit scraping but make an exception
| for "general purpose search engines". So if you're providing a
| specialized scraper that's used for specific purposes, you'd
| likely run into problems there too.
|
| As well as the actual legal challenges, you'll also have the
| perception thereof, which could make businesses wary of relying
| on your services.
| sjg007 wrote:
| It is valuable. There's a lot of Robotic Process Automation,
| competitive analysis etc...
|
| >the product it's not magic and, after a certain point it does
| require some manual work from the customer, hence this is an
| aspect I should prepare for.
|
| Can you make it magic or maybe develop end to end solutions for
| your first "customers" using your product? That sounds like the
| schlep you need to do.
|
| Sounds promising! Find yourself a customer or two!
|
| If you really want to go the open source route, just focus on
| that and then see if people pick it up and use it. Then you'd
| offer the SaaS.
| authed wrote:
| Scraping user content or recipes should be fine.
| itake wrote:
| You might want to look at web scraping for data scientists. I am
| trying to build a ML Model for NSFW text detection in multiple
| languages and I am not looking forward to scraping p*rn and
| youtube websites for comments.
| stef25 wrote:
| You probably need NSFW text which is its own niche and should
| be straight forward to collect?
| exdsq wrote:
| Could you expand on this a little? What's the problem and what
| are you looking for as a solution?
| markl42 wrote:
| Maybe only kinda sorta relevant, but I interned at RefME for a
| summer - basically a Zotero competitor. There was a lot of value
| (for users) in scraping web pages to autofill to author, title,
| date etc so it could generate the references, something I started
| to work on (but then they got bought out so not sure what ended
| up happening)
|
| Any I could've imagining this being a paid pro feature down the
| line
| JoeAltmaier wrote:
| Did it back in the old days, scraping stock quotes to build a
| database for display by our Java app and web services. Called
| NetProphet, it would do a score of trend lines etc as overlays.
|
| I wrote the scraping code. Had a list of sites and macros for
| extracting quotes, updated every day to every customer. If one
| quit working (the site attempted to prevent scraping) the app
| would use another and give a notice back to me. I'd tweak the
| macro for that site, and we'd be back scraping it the next day.
|
| We eventually hired a finance student (Josh Hatwich, now a fellow
| at Adobe) to parse a Comstock satellite feed we put on the roof.
| That ended the era of scraping at StockPoint.
| gtsop wrote:
| Point them to the disgusting but successful pracrices of
| ClearviewAI. That should do it.
| launchiterate wrote:
| Yes if you point it at a well defined target market and create a
| solution.
| mukundesh wrote:
| Not sure if this company survived the pandemic but check out
| Applaudience it was crawling seat level data from event websites.
|
| https://www.screendaily.com/features/how-uk-data-company-app...
|
| Applaudience's algorithms trawl through every exhibitor website,
| looking at every showtime of every film, and tracks the
| auditorium layout as each seat flips from available (unsold) to
| unavailable (sold).
| wombatpm wrote:
| If your potential customers are willing to pay to scrape data,
| why aren't they will to pay for the data from the source
| directly? Is it not available or is it considered exclusive or
| proprietary. I'm thinking about the lawsuits around deep linking
| and TicketMaster. Web scraping at scale is a never ending arms
| race because designs evolve or the host is actively trying to
| thwart you.
| fforflo wrote:
| I like these two questions! They can crystallise things.
|
| Wasn't aware of the Ticketmaster v. Tickets.com case. Will have
| to print it along the SCOTUS LinkedIn v. hiQ ruling.
| max002 wrote:
| So for new companies the data (that a lot of time IS publicly
| available) cost a lot of money, i.e. Api access or charing per
| requests. A lot of time companies start with scraping and then
| once they have more customers (and data access price can be
| shared amongst them) they switch to paying for it.
| apienx wrote:
| Google, a trillion dollar company, is essentially the world's
| largest web scraper. So...yes! You'll almost certainly find a way
| to monetize that.
|
| Monopolies, lobbying and protectionism got in the way of keeping
| the web truly machine readable. There's tremendous value in
| restoring some of it.
| dbtc wrote:
| Web scraping is just one way to get the data to facilitate the
| sale and service of ads.
| fforflo wrote:
| > keeping the web truly machine readable. There's tremendous
| value in restoring some of it.
|
| :-)
| elliottinvent wrote:
| > Monopolies, lobbying and protectionism got in the way of
| keeping the web truly machine readable.
|
| Exactly and that ship has long since sailed. The good ship Web
| 3.0 (semantic web) launched in '99 and was a ghost ship until
| recently when it was boarded by crypto pirates now flying the
| web 3.0 flag.
|
| > There's tremendous value in restoring some of it.
|
| To this comment and OP, my startup is using web scraping to
| pre-populate machine-readable data for a DNS-based protocol
| called NUM [0]. So as others have said, whilst web scraping
| itself may be difficult to build into a viable business, it can
| be a key component of a viable business. Email in my profile if
| you want to discuss.
|
| 0. https://num.uk/blog/we-crawled-5m-uk-websites-and-
| published-...
| melony wrote:
| Web3 (without the decimal point) these days usually refer to
| internet Ponzi schemes.
| max002 wrote:
| My only advice would be have a backup plan (redundancy) . I.e.
| Design basic version that works and dont get blocked as bot, then
| design another one. This will save you from situation like
| described below where your original method stopped working, but
| your client wants data now (because yjey pay for it). And be
| nice, dont take data you dont need. Keep it easy on servers.
| deanebarker wrote:
| There was a company in my city that did something like this. They
| didn't survive.
|
| They crawled the data, but also had a services component to do
| something with the data. Like, they had contracts with pharma
| companies to search for indications that a page was selling
| counterfeit drugs.
|
| I'm not sure of the exact details of why they didn't make it.
|
| Also, I'm thinking about Recorded Future
| (https://www.recordedfuture.com). They do something like this --
| again, the mechanics of scraping, and a services component for
| analysis.
| bootcat wrote:
| I think yes, web scraping can give a viable product. Now with the
| ability to scrape react/js pages and availability of 24GB cloud
| free tier machines and transformer models - i think atleast for
| the next 5 years, should be possible !
| asimjalis wrote:
| Another example is Zillow which scrapes public records. Public
| records have some advantages.
| subpixel wrote:
| Pitch your investors the VPN business. That's what web scraping
| is at scale - a series of networking techniques that allow users
| to do what they want without being blocked.
| katzgrau wrote:
| Ultimately people want information from scraping, because to a
| lot of businesses, the information is what's valuable.
|
| Consider, apart from a tool for general purpose scraping, what
| information a specialized scraper might obtain for a valuable but
| underserved industry that can profit from the data.
|
| I have a buddy that scrapes data specifically for the
| tanker/shipping industry, for example.
|
| General purpose scraping will involve a lot of competition and a
| bit of an arms race. Niche scraping lets you fly a little below
| the radar.
| eruci wrote:
| Yes! Everybody scrapes everybody. The difference lies on what
| they do with the data afterwards.
| xedarius wrote:
| I went for an interview once at a hedge fund. There were a
| surprising amount of questions about web scraping. I very much
| got the feeling it was an active and ongoing problem. So yes I do
| think there's a business in there.
| fforflo wrote:
| Having recently worked for a hedge fund I noticed that too -
| although my idea predated that engagement . That's probably my
| first go-to market (if anyone has leads send me end email) .
|
| Hedge funds actually call this "alternative data".
| dreyfan wrote:
| If you worked for a hedge fund and you know they purchase
| scraped data from quite a number of vendors then why are you
| asking if it's a viable business model?
|
| Of course it is.
| lubujackson wrote:
| AFAIK Plaid does a fair bit of old school scraping behind the
| scenes - a lot of the "nice new web" is built on the backs of
| old kludgy websites doing the same scraping things people were
| doing 20 years ago.
|
| Finance/banks are especially... inconsistent, to say the least.
| wcerfgba wrote:
| You've already identified some of your competitors, you should go
| in to more detail and try to answer:
|
| * What features are common among my competitors?
|
| * What features are unique?
|
| * Who are target customers and users? Is there any overlap, or do
| some competitors target unique market segments?
|
| This last question ties in to a discussion I was having with a
| friend recently. In B2B sales, your customers are businesses, but
| your users are people in those businesses with certain roles and
| responsibilities. Understanding the difference is key, because
| you will often need to develop your sales and marketing
| strategies based on the business/customer profile, but your UX
| will depend on the needs of the users within those businesses.
|
| In my opinion you are more likely to be successful if you can get
| an initial foothold in a market by identifying a specific target
| of customers and users, solving their use case very well,
| developing a moat, and then growing out from that foothold to
| provide a wider set of options. Web scraping is just a tool. You
| need to find businesses who can gain value from scraping or from
| scraped data. Are there businesses who, for whatever reason,
| would not be able to adopt one of your competitors' products, or
| would find that adoption difficult? Maybe you could specialise in
| scraping a particular kind of data, or providing a full-stack
| solution for companies with limited in-house technical expertise
| (like some kind of consulting, you hop on a call with the client,
| they tell you what they want to scrape, and you set up a hosted
| solution which provides a SQL or Excel interface to the data).
|
| In short, successful product development is all about
| understanding customer and user pain and needs. If you can find
| pains or needs which are a common theme for a particular
| demographic of companies and roles, you can work with those
| people to understand their problems and make a product which is
| very valuable to them.
| fforflo wrote:
| Thanks a lot for your comment.
|
| You're exactly right on the tricky relationship between
| developers as internal ambassadors to businesses - customers. I
| think it somehow applies to almost any developer-facing tool.
|
| Your recommendation to focus on a specific vertical at first
| makes sense too. Helps prioritising the backlog as well.
| mrleinad wrote:
| An online law editorial I used to work at around 10 years ago
| basically scraped free available content from different sources
| around the web, repackaged it, and sold it online via
| subscription model. They're still around.
| erichocean wrote:
| Scraping isn't the problem, it's getting access to the pages _to_
| scrape at high-enough volume, at low-enough cost.
| tommoor wrote:
| ClearBit and Plaid haven't been mentioned yet - both examples of
| multi-million/billion dollar business built on the back of
| scraping.
|
| The more specialized you can get the better chances of success
| imo. Generic web scrapers are dime-a-dozen
| bdcravens wrote:
| Yes, but scraping is a small part of the overall puzzle. As
| developers, we overestimate how valuable tools are (as opposed to
| solutions). I think the better opportunity is not to be another
| scraping-as-a-service provider, but to niche down to a solution
| that uses your scraping technology.
| DonHopkins wrote:
| If you start yet another scraping-as-a-service provider, you're
| attempting to provide a paid service for people who just want
| to steal content for free. Not gonna work.
| that_guy_iain wrote:
| Scraping as a service can be profitable if you target people
| who are looking for leads to cold email/spam. Lead gen is one
| of the few areas you can easily charge $100 as your entry
| level package and at that price it's very easy to make money
| if you have half decent marketing.
| bdcravens wrote:
| There are legit use cases for browser automation, but a ton
| of bad actors too. I think your point illuminates something
| that many fail to fully consider: What are all the ways my
| service/software can be abused?
| hooande wrote:
| Web scraping can be a viable business. It depends on what you're
| scraping and who your customers are.
|
| Are there a couple thousand people who would pay for a SaaS
| offering? Then it's a business. The real goal would be
| identifying a hair on fire problem that you are in a unique
| position to solve. That's always the problem, and it has nothing
| to do with web scraping in particular.
| gkoberger wrote:
| Hey! Hopefully this comment comes across as helpful rather than
| hurtful. I'm also the founder of a developer tool, and it's hard
| to raise money for! (I know you said yours isn't strictly a
| developer tool, but you mentioned open source so I think it's
| fair to assume it falls in this category.)
|
| I think you're missing a step, which is where _you_ answer if web
| scraping can be a viable business model. You're attempting to
| convince VCs with logic (and a few assumptions), but there's an
| easier (or harder) way to do it... convince them by making money.
|
| Most VCs aren't ideologues, and don't have an opinion about
| business models. They will be convinced if you simply show them
| you're making money. It's not their job to decide if an idea can
| make money or not; that's your job as a founder.
|
| I applied to YC twice. The first time we spent the 10 minutes
| talking about if it could make money or not, and never got
| anywhere good. We got rejected. The second time we were making
| money, the conversation was smoother, and we got in. It's so much
| easier to be able to replace "I believe" with "our customers
| believe". It changes the conversation completely. You don't need
| to be making billions of dollars; just enough to show that people
| want what you're making!
|
| tl;dr You're trying to convince VCs when you need to be
| convincing customers!
|
| (For the record, a lot of what I said here is very money-driven
| and that's not how I build my company. However, in the context of
| VCs, which are purely financially driven, it's how you should be
| thinking about it.)
|
| Good luck, and let me know if I can help! My email is in my
| profile if you want to talk!
| fforflo wrote:
| You're not being hurtful. :) I'll send you an email too, but
| because others are raising this:
|
| Good point and it's a potential trap of tools built from
| developers for developers.
|
| There's this fine line between finding it useful and being
| willing to pay for it. Being honest to my self: Do I find this
| useful ? Yes Makes my web scraping easier? Yes Would I pay for
| it? Maybe... but the problem is I've never paid for a scraping
| tool either.
|
| But other people may pay for it. To get to that point though
| I'll probably need a few more hands on deck, who need salaries
| and for that you need external funds.
| gkoberger wrote:
| Great! Looking forward to talking :)
|
| If I were you, I'd do everything possible to answer your
| initial question (is web scraping a viable business) before
| bringing other people and their money into it. Everything
| gets much, much harder once other people's time and money is
| involved (both the process of convincing them, AND the
| potential process of realizing you were wrong).
|
| There's lots of ways to build a business, so take my advice
| with a grain of salt. But especially if you're worried about
| it, I'd highly recommend focusing on getting to a few paying
| customers before involving VCs or employees. If you're here
| asking that question, you can be sure they will be too (minus
| the initial conviction you have).
|
| (The one exception is a co-founder! It's lonely starting a
| company, so you should definitely consider trying to find a
| co-founder that you love working with.)
| shtopointo wrote:
| Isn't one big issue that any website that has data worth scraping
| has ToS that disallow scraping? e.g. craigslist?
|
| Would think this is a legal concern...
| colecut wrote:
| Can breaking terms of service result in anything more than
| being banned from the site you are scraping?
| sovnade wrote:
| Also it's very easy to get blacklisted if you scrape too often
| or too quickly. You definitely have to be "polite" about it.
| fifilura wrote:
| What about government/municipality data?
|
| We paid for that once, surely it shouldn't be necessary to
| charge for it again?
| dradicchi wrote:
| Please, see https://www.datallog.com/
|
| They are a profitable startup and have a SDK solution for scale
| web scraping bots on many business fields.
|
| I'm a advisor/investor in Datallog. You can connect with the
| founder in https://www.linkedin.com/in/joelder-maragno-arcaro/
| rdbell wrote:
| I co-founded https://packetstream.io
|
| There are a handful of companies doing very well with models
| similar to what you're describing. I can't mention specific
| customers, but I see some of them doing very large scraping
| volume through our network.
|
| It's an industry where having a good product is more important
| than the amount you're spending on marketing. If developers are
| happy with your product they'll take it with them to future
| companies/projects and share it with colleagues.
|
| It can be a cat-and-mouse development cycle where the sites you
| target break your functionality and you'll have customers that
| will want fixes to be implement ASAP because they rely on your
| tools to make money.
|
| I don't know what you're building exactly, but keep in mind
| there's a good chance that you'll need to commit to long-term,
| continuous, rapid development cycles if you want to retain
| customers.
|
| Best of luck!
| erichocean wrote:
| My dude, how do you actually use your service (loading URLs)?
|
| I just signed up, got sent to a dashboard...NO API
| DOCUMENATATION, just a link to download an app (!) for people
| who want to sell their residential bandwidth.
|
| See ScraperAPI for a company that does API documentation well
| in this space. (I've spent well over $100K with them.) Or
| Stripe. PUT THE CODE UP FRONT.
|
| There's also no API docs here, under "Support":
| https://packetstream.io/support/faq
|
| Update: I see it's hidden under "Network Access." This kind of
| thing should be OBVIOUS, not the sixth option (looks like all
| of the others) on some random SaaS dashboard.
| suifbwish wrote:
| I'm curious how you deal with JavaScript that will load other
| pages including other JavaScript documents that cannot be loaded
| until the first set of JavaScript is executed. I've played with
| the chromium web driver a few times but it seems to be tricky to
| implement in a completely headless environment.
| max002 wrote:
| Sometimes you need custom driver or taking control of real
| browser (and i mean real browser, not via webdriver), headless
| is easily detected.
| webdoodle wrote:
| I did so for years, scraping university press releases, obscure
| government data (like the Federal Register), jail/prison rosters,
| Reddit posts/comments/users etc. Companies like LexusNexus have
| been doing it for far longer than the digital world, offering it
| as a clipping service. If you can find a niche, it can be a
| regular subscription income stream.
| nomdep wrote:
| Instead of competing head-on with ScrapingHub/etc, you could ask
| yourself "why companies doing X pay them" and sell a specific
| product for that niche.
| arasx wrote:
| If you can apply the scraping to a use case and sell that use
| case you have a better chance at a viable business model.
| Examples come to my mind is; builtwith (scrape sites and publish
| the list of technologies they use), ahrefs (scrape sites and find
| outgoing/incoming links) etc.
| orev wrote:
| If you're scraping someone else's data, do you know what they
| copyright status is? Have you made deals with the original
| sources that permits you to use their data? How will you deal
| with lawsuits and the constant blocking of your scrapers?
| max002 wrote:
| Amd what if youre scraping data thats... already being scraped?
| :D
___________________________________________________________________
(page generated 2021-12-18 23:01 UTC)