[HN Gopher] Show HN: HackerNews-new-jobs - insights into fresh a...
___________________________________________________________________
Show HN: HackerNews-new-jobs - insights into fresh and recurring
job ads
The website is meant for people that look for jobs on HackerNews
"Who's Hiring" threads and want to focus more on fresh ads and
companies, or to quickly look up ad history of any company. Github
repository: https://github.com/nemanjam/hn-new-jobs Demo website:
https://hackernews-new-jobs.arm1.nemanjamitic.com I used Algolia
API as a data source, along with scheduled task that parses new
threads few times at the beginning of each month. The extracted
data is then stored in SQLite database for fast querying, and the
results are cached with Keyv for faster page responses. I will see
in the future how much traffic the website receives and if this
stack is performant enough. For the website I used Next.js app with
default ShadcnUI components and charts. I just wanted a quick
functional prototype to test how much public interest is there for
an app with functionality like this. If you are interested in more
implementation details you can find them in the Readme file on
Github. The project is free and open source. Feel free to use,
self-host, fork and modify, and contribute. I would love to hear
your impressions and suggestions and look forward to discussing
features and technical details.
Author : nemanja_codes
Score : 106 points
Date : 2024-12-10 05:04 UTC (17 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| OccamsMirror wrote:
| Location / remote / hybrid would be interesting to visualize.
| nemanja_codes wrote:
| True, I thought about it, technologies too, and other filters.
| Currently website has clear focus new/repeated job ads. I will
| see how much public interest is there for the website and if it
| is worth adding more features.
| xnx wrote:
| I don't know who's to thank for it, but someone set up a
| constantly updated ClickHouse database of all data from the
| Hacker News API at
| https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...
|
| It gets you 80% of the way there on any HN data project.
| nemanja_codes wrote:
| This very is useful, thank you. Algolia API is also pretty
| generous, it made my work a lot easier. It returns even 1k
| comments in a single response. I started with low level Axios
| parser but I soon discovered that HackerNews webserver has very
| strict rate limiting. Good to know that there are already
| available data sources so you can focus on querying logic and
| data presentation.
| adhamsalama wrote:
| This is very useful, thank you!
| jcuenod wrote:
| This looks lovely! Good work.
|
| On my wishlist are some fuzzier categories: 1.
| Tech trends (rust, docker, postgres...) 2. Role trends ("ml
| engineer", "full-stack developer" ...)
| nemanja_codes wrote:
| Thank you, I highly appreciate it. As is website has clear and
| strong focus, it's "fresh" vs "repeated" job ads. Naturally,
| you would always prefer to give a shot with company that just
| posted a fresh ad, then to apply for a role that is repeated in
| 10 months of previous 12 but company still hasn't found a
| candidate that satisfies criteria.
|
| For the additional filters, by technologies and role types it
| would be of great help if I could find some high level indexing
| and fuzzy search tools/libraries. I would probably need to
| migrate from SQLite to Postgres and when I am already there
| probably use ORM too.
|
| Certainly, I would need to do serious research, if there is
| enough public interest in the current website and I manage to
| find some contributors with data science and information
| retrieval experience maybe we can add many interesting filters
| like tech, roles, location, visas, remote, etc.
| shermantanktop wrote:
| Me: hmm what's the big dip in the middle?
|
| Spouse: uh, COVID?
|
| Me: ohhhh
| nemanja_codes wrote:
| "HN Who's hiring" thread is just a random sample that pretty
| accurately describes trends in the entire industry.
| andersa wrote:
| This is a rather depressing graph, what happened in 2022-2024?
| nemanja_codes wrote:
| Collapse, unfortunately.
| robbie-c wrote:
| end of ZIRP
| game_the0ry wrote:
| This and over-hiring.
| myth_drannon wrote:
| And to put a more depressing angle to the graph, HN and Who's
| hiring were not that popular pre-2017, and now 2024 numbers are
| lower than the pre-2017 data
| tredre3 wrote:
| > HN [...] were not that popular pre-2017
|
| Is that true? What happened to change that in 2017?
|
| I've been around longer than that and I admit I haven't seen
| that much of a difference in engagement around here. But over
| such a long period it could easily have gone unnoticed by me!
|
| The only thing I've noticed is the increase in activity
| during the recent Reddit shenanigans (which resulted in a
| drop of quality of the conversation on HN, but it seems back
| to normal now).
| lubujackson wrote:
| The tax change that causes companies to have to weirdly treat
| developer salaries as some sort of asset such that they can
| only write off 20% of it per year. Outcome of this is killer
| for startups and causing a huge issue everywhere.
|
| Bottom line: If a company makes $1MM in revenue and pays $1MM
| in salary, they owe taxes on $800k profit. Yes, this is
| actually the law now.
| seeking_re_msw wrote:
| do you have a link to the tax change?
|
| surely tax changes come in the form of
| congressional+presidential bills and amendments
| freeone3000 wrote:
| Tax Cuts and Jobs Act of 2017 moved domestic R&E
| expenditures (including salaries) under IRS Sec 174 from a
| same-year credit to a five-year amortized expense (similar
| to capital expenditures). It also amended 174(c)(3) to
| ensure that software dev is unequivocally an R&E
| expense[^1].
|
| 1: https://irc.bloombergtax.com/public/uscode/doc/irc/secti
| on_1...
| andersa wrote:
| What the hell? This seems almost purposefully written to
| destroy the one industry where ordinary people could get
| good salaries. It doesn't even make any sense whatsoever,
| just a straight "fuck you" regulation?
| jvanderbot wrote:
| It almost surely was either a slap at large tech
| companies or was meant to generate short term revenue to
| cover some other cut.
|
| Same thing happened when they restructured tax code to
| interpret withholdings differently. Everyone saw more on
| their paycheck temporarily (and they gave speeches about
| it!) but owed more later if they didn't change their
| withholdings.
| freeone3000 wrote:
| Note: these changes were signed into law in 2017 but came
| into effect in 2022
| fzzzy wrote:
| https://www.resourcefulfinancepro.com/news/irs-
| section-174-c...
| dragonwriter wrote:
| > The tax change that causes companies to have to weirdly
| treat developer salaries as some sort of asset
|
| The salaries are not an asset, they are the cost of creating
| the asset. They are capitalizable similar to (but on a
| different schedule than) costs of acquiring software that is
| not developed in house.
| awongh wrote:
| Has anyone heard opinions from the incoming administration on
| getting this changed?
|
| I guess the one area this tax law particularly affects are
| bootstrapped revenue-generating (non-VC funded) startups with
| high dev costs? i.e., actual running businesses not playing
| with monopoly money.... which maybe Elon doesn't care
| about....
| _djo_ wrote:
| The change was introduced in Trump's tax cuts, so I doubt
| he will reverse it.
| jrs235 wrote:
| Did the change take effect during his administration or
| after the tax cuts expired or rather after the income
| brackets also increased? That change was intentional so
| that the following administration if it wasn't the GOP
| would feel the heat of and get the blame fortaxes going
| up.
| ryandrake wrote:
| The "Section 174" tax change always gets brought up, but
| wouldn't that be kind of a "tail wagging the dog"
| explanation? Surely, if it's worth it for a business to hire
| talent, then it's still worth it regardless of some esoteric
| tax rule. Are there actually companies sitting there in their
| Hiring meetings saying "Gee, we really need to expand our
| business and hire some engineers--if it wasn't for this tax
| law, we'd be hiring!"
| Jarwain wrote:
| If it becomes meaningfully more expensive to hire talent,
| it needs to be meaningfully more worth it to hire.
|
| It's more like if before the law was passed they'd hire 5
| new devs, now they can only afford 3 and have to make do
| until more revenue comes in
| sixhobbits wrote:
| Cashflow problems regularly kill business. Now you need a
| much higher cash buffer than before (in parent example
| where is that 800k coming from?). Combine with much tighter
| VC market and there are definitely many startups closing up
| shop because of this.
| neilv wrote:
| How do _non_ -developer salaries affect those taxes?
|
| (In 2024, if a company paid salaries for a software
| developer, a novel writer, and a cook, does each of those 3
| positions affect taxes in the same way?)
| jvanderbot wrote:
| No, just software devs. Thanks so called "Tax Cuts and
| Jobs" act.
| daemonologist wrote:
| They're unaffected - the law specifically applies to
| software development: (c) Special rules.
| (3) Software development. For purposes of this section, any
| amount paid or incurred in connection with the development
| of any software shall be treated as a research or
| experimental expenditure.
|
| (c.1 and c.2 are the opposite - carveouts for land
| acquisition and fossil fuel and mining exploration)
| seeking_re_msw wrote:
| Interest rates
| h1fra wrote:
| Nice to see the trend confirming my suspicion, the market has
| been complicated for the last year and it's not really improving
| (whether you are hiring or looking for a job)
| BillFranklin wrote:
| I found the same - market is actually getting worse (if HN is
| representative of the market). This month there were more job
| seekers on HN than jobs for the first time since 2014.
|
| https://bilbof.com/posts/tech-hiring-is-bad-right-now
| daemonologist wrote:
| This site I think counts only top-level comments, and the
| streams have been crossed since ~April 2023:
| https://www.hnhiringtrends.com/ . The overall trend is the
| same though - the market seems pretty historically bad.
|
| There's also Indeed postings data, which unfortunately only
| goes back to 2020 but is similarly bleak:
| https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE
| welder wrote:
| This is awesome! But it's missing at least one company from
| 2024-08. Maybe a parsing error? Or does it exclude hiring
| freelance/contract posts?
| nemanja_codes wrote:
| Thank you very much, I am glad you like it.
|
| I don't understand what is missing in 2024-08? You can link
| month by slug, I see nothing unusual here:
|
| https://hackernews-new-jobs.arm1.nemanjamitic.com/2024-08
|
| But as a side note, yes, this is not meant as an exact analytic
| tool, rather just a best effort website that gives some
| interesting insights.
|
| To clarify, only the "Who's hiring" thread is parsed, you can
| see it clearly in this constants file:
|
| https://github.com/nemanjam/hn-new-jobs/blob/main/constants/...
|
| Also, in there you can see how simple the parsing regex is, it
| just looks for "|" separator in the comment title.
|
| Another thing I noticed, some companies used different letter
| casing for their name in some comments, and the company name is
| part of the primary key, so same company is perceived as
| different, I should probably handle this better.
|
| https://github.com/nemanjam/hn-new-jobs/blob/main/modules/da...
|
| For example you can search for "ConsenSys" on the Search page:
|
| https://hackernews-new-jobs.arm1.nemanjamitic.com/search?com...
| welder wrote:
| I didn't realize freelancer jobs go to a different HN post
| `Ask HN: Freelancer? Seeking freelancer?` Makes sense now,
| because the job I noticed missing was a freelance post.
| nemanja_codes wrote:
| You can try searching directly on Algolia, my website uses
| it as a data source:
|
| https://hn.algolia.com/
|
| https://hn.algolia.com/api
|
| If it's there my website has it also. But Algolia does
| pretty good job parsing HackerNews, I am pretty confident
| 99.9% of comments are included.
| mdaniel wrote:
| It would appear you have a non-trivial "company" parse problem,
| based on eyeballing the "First Time Companies" section. I do
| appreciate the problem of trying to do semantic analysis on HN
| comment text, but several of them follow the "X@Y" nomenclature
| and a few others are "X-Y" so it doesn't seem to be
| insurmountable to try and fix
| nemanja_codes wrote:
| This is just a first iteration of a quick prototype to gauge
| public interest in the idea. It just looks for the "|"
| separator in the comment title, which is something that
| majority of posters enforce.
|
| https://github.com/nemanjam/hn-new-jobs/blob/main/constants/...
|
| If people show constant interest I can iterate it further,
| enhance it, ad features, etc.
| mdaniel wrote:
| Well, if that's your attitude then I wish you all the luck
| with your project
|
| I was also going to say $(git commit -a) is evil based on
| <https://github.com/nemanjam/hn-new-
| jobs/blob/main/data/datab...> but it seems that you just
| _want_ an always changing binary blob to make your git repo
| grow without bound :-( https://github.com/nemanjam/hn-new-
| jobs/blob/main/.gitignore...
| thih9 wrote:
| Possible bug report, I noticed a company that posts regularly but
| I didn't see it listed:
| https://news.ycombinator.com/threads?id=spark_CM
| nemanja_codes wrote:
| Here it is:
|
| https://hackernews-new-jobs.arm1.nemanjamitic.com/search?com...
| thih9 wrote:
| Something still looks off, the earliest ad is from 2024-07,
| but the company posted more recent ads too.
|
| I guess this is because they changed their ad's headline
| from: ChartMogul | Remote (EU) | Full-time
|
| to: ChartMogul (https://chartmogul.com )|
| Remote | Full-time
|
| Other companies are similarly affected, e.g. Medusa:
|
| - listed: https://hackernews-new-
| jobs.arm1.nemanjamitic.com/search?com...
|
| - earliest ad listed:
| https://news.ycombinator.com/item?id=34222858
|
| - more recent ad on hn:
| https://news.ycombinator.com/item?id=42315828
|
| Perhaps it's the ")|" bit causing problems with some regexps.
| nemanja_codes wrote:
| Yes, there is a lot of room to improve parsing.
| ryandrake wrote:
| This is great! I had long ago started a similar project to parse
| HN job ads month after month looking for the ones that just stay
| the same each month, which might suggest the company is probably
| "fishing" instead of "hiring". But it never really got off the
| ground. This platform seems more than capable of doing that kind
| of analysis and more. Great work.
| nemanja_codes wrote:
| Thank you very much, very much appreciated. I was curious to
| see how much people would be interested in such website.
| ofcrpls wrote:
| Nice work for Central Defender /s.
| nemanja_codes wrote:
| Thank you very much. He is Chelsea legend that messes up my
| SEO.
___________________________________________________________________
(page generated 2024-12-10 23:00 UTC)