Unsure of the technical term of what I'm trying to do - proxy
forwarding? Creating a home proxy server to redirect traffic to other
proxies
reddit / selfhosted

----------------------------------------------------------------------
sleeponcat · 2023-12-19 17:58

Issue: I'm building a webscraper using Selenium. Selenium can use
proxies natively, but cannot authenticate to proxies that require
auth^(1). I have paid proxies that fulfill my needs, but they do not
support whitelisting, only auth.

Plan: I want to set up a private server that I can whitelist myself
on. Selenium would output its traffic to that server using the built-
in proxy system. The server would catch the request and redirect them
to the paid authed proxies.

[https://i.imgur.com/ZiN4maV.png](https://i.imgur.com/ZiN4maV.png)

Is this proxy forwarding?

Is my idea feasible?

Would you recommend doing this on a home machine, or rent an external
server?

Thank you!

^(1): I don't know why it's like this. I considered forking Selenium
and adding this feature myself. But Selenium is a mature product, so I
figure there may be a good reason why it's not supported. There is a
library that adds the ability to connect with auth, Seleniumwire,
however, it is extremely performance heavy as it does a whole lot more
than base Selenium.

----------------------------------------------------------------------
clericc-- · 2023-12-19 19:03

Also unsure what the Name would be. I found this as a first attempt
towards a solution: Create a virtual network interface that sends
traffic towards a (socks) proxy. this might guide you to a viable
solution. https://superuser.com/questions/1339015/virtual-network-
adapter-that-forwards-request-to-a-socks-proxy#1342227

----------------------------------------------------------------------
adamshand · 2023-12-19 19:09

If I'm understanding correctly, what you are wanting is just a normal
web proxy.  Have a look at Squid.