[HN Gopher] We have successfully completed our migration to RAM-...
___________________________________________________________________
We have successfully completed our migration to RAM-only VPN
infrastructure
Author : dgavrilov
Score : 309 points
Date : 2023-09-20 07:38 UTC (15 hours ago)
(HTM) web link (mullvad.net)
(TXT) w3m dump (mullvad.net)
| mika69 wrote:
| Can somebody explain in more detail - what does this mean for the
| user? What are pros and cons?
| dangus wrote:
| There is no disk in the servers, so there is no chance for user
| information to persist anywhere.
|
| I also wouldn't be surprised if it's a performance benefit,
| since RAM is far faster than any permanent storage.
|
| The cons are probably just that this is a pretty unusual
| architecture that they probably had to put some work into
| setting up and making it reliable.
| Daviey wrote:
| It's essentially a PXE-boot diskless environment, what makes
| you think it is unusual and possibility of being unreliable?
| dangus wrote:
| I should say, unsuitable for certain use cases.
| johnklos wrote:
| I have servers which have literally been running non-stop
| for multiple years. Unless / until the kernel itself
| needs to be upgraded for security issues, and so long as
| backup power is good, they can run indefinitely.
| justapassenger wrote:
| Not a best setup to run database, yes. But stateless farm
| of servers that essentially forward the traffic for you?
| Not that much downsides.
| panick21_ wrote:
| What is unusual is the firmware that they have.
| Brian_K_White wrote:
| What important things do you store only in ram? Why not?
| Isn't it reliable?
|
| I think they just mean ephemeral.
| [deleted]
| thathndude wrote:
| Technically, researchers have proven that you can shutdown a
| machine, hit the RAM with a cold spray (like liquid nitrogen)
| and keep the bits "alive" long enough to dump them for
| analysis.
|
| But, obviously, that's pretty insane. Agree with everything
| that this is a big leap in the step of better protection for
| users.
| caeril wrote:
| There's a fairly easy physical mitigation for this.
|
| Once DIMMs are seated, secure the ends with superglue, then
| brush conformal coating over the bus traces.
|
| The second step is likely not even necessary if the
| motherboard is a 4 layer pcb with traces in the middle.
| sumtechguy wrote:
| The likelihood of them showing and doing that is low.
| However, the likelihood of them showing up with a set of
| USB drives and just running rsync/cp/dd is higher.
| heyoni wrote:
| Normally you unplug the drives and take them to a lab.
| Never let the host operating system continue running with
| those disks!
| jstarfish wrote:
| Maybe in the 90s. Unplugging the drive is how you kick
| FDE in now. The drive only has value while mounted and
| running on the host OS.
|
| Even cellphones...you want them running decrypted, but
| inside a Faraday cage of some kind to block remote wipes.
| heyoni wrote:
| I don't know how FDE works so thanks for the correction.
| I've read stories about feds pulling out drives and
| asking for keys later.
|
| But to run dd wouldn't you need root access? And couldn't
| you use that to dump the FDE keys from memory?
| NhanH wrote:
| Even if that attacks has close to 100% success rate, I'd
| imagine it being nigh physically impossible to execute a
| _targeted_ attack, as you don 't know which machine to hit
| for a specific user. And that seems to be the main threat
| model we would be concerned about for this.
| foobiekr wrote:
| Of course they know which machine to hit. How do you do
| customer service without such a basic function?
| hiatus wrote:
| Mullvad gives you the option to connect to multiple
| servers. They offer wireguard configs for every endpoint.
| How does law enforcement know which server the client
| plans to connect to? There is no metering either, just a
| flat monthly rate so nothing to track there either.
| foobiekr wrote:
| I find these discussions so tiring. Let me turn it
| around. In their position, how would _you_ manage this?
| Might you hook authentication events? Why are you
| pretending this is hard?
| densh wrote:
| You can still mount a remote networked file system to a
| dikless node. Lack of disks does not guarantee inability to
| persist data.
| Brian_K_White wrote:
| If only the system were open source so you wouldn't have to
| wonder about that...
|
| But we do still have to trust that they are actually
| running the code they posted.
|
| Unless that code somehow contains some way to verify
| itself?
|
| I wonder if there is some way to do that? Have the code
| include a hash of itself and some way to query the running
| service that guarantees that the running service must be
| running the code you are looking at?
|
| At first glance it seems any response could always be
| faked, but maybe there is some cryptography trick where you
| submit something, like an encrypted copy of the public code
| maybe, and it crunches and returns something, and that
| somehow proves that the running code you can't see must be
| the same as the code you can see.
|
| Depending on how the protocol for the challenge works, that
| could still be faked. The challenge has to somehow not be
| seperable from ordinary traffic so that you can't have one
| piece of code handle the challenge and another piece of
| code handle other traffic.
| meithecatte wrote:
| There are two known ways to achieve this:
|
| - Multi-party computation. Too much overhead for
| something like this.
|
| - Remote attestation, as seen in e.g. Intel SGX. Usually
| provided by the CPU vendor. Not a cryptographic
| guarantee, more of a "it'd be very hard to defeat this if
| you're not Intel". Probably not that warrant-resistant.
| c22 wrote:
| Homomorphic encryption:
| https://en.wikipedia.org/wiki/Homomorphic_encryption
| lanstin wrote:
| I think the normal solution to this is all the prove you
| are the software you say you are calls is proxied to that
| software and all the normal services calls you want to
| log and duplicate or otherwise violate the contract are
| sent to the modified code.
| johnklos wrote:
| You can talk about hypothetically doing anything. You're
| not really making a point.
|
| Hypothetically, you can rewrite the firmware for the IPMI
| with a backdoor and extract data.
|
| Hypothetically, you could kidnap the family of a developer
| and force them to add a surreptitious side channel for
| exfiltrating data.
|
| Hypothetically, you could run the universe in a simulator
| and use the simulator's controls to read data from RAM in
| the simulated Mullvad servers on the simulated Earth.
| beckler wrote:
| https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...
| tamimio wrote:
| Still doesn't protect you against hardware based backdoors, or
| other types of backdoors like memory injection or supply chain,
| to get data on the fly
|
| > When servers are rebooted or provisioned for the first time, we
| can be safe in the knowledge that we get a freshly built kernel
|
| Any info what's the period of time doing so? Do you provision
| them every day, week? An hour maybe? The more the period the less
| chance of some attack vectors.
| INTPenis wrote:
| This is really cool, you'd expect any VPN provider that cares
| about security and transparency to act like Mullvad. Some pour
| thousands of dollars into forcing influencers to say they care
| about security, while others focus on actually improving
| security.
|
| And it's all open source btw. https://github.com/system-
| transparency/stboot
| flanfly wrote:
| The work on stboot and it's supporting components, including
| documentation was moved to it's own Gitlab:
| https://git.glasklar.is/system-transparency/core/stboot
| jjice wrote:
| > Some pour thousands of dollars into forcing influencers to
| say they care about security,
|
| Tangential to this, it always irks me how they talk about how
| they all act as if the majority of the websites their users are
| going to aren't HTTPS and they act like their main benefits are
| filling in the gaps that HTTPS actually fills in.
|
| HTTPS isn't a cure all by any means but most of the scare
| tactics that the big VPN companies that advertise via YouTube
| act like anyone will rip you credit card because you happened
| to be on Amazon while you were at the coffee shop.
|
| Tom Scott is the only person I've ever seen have a great video
| about this [0]
|
| [0] https://www.youtube.com/watch?v=WVDQEoe6ZWY
| robertlagrant wrote:
| > how they all act as if the majority of the websites their
| users are going to aren't HTTPS and they act like their main
| benefits are filling in the gaps that HTTPS actually fills
| in.
|
| I hear most of them saying "Don't want your ISP spying on
| where you're browsing? Use a VPN." Which HTTPS does not
| cover.
| caseyohara wrote:
| With HTTPS, ISPs can see _where_ you are browsing, but not
| _what_ you are browsing. Of course them seeing the top
| level domain still violates certain aspects of privacy,
| personally I'd prefer ISPs couldn't even see that. But it's
| not like they are peering into the actual content of what
| you are browsing.
| robertlagrant wrote:
| True, and with the rise of CDNs it's probably even harder
| to figure out what's happening. But it still gives them
| more info than is necessary, given they like keeping logs
| for the powers that be.
| [deleted]
| mmarq wrote:
| Providers, at least in Europe, are much more strongly
| regulated than "vpN ProVidErs" re: privacy and everything
| else.
| mardifoufs wrote:
| Which can be a bad thing, if you want to access banned
| websites.
| robertlagrant wrote:
| I imagine it's the reputation of the provider that's the
| main driver, not the regulation.
| hamburglar wrote:
| I'd agree with you about HTTPS providing most of the benefit
| that VPN advertising focuses on if I hadn't seen repeated
| direct evidence that even most technical users will blithely
| click through HTTPS errors' "accept the risk" bypass. It's as
| if knowledgeable users think "sure, this _could be_ a man in
| the middle attack, but it's most likely just a benign cert
| problem, because certs are hard." Sigh.
| c7DJTLrn wrote:
| >if I hadn't seen repeated direct evidence that even most
| technical users will blithely click through HTTPS errors'
| "accept the risk" bypass
|
| As far as I recall this is not possible on Chrome if you
| are MITM'd. If the cert presented doesn't match the cert in
| the HSTS cache, there is no option to bypass. If the
| server's cert is expired, then you do indeed see the
| option, but an expired certificate doesn't necessarily mean
| danger.
| thedaly wrote:
| I haven't experienced an HTTPS error on a legitimate site
| that I would input any personal information into in years.
|
| I couldn't imagine clicking past one of those warnings to
| login to my bank or even amazon.
| noirscape wrote:
| To be frank that's also because the cause for an HTTPS
| certificate error ranges from "malicious hijack" to
| "misconfigured server setup" to "I lapsed the expiry date"
| to "I am using a self-signed certificate".
|
| The degree of which these should be scares is not
| equivalent, yet browsers will treat all of these as
| equivalent even though they can distinguish between them in
| the error page. It just results in clickthrough fatigue,
| where technical users just ignore the warning because it's
| not worthwhile to deal with even when they really should.
|
| Plus a VPN won't protect you from a malicious hijack, it
| just prevents them from grabbing your IP address.
| maccard wrote:
| The reason the browser doesn't differentiate between them
| is because the end result is the same - the cett doesn't
| match the browsers trusted store. The battle has beenosr
| on self signed certs at this point (unless you're an
| enterprise, at which point bundle them with your image).
|
| The difference between a misconfiguration and a
| compromise is intention, both should be treated as
| equally suspicious.
| eximius wrote:
| The problems with clicking past those errors are typically
| not due to network sniffing but with whatever crazy shit is
| on the page they are going to.
|
| The _only_ two valid usecases of big VPNs like these are
|
| 1. _Very mild_ security increase over public wifi 2.
| Shifting your risk from the ISP spying to mullvad or the
| VPN provider spying or slightly anonymizing if mullvad
| rotates IPs.
|
| (2) is a real benefit because ISPs are pretty terrible, but
| it's still pretty minor in the grand scheme of most
| people's threat models.
| digging wrote:
| Yeah, I hate my ISP. I am certain they sell every bit of
| data they can. Ergo, I use a VPN most of the time.
| zarzavat wrote:
| 3. You live in a country where your ISP is legally
| mandated to record all of your browsing history and make
| it available to the government.
|
| 4. You live in a country where certain websites are
| blocked because the government doesn't agree with them,
| or because those websites don't want to deal with your
| country.
| Kwpolska wrote:
| Those countries probably block VPN services, especially
| the popular ones which buy all the ads.
| zolbrek wrote:
| Certain Russian news websites are DNS blocked in the EU.
| I haven't heard of anyone having serious issues using a
| VPN.
| zarzavat wrote:
| There are some countries that block VPNs but there's also
| many countries that don't. For example, TPB is blocked in
| UK by court order but VPNs work just fine.
| Nevermark wrote:
| Nice work!
|
| But, if anything should be a decentralized anonymous crypto-paid
| service, it should be a VPN network.
|
| Centralized VPNs are still a single point of failure privacy
| risk. We have to trust they don't share our identity/account info
| and activity.
|
| I am surprised dVPNs are not THE first rationale given for
| crypto. I.e. since separately and together they (ideally) have a
| clear comparative advantage over other alternatives for strong
| privacy.
|
| A performant global open-standard dVPN could become an
| indispensable layer of web access.
| [deleted]
| Spoom wrote:
| I wasn't sure what a decentralized VPN would look like, so I
| searched and found https://surfshark.com/blog/decentralized-vpn
| . Obvious bias coming from a VPN provider, but if they are
| stating the technology correctly, then I think it's important
| to determine if this is correct:
|
| > A decentralized VPN is a distributed VPN service where
| volunteers supply your VPN servers instead of a single company
| - but paid by crypto. Like with regular VPNs, you have to trust
| that the VPN server isn't monitoring your data. But instead of
| there being a single VPN provider company behind it all, you
| have to trust that none of the thousands of server volunteers
| are spying on you.
|
| Is this a correct understanding of dVPNs? Is there a rebuttal,
| especially to that last sentence?
| Nevermark wrote:
| No that isn't accurate.
|
| You have a network of VPN point providers. As you
| communicate, data can be sent through any series of points.
|
| Data is encrypted end-to-end, and the addresses for the point
| providers are also encrypted so that each point can only
| decrypt and see the next point to forward data to.
|
| So each point knows where data last came from, and where they
| are sending it. But they don't know:
|
| 1. Which step of a chain of points the data is at.
|
| 2. If they are the first in the chain (i.e. the "from" is the
| source)
|
| 3. If they are the last in the chain (i.e. the "to" is the
| destination)
|
| And (as long as two or more points are traversed, which would
| be always), no point ever has access to:
|
| 4. Both source and destination info.
|
| Finally, since payments to each point are handled through a
| combination of peer-to-peer point bookkeeping, and a crypto
| block chain account, no point ever knows:
|
| 5. Any identity information about who uses the VPN.
|
| 6. Any way to identify activity over time that is related.
|
| Acting as a point, as well as using the network, serves to
| further cloak activity, as being from you vs. passed through
| you.
|
| And an alternative to crypto payments, would be earning usage
| by providing point service.
|
| EDIT:
|
| > so I searched and found https://surfshark.com/[...]
|
| Any VPN provider that is claiming decentralized VPNs are a
| greater risk is either misinformed, or willing to misinform
| users.
|
| I wouldn't trust a VPN provider from either category.
|
| Actual reasons to not use a dVPN might be that it is a work
| in progress, not supported well, its source code is not open,
| or not yet vetted by experts, too slow, not many points yet,
| etc.
| tredre3 wrote:
| Aren't you just describing Tor?
| KomoD wrote:
| Yes, that is correct. It's great for getting residential IPs,
| but connection quality is much worse
| max_ wrote:
| Have a look at Nym
|
| - https://nymtech.net/
| ShowalkKama wrote:
| >But, if anything should be a decentralized anonymous crypto-
| paid service, it should be a VPN network
|
| so it should be tor?
| w1nst0nsm1th wrote:
| In north america and europe, VPN are required by law to keep logs
| of your use of vpn (site you visit, inscription email,...) for 1
| or 2 years.
|
| Most VPN company advertise they do not keep logs of your
| browsing...
|
| Which would be in infraction with european and american laws.
|
| So I don't what to think of diskless VPN.
| PaulHoule wrote:
| "They" will just spray the machines with liquid nitrogen, pull
| them out of the rack, put the DRAM in a thermos w/ LN2 and read
| the data at their leisure.
|
| https://ieeexplore.ieee.org/document/8388826
| maccard wrote:
| That seems like significantly more work, and significantly more
| error prone than pulling an SSD out of a machine and reading
| from a log file
| skeaker wrote:
| The word "just" is doing some heavy lifting here... To "just"
| do this, the agent would need to more or less completely take
| over the building infrastructure before Mullvad could react
| which is a lot easier said than done. Even if it were trivial
| it's still quite a few cuts above any competing VPN service.
| Cyphase wrote:
| IANAL, but I think "reacting" to a warrant in the way you're
| implying might be illegal in some places.
| tredre3 wrote:
| AMD processors support encrypted RAM, called SME[1]. The key is
| internal to the CPU and randomized at boot. Sniffing a live RAM
| chip or reading a perfectly preserved frozen RAM will give you
| nothing. It's a big part of why the xbox one was never hacked.
|
| You can enable SME in the BIOS on all AMD-based business
| laptops and AMD EPYC servers.
|
| 1. https://www.amd.com/content/dam/amd/en/documents/epyc-
| busine...
| sneak wrote:
| With modern encryption protocols, this yields you nothing.
|
| The feature is called Perfect Forward Secrecy, and protects
| past flows from later key compromise.
|
| Wireguard supports this, which is what Mullvad uses. (For some
| reason, speculation about which is an exercise left to the
| reader, WPA in Wi-Fi still does not.)
| woodruffw wrote:
| Not exactly nothing, just not ongoing compromise. TLS session
| keys can be pretty long-lived; I don't know how long-lived
| Wireguard's equivalent keys are, but even a relatively
| conservative few minutes can yield valuable traffic and
| metadata.
|
| (That being said, I think having your RAM frozen to extract
| ephemeral secrets is firmly in the "fully hosed" threat
| model, and is not a realistic model for 99.9% of users to
| plan for.)
| dheera wrote:
| > freshly built kernel, no traces of any log files, and a fully
| patched OS
|
| Wouldn't using a disk in read-only mode accomplish the same
| thing?
| altairprime wrote:
| Disks don't always have a readonly switch these days, though I
| do still miss the physical notch on floppy disks, and no third-
| party auditing could exist for proving that switch to be set
| correctly.
| dheera wrote:
| No third-party auditing could exist that proves you only have
| RAM in the system and don't have a secret disk in there with
| a magnetic reed switch in-line with the SATA power cable such
| that without sticking a magnet on the case the disk doesn't
| show up. Or that you aren't booting off a USB drive that you
| plug in only after the auditors leave.
|
| Third-party audits are a scam to begin with and don't prove
| anything.
| altairprime wrote:
| A third-party audit can prove that the system functions as
| shown without a hard drive, and a third-party auditor can,
| using contractually-authorized random unscheduled spot
| checks, physically inspect the live deployed servers to
| confirm the absence of any disk media.
|
| Third-party audits prove something. They don't prove
| everything.
| nashashmi wrote:
| > All of our VPN servers continue to use our custom and
| extensively slimmed down Linux kernel, where we follow the
| mainline branch of kernel development.
|
| The custom server is a niche security point. While every server
| is continously researched and patched, we cannot expect the same
| from a a server like this. If someone were to find a security
| hole, an attacker would purchase it and no one else would ever
| know the system was compromised.
| nodesocket wrote:
| Great point brought up in the comments that VM's allow for
| snapshotting of entire memory state as well. So something to be
| aware of.
| crabbone wrote:
| Does pmem count as RAM?
| infofarmer wrote:
| Not to provoke predictable responses, but I find it interesting
| that the tech-talented VPN providers are not using BSD in favor
| of Linux, especially with requirements like diskless operation,
| kernel customization, and tighter security.
| latchkey wrote:
| For me, the pool of people to hire that know Linux inside and
| out would be much larger. This is worth any perceived security
| issues.
|
| In terms of diskless, I've run 25k+ iPXE deployments on
| diskless blade servers using a highly customized Ubuntu, and it
| was fantastic.
|
| Regardless of OS choice, being diskless is also quite nice...
| if there was a security issue or you need an upgrade of some
| sort, you just reboot. Only thing is that it takes a while to
| reboot 25k servers... even on gigE. It was a bit of work to
| build the scheduling system to make that happen reliably, but
| it worked out quite well.
| kwanbix wrote:
| One thing that I always wondered from VPNs.
|
| Let's say a pedophile uses Mullvad to get forbidden images, isn't
| the VPN liable?
|
| I mean, the law enforcement will see that the IP was from
| Mullvad's office, so I assume they are the ones doing it? How do
| they avoid this?
|
| It is a real doubt. Maybe stupid, but real.
| darreninthenet wrote:
| IANAL but my understanding of current case law is that it IP
| address does not automatically mean a particular person.
| Obscurity4340 wrote:
| Pretty sure if they live by themself and nobody else comes
| into their dwelling and there is no other name attachef to
| their subscriber info it does
| mrweasel wrote:
| What if they are on cellular and that hasn't been upgraded
| to IPv6?
|
| Years ago I handled fraud cases for an e-commerce site with
| local police, at some point they started asking for IP and
| port numbers for the offenders, rather than just the IP.
| Turns out that one of the cellular phone providers had
| basically run out of IPv4 addresses for their 4G network
| and did some NAT solution. If you didn't have the port
| number the client had connected from then they could only
| tell you which cell tower had been used, not who the
| customer was.
| 14 wrote:
| Definitely not. I still am logged into my ex girlfriend
| wifi so if I wanted to harm her I could easily go stand
| outside her home at night and download malicious files.
| That would not make her guilty. They may investigate but
| that is not proof she did something unlawful.
| darreninthenet wrote:
| Nope... wardriving? Spoofing? Too much uncertainty to
| convict with. Basis for a warrant on the property? Yes,
| probably.
| uxp8u61q wrote:
| Case law in what country? Mullvad is Swedish, are you
| knowledgeable about Swedish case law?
| jameskilton wrote:
| I am not a lawyer, but my understanding is that this generally
| falls under Section 230, as you can make the same argument
| about Comcast, AT&T, et.al. who lets the bytes go over their
| infrastructure.
| kwanbix wrote:
| But the difference is that Comcast, AT&T, et.al can say,
| jameskilton was using this IP. The VPN is saying, I don't
| know.
| hollander wrote:
| What if you get the same IP time and again?
| nwellnhof wrote:
| That's only a problem if the VPN is in a jurisdiction with
| data retention laws:
| https://en.wikipedia.org/wiki/Data_retention
| manishsharan wrote:
| and then suppose you login to that VPN and are looking up
| children's sweaters for your kids and keep the session on ..
| while law enforcement is looking up the ip address associated
| with the earlier activity which is now assigned to you . Good
| luck explaining to the the cops about VPNs and IP addresses.
|
| This is my fear.
| dspillett wrote:
| You are not going to be the only person appearing to come
| from that IP address - many will likely be NATed through it.
|
| The more significant concern is if you are the other side: if
| you deliberately run some sort of VPN or other proxy that
| others can use, or less deliberately do so. Many hacked or
| otherwise suspicious browser add-ons, and other malware, will
| make HTTP(S) requests & other connections on behalf of their
| C&C hosts and to your ISP or anyone else those requests will
| be largely indistinguishable from those that are the result
| of your activity.
| flkenosad wrote:
| You don't have to explain anything to cops. You explain it to
| lawyers and judges.
| dylan604 wrote:
| You actually shouldn't even say anything to the cops. If
| they show up with a warrant for arrest as well as search,
| you're going to jail no matter what you say. If they show
| up with just a search warrant, they are going to take
| whatever they want to take whether its outside the purview
| of the warrant or not. It will be up to a lawyer to
| convince a judge it was out of scope at a later date after
| it has already been taken. You will never convince a team
| of cops that their warrant is wrong when they show up. The
| only chance you have is if you're uber criminal and have
| your attorney present when they arrive.
| mrweasel wrote:
| > You actually shouldn't even say anything to the cops.
|
| Unless you're in the UK, in which case: "You do not have
| to say anything. But it may harm your defense if you do
| not mention when questioned something which you later
| rely on in court. Anything you do say may be given in
| evidence."
| dylan604 wrote:
| As a Yank, that line always felt odd when watching
| BritCop dramas. How is the alleged meant to know the
| specifics of a defence when the full charges haven't even
| been levied, or how is the alleged meant to read the mind
| of a lawyer? It just feels like something rigging the
| system
| dspillett wrote:
| And the court of public opinion. By the time lawyers and
| judges are involved, unless you are very lucky, your name
| and photo is all over the tabloids. Any retractions
| published when you are later found completely innocent will
| be the equivalent of a column inch or two on page 17.
| WendyTheWillow wrote:
| Simply not an issue for nearly everyone.
| ThePowerOfFuet wrote:
| Until it happens to you.
| WendyTheWillow wrote:
| No, even if it happened to me it would not be relevant
| for the vast majority.
| hiatus wrote:
| Maybe if you live in Florida.
| x86x87 wrote:
| Who enjoys privacy when we can all live in fear?
|
| You need a VPN that actually cares about your privacy and
| goea the extra mile to ensure it. On top of that if the VPN
| service does not know who you are how can they actually tell
| the cops. On top of that you don't need to explain it to the
| cops - if you are ever accused this should be done in a court
| of law where we understand what ips are (heck, even some cops
| understand it - it's not exactly rocket science nowadays)
| madspindel wrote:
| Mullvad actually got raided by the police in a similar case,
| described here: https://mullvad.net/en/blog/2023/5/2/update-
| the-swedish-auth...
|
| > However, had they taken something, it would not have given
| them access to any customer information.
|
| > These are the national laws that makes it possible to run a
| privacy-focused VPN service in Sweden:
| dylan604 wrote:
| All this would do would be to lead the investigation to get a
| warrant/subpoena to have the VPN service provide user details
| about the account and anything else relevant like logs. This is
| where the "we don't log shit" bullet points comes into play as
| well as running only from RAM. If the warrant allows for
| removal of hardware, all data is lost once power is removed.
| LEOs would have to bring lots of batteries.
| throwmeaway2232 wrote:
| or liquid nitrogen.
| https://en.wikipedia.org/wiki/Cold_boot_attack
| bgm1975 wrote:
| If the feds have physical access and considering the high
| likelihood that these are VMs and not physical, it would be
| a whole lot easier to get the hypervisor to just snapshot
| the VM w/ its memory and perform forensics against that
| file(s).
| nerdbert wrote:
| They're going to freeze the whole data center? It's rack
| after rack of machines that the traffic could have passed
| through, right? And if they're not logging IPs to RAM then
| they only have a fraction of a second to get the right one
| before the register is overwritten with the next user's
| info.
| mlyle wrote:
| You do need to know where to send the user's return
| traffic, so you'll need a table ultimately comprising
| mappings of network flows to end-user addresses. Of
| course, once the flows close you don't need to retain
| this information. In practice, you'll also need
| information about all currently-open VPN sessions.
| Dennip wrote:
| They don't avoid it - which is why they were raided by the
| police at one point and why they're no longer offering port
| forwarding
| HPsquared wrote:
| I wonder about those VPNs that say "we don't log or store
| anything". That may be the case, but they probably just send a
| continuous stream of data to the law enforcement / intelligence
| services or whoever instead of storing it themselves. They can
| then correctly say "WE don't log".
| NegativeK wrote:
| That's an... interesting? interpretation of what logging is.
|
| What you've described, to me, is the VPN logging customer
| activity and then sending it elsewhere to be stored.
| [deleted]
| NoMoreNicksLeft wrote:
| I don't think that the Houston Police Department (or whichever
| city you're in) is sophisticated enough to set up an NSA-fiber-
| optic-backbone-data-siphoning operation. Yes, the NYPD is known
| to buy illegal million dollar cell phone interceptor machines,
| but that's about it the maximum level that sort of thing ever
| goes, for the largest municipal law enforcement agency in the
| United States.
|
| Like, how would that even work? Without a court gag order,
| gossip would make its way out of the building in weeks. The
| cell phone shit only was only quasi-secret because _only_
| police department employees were involved, something that 's
| impossible for these VPN outfits. They don't get any of the
| (unjustified) privilege that the CIA or NSA (or even the FBI,
| sometimes) receive.
|
| Anything I might do that could pique the curiosity of law
| enforcement is definitely below the level of federal
| intelligence agency interest. Maybe your life is more exciting
| though.
| lopkeny12ko wrote:
| This has never made any sense to me. I'm surprised this isn't a
| massive red flag from anyone on HN.
|
| Running a production-grade service with zero metrics and logs?
| If there's an outage, or even something as mundane as a VM
| failing to provision, you're telling me that Mullvad developers
| just shrug and say "well, we can't do anything, because there's
| no logs!"
|
| I don't use a third party VPN, but if I wanted to, "we
| deliberately eschew all observability" is not a positive
| selling point.
| CydeWeys wrote:
| Who said they don't have metrics? The VPN servers don't have
| any storage, but they can still send metrics to an off-
| machine API, or a different server that does have storage can
| send requests to the VPN servers and do metrics that way.
|
| Ditto for logging. They claim to not log activity over the
| VPN itself, but I don't see any claims about not logging more
| mundane infra stuff like "a VM failed to provision". I think
| you're arguing here against claims they aren't making.
| portpecos wrote:
| > but they can still send metrics to an off-machine API
|
| And that would be the next most interesting post, imo. A
| post about how they metric and log in a RAM-only
| environment while obscuring or obfuscating the details that
| could lead to "compromise". Even if the answer is something
| so simple like "we regex and discard this out", I would
| feel more trusting of their services.
| blackoil wrote:
| Metrics are mostly by nature anonymous. Things measured are
| CPU/Mem usage, network rate. Metrics at IP/user level aren't
| of much value. Companies add country/device type attr. but
| they can be done without.
|
| Logs can similarly be of system events only.
| klysm wrote:
| you can send logs and metrics over the network. the important
| part to users is not logging the traffic info
| lopkeny12ko wrote:
| Sending them over the network to where? "We don't store
| logs" means they certainly aren't being ingested into any
| persistent storage. I'm highly interested in how one can
| run time-series queries over /dev/null.
| CydeWeys wrote:
| They never said "We don't store logs". What they said is
| they don't keep logs of user activity. You're arguing
| against a strawman.
|
| https://mullvad.net/en/help/no-logging-data-policy/
| klysm wrote:
| you are conflating logs & metrics with user activity
| logs.
| eddythompson80 wrote:
| Yes, because it's a simple service (VPN) that hasn't added a
| billion nonesense features over time. You can log VM
| provisioning and health logs but as long as you don't log any
| wireguard logs or user provisioning logs you're good.
| traceroute66 wrote:
| > I wonder about those VPNs that say "we don't log or store
| anything". That may be the case, but...
|
| There may inevitably be some bad actors.
|
| But then there are other companies like OVPN who proved in
| court that when they say no logging they mean it[1].
|
| [1] https://www.ovpn.com/en/blog/ovpn-wins-court-order
| vinay_ys wrote:
| Even with an honest company, the pressures on them are twofold
| - security and legal. Their systems can be compromised through
| security vulnerabilities and social engineering (including
| coercion - money, ideology, compromise, ego - classic psyops
| playbook). Or they can get legal government orders - which
| pretty much every government in the world have laws on books
| and operational practices to force any actor to hand-over data
| in the name of fighting money laundering and terrorism
| (AML/CFT). It is very expensive to put up strong defenses
| against these. I don't see a viable business model charging
| $5/month that covers regular operational expenses and covers
| these types of events.
|
| Edit: Forgot to mention backdoors built into basic technologies
| they may already be using - like the Cavium HSM thing that came
| to light earlier this week.
| mplewis wrote:
| What evidence makes you believe this is happening?
| rsaxvc wrote:
| Historical: Room 641A at AT&T in the US.
| [deleted]
| karaterobot wrote:
| I inferred that they were speculating, not that they had a
| smoking gun piece of evidence. The word "probably" is what
| tipped me off, ymmv.
| 2OEH8eoCRo0 wrote:
| To claim such a thing you should put up some evidence. I think
| they are actually streaming all data to the Martians.
| eddythompson80 wrote:
| Given the ridiculous number of VPN providers that say that, how
| are they keeping the secret exactly?
| KomoD wrote:
| > but they probably just send a continuous stream of data to
| the law enforcement / intelligence services or whoever instead
| of storing it themselves. They can then correctly say "WE don't
| log"
|
| No they can't, because THEY are still logging.
| aaomidi wrote:
| https://www.assured.se/publications/Assured_Mullvad_relay_se...
|
| Honestly I don't think audits are worth anything. But it'd be a
| huge conspiracy to mess with so many parties.
| foobiekr wrote:
| This is a sec eval. It doesn't eval what the service can do.
| Ajedi32 wrote:
| Sections 2.1.1 and 3.1.18
| verandaguy wrote:
| Audits are IMO worthwhile, but end users should be aware of
| the scope of an audit. In the context of commercial VPN
| providers, it's usually just a code security audit -- are
| there any memory leaks? Is sensitive data being passed around
| a little bit too loosely? Is there some way for unprivileged
| users to gain privilege escalation by crafting a malicious
| request against one of your services?
|
| In this sense, they're valuable. As someone working in
| software, I can figure out if the bugs were subtle or
| blatant, which is often a good proxy metric for the
| competence of the team behind the product. Are the same bugs
| cropping up year after year, even if they've already been
| previously fixed in other parts of the code? Again, a good
| red flag to use there.
|
| Audits do not and often cannot cover things like "is the
| company reselling connection/user metadata to other
| companies," though, and in most cases consumers will care
| that there _is_ an audit rather than caring what 's in the
| audit.
| patrakov wrote:
| Well... using PureVPN as an example. They claim that they
| have been audited twice.
|
| First audit, from 2019:
| https://my.purevpn.com/pdf/Privacy_No_Log_Audit_Report.pdf
|
| I tried to contact the auditor, Altius IT, in order to
| confirm whether exfiltrating connection data to a third
| party would result in the audit failure. They replied, but
| in a very vague way (refused to answer any questions
| regarding Altius IT's audit of PureVPN's environment).
| Well, at least they confirmed indirectly that the audit did
| exist.
|
| Second audit, from 2023: https://www.purevpn.com/wp-
| content/uploads/2023/07/KPMG_Pure...
|
| I tried to contact KPMG to verify the authenticity of that
| report, and also asked the same question - "whether
| deliberate real-time exfiltration of origin IP addresses,
| assigned VPN IP addresses, connection timestamps, or
| connected user activities to a third party by PureVPN,
| without PureVPN (as opposed to that hypothetical third
| party) storing anything locally in any form of logs, would
| have constituted a failure of the privacy assessment."
| Result: no reply from KPMG at all, so I cannot be sure even
| that the report indeed comes from KPMG and is not a fake.
| verandaguy wrote:
| There are bad auditors, of course. Having had the
| displeasure of working with KPMG (not in a code-security-
| audit setting, mercifully), I genuinely don't understand
| how their staff can be allowed within a ten mile radius
| of source code.
|
| The ideal way to authenticate audits IMO would be for the
| audited entity to link back to a PDF or other report
| hosted on the auditor's site.
| verandaguy wrote:
| I formerly worked for a somewhat-older mainstream consumer VPN
| provider for a few years, to the extent that you can take my
| word for it, this is not industry-standard practice at least as
| far as the provider is able to control it.
|
| Commercial VPNs typically run on rental servers -- usually a
| mix of the major cloud providers and smaller hosting providers
| -- and in my former company's case, using dedicated hosting
| (bare metal where available). Steps were taken to restrict
| access for physical actors, but ultimately, the mantra's always
| that physical access basically guarantees data access on a long
| enough timeline if you assume there's a bad actor in the mix.
|
| That said, to the best of my memory, there were no indications
| of this kind of data siphoning happening without our knowledge,
| and we absolutely didn't take part in it ourselves knowingly.
| Occasional requests would come in from various international
| law enforcement orgs, and every time they'd be replied to with
| a message about how we don't store user records (which was a
| truthful reply AFAIK).
|
| The biggest challenge for us was competing with some of the
| newer actors in the space, taking advantage of deceptive
| marketing and engaging in (IMO) unethical business practices
| for the sector:
|
| - Claims of "no logging," even backed up by audits, are only
| ever point-in-time measurements, and may not reflect reality if
| the VPN provider approaches the auditors in bad faith (say,
| with a sanitized code base); a good auditor in my experience
| will refuse to make this claim in the report
|
| - Claims about having the corporate HQ in one country making it
| immune from the laws of countries they operate servers in (this
| is deceptive marketing; failure to comply with laws will get
| you shut down, and at my old employer we'd make calls about
| whether to just drop our server presence in a country entirely
| in response to local laws and political happenings)
|
| - Commercial resale of user data is (allegedly) _rampant_ among
| many of the newer providers you see constantly plugged on
| Youtube. This isn 't helped by the _massive_ consolidation of
| the VPN market under just 2 or 3 holding companies.
|
| I won't name names for the companies I mentioned above, but my
| recommendation is to adjust your threat model from "nation-
| state level surveillance" to "commercial data resale just like
| every other web service."
|
| As far as data collection went for my old company: we collected
| system metrics like resource usage over time, and kept minimal
| sanitized logs to help diagnose any production issues that'd
| come up -- basically the absolute minimum amount of data we
| needed to keep the service operating smoothly. I have every
| reason to believe this is an industry norm, since otherwise
| development and troubleshooting would be nearly impossible.
|
| Anyway, there's also the looming "threat" (lol) of HTTPS and
| encrypted DNS proliferation and improvement making the core use
| case for commercial VPNs obsolete. I think anyone who's spent a
| bit of time in that industry realizes that the business model
| isn't long for this earth as a result, so I suspect many are
| trying to milk the industry for all it's worth. Personally, I'm
| all for HTTPS and encrypted DNS proliferation, and I'm also
| hoping more and more commercial public networks start using
| virtual private subnets and other device isolation features to
| make it even harder to abuse coffee shop Wi-Fi.
| azalemeth wrote:
| The other argument is to frustrate network correlation
| analysis. Many VPN providers have an internal high-bandwidth
| network (virtual or otherwise); you can send a packet to
| $VPN_SERVER_X, it sends it to $VPN_SERVER_Y possibly via
| other intermediate servers, and $VPN_SERVER_Y then forwards
| it on to your destination.
|
| If you live in a country with detailed data retention laws,
| this massively changes the shape of the graph: rather than
| your computer connecting via HTTPS to lots of other IP
| addresses, it only connects to one, which a large number of
| other customers do too. The argument then goes that there's
| enough inherent jitter and generic "chaff" on the internal
| network to make it very hard to deterministically work out if
| one of your packets going in to a popular service is the same
| as that coming out at any moment in time; the greater the
| traffic of the network and the provider the better the
| statistical protection becomes as the packets become
| indistinguishable.
|
| This, and the fact that it represents a giant "no thanks" to
| dragnet surveillance, is arguably a good reason to just put a
| VPN on your router (as many people do).
| robertlagrant wrote:
| > Anyway, there's also the looming "threat" (lol) of HTTPS
| and encrypted DNS proliferation and improvement making the
| core use case for commercial VPNs obsolete
|
| For a lot of people the core use case is accessing Netflix in
| a different country!
| mixmastamyk wrote:
| If you have to pay for safe, encrypted DNS, how is that
| substantially different than using a VPN? Still need an
| external service.
| ordinarycode wrote:
| You can use dns over https over tor(dohot)[1]. Safer than
| a vpn if you dont mind your isp knowing you go to tor.
| 1.https://github.com/alecmuffett/dohot
| verandaguy wrote:
| I'm not sure how this relates to the parent comment, but
| there _are_ free encrypted DNS services out there, though
| the same can 't be said for encrypted and anonymous ones
| (which is, frankly, a hard problem to solve,
| realistically speaking).
|
| With encrypted DNS you're just shifting the burden of
| data privacy away from the local network to the DNS
| operator. How you determine which operators to trust will
| probably vary from person to person.
|
| Anyway, the major difference here would be that a VPN
| will encrypt all traffic in a tunnel, from your DNS
| requests to your actual followup web requests. On the
| flipside, you may use encrypted DNS to look up records
| for a domain that serves content over an unencrypted
| connection.
| verandaguy wrote:
| This is also true and shockingly difficult to do reliably.
| dtx1 wrote:
| I always assumes the core usw case was piracy
| gs17 wrote:
| I'm constantly amazed VPNs get away with advertising that,
| more specifically the ones that advertise lower prices for
| subscriptions/products. I guess Netflix themselves won't
| really care if you switch regions for different shows and
| might write off discounted subscriptions as alternative to
| piracy, but the companies that license content by region
| don't care?
| foobiekr wrote:
| This is how lawful intercept systems have worked since the
| nineties. You'd have to go look at their jurisdiction to see
| whether there are laws mandating LI that impact vpn providers.
| Or Mulvad could clear it up, I guess.
|
| Sweden absolutely has LI requirements for all telecom gear but
| vpns I have no idea.
| eviks wrote:
| If you're that compromised, wouldn't it be much easier to just
| log and lie about it?
| smith7018 wrote:
| Lying about it opens you up to potential litigation and being
| exposed through discovery. It makes less sense to outwardly
| lie to paying customers rather than simply lie by omission.
| hinkley wrote:
| I can just imagine EFF drooling over such a prospect.
| TheRealPomax wrote:
| The EFF wants a world where this kind of BS, and
| consequent litigation, is a thing of the past. If there's
| water running down their face, it's tears, not drool. We
| deserve a better world.
| hinkley wrote:
| You get there by either getting congress to pass laws or
| by setting legal precedent in the court. That's the goal
| of the legal arm. Not to fight every bullshit civil
| liberties case, but to fight the last one that doesn't
| get thrown out in court.
|
| If you don't think they look forward to those cases, I
| think you're the one who has read them wrong, not me.
| CrazyPyroLinux wrote:
| this is what "warrant canaries" are for. dont use anyone who
| doesnt have one
| NegativeK wrote:
| There should probably be case law before anyone actually
| believes in warrant canaries.
|
| 'If it's illegal to advertise that you've received a court
| order of some kind, it's illegal to intentionally and
| knowingly take any action that has the effect of
| advertising the receipt of that order. A judge can't force
| you to do anything, but every lawyer I've spoken to has
| indicated that having a "canary" you remove or choose not
| to update would likely have the same legal consequences as
| simply posting something that explicitly says you've
| received something. If any lawyers have a different legal
| interpretation, I'd love to hear it.' --Moxie Marlinspike
| powersnail wrote:
| What happens when someone asks you whether you have
| received a court order of some kind? Are you compelled by
| court to lie about it?
| NegativeK wrote:
| I'd hire a lawyer that knows how to deal with them and
| knows how to say no comment.
|
| I've always felt that the warrant canary is a nerd's
| gotcha designed to get out of a sketchy legal process
| (NSLs) and that judges would be very unsympathetic. But
| IANAL.
| Jerrrry wrote:
| "We must comply with legal subpoenas in the jurisdiction
| in which we operate,"
| TheRealPomax wrote:
| Why would someone have to lie? They can just say "We
| can't comment on that" without providing an answer. And
| then customers can go "sounds pretty suspicious, time to
| switch VPN services".
| Jerrrry wrote:
| Literal security theatre.
| Terretta wrote:
| > _This is what "warrant canaries" are for. dont use anyone
| who doesnt have one_
|
| What if they have one like this quarterly canary at
| privacy-forward "write.as" last updated 9 months ago?
| It should be noted with significance if this message
| fails to be updated on a quarterly basis.
| 2023-01-05 21:06:06 UTC No warrants have
| ever been served to Write.as, or Write.as
| principals or employees.
|
| https://write.as/privacy --> https://write.as/canary.txt
___________________________________________________________________
(page generated 2023-09-20 23:02 UTC)