PRISM, or, How I Learned to Stop Worrying and Love the Police State
Multiregion deployment has been my white whale the whole time I have been working on Knudge. Setting up infra stuff was one of the first things I did when I started working on it part time, and it was certainly my first sizeable contribution. Even at the time, I knew I wanted to do this but had no idea about the constraints and weirdness - some of it obfuscated (purposefully? nefariously?). Computer networking stuff is already incredibly complex, and it gets measurably more so when there are people purposefully getting in the way.
At the same time, there are hundreds of services that let you throw up a globally distributed web app and database in minutes. So what gives? Why didn't I figure out the (seemingly simple) task, overcome a many-times-solved problem, and move on?
Web application networking is
There's a fast track for deploying a networked app in any cloud provider. A web server is an incredibly standard piece of infrastructure at this point, and it's no mystery in how to set one up. It's even very simple to deploy something like that globally, across many regions, and have it Just Work™. And it does. You have a Layer 7 global load balancer set up that clients get routed to via a single IP (this is referred to colloquially as an "anycast" IP). That IP refers to (potentially) many different physical data centers - different endpoints on the load balancer - and Internet Magic™ routes the client to the closest one. At that point, the SSL connection is terminated and traffic is decrypted and sent through the cloud provider's network. If the traffic cannot be handled locally (if you don't have servers in that region) the traffic routed via the cloud provider's "backbone" (private fiber connecting their various data centres across the world) to a place that can handle it. Because the traffic is decrypted, the load balancer can do smart load balancing based on headers (e.g. session cookies) being able to better distribute traffic evenly across servers, or better direct specific users to specific regions.
Hopefully, alarm bells went off at the SSL connection is terminated and traffic is decrypted. This is, at a glance, bad. A app run like this is exposing the full-context of someone's interaction with the app to the cloud provider. With increasing paranoia:
- If we fully "trust" our cloud provider (we do not), this is still security gap - unencrypted traffic in servers we don't control, which could be intercepted or rewritten if the cloud provider had a security breach.
- Our cloud provider is Google. Their whole thing is harvesting user data and selling it. They love other peoples' data. I do not trust them to restrain themselves. I'd have to scour their ToS to see if anything in it prevents / allows for this kind of data harvesting, but it wouldn't surprise me if there's some loophole. This is the kind of thing that, when another large corp agrees to use Google Cloud, they probably have a separate contractual agreement that includes some clause about not being able to do precisely this.
-
We live in a post-PATRIOT-Act (et al.) world, and thank whoever
we've had some whistleblowing. This is precisely the channel used by
PRISM PRISM is a code name for a program under which the United States National Security Agency collects internet communications from various U.S. internet companies. The program is also known by the SIGAD US-984XN. PRISM collects stored internet communications based on demands made to internet companies such as Google LLC and Apple under Section 702 of the FISA Amendments Act of 2008 to turn over any data that match court-approved search terms. Among other things, the NSA can use these PRISM requests to target communications that were encrypted when they traveled across the internet backbone, to focus on stored data that telecommunication filtering systems discarded earlier, and to get data that is easier to handle. PRISM began in 2007 in the wake of the passage of the Protect America Act under the Bush Administration. The program is operated under the supervision of the U.S. Foreign Intelligence Surveillance Court pursuant to the Foreign Intelligence Surveillance Act.
No need to make the NSA's job any easier, they do a good enough job contributing to the abduction and torture of humans as it is.
The thing we're trying to preserve is a TCP (Level 4) connection all the way down to the servers we control, so that we terminate the SSL connection and decrypt the traffic, and nothing else does in-between. This means our traffic remains encrypted even if it travels on Google's backbone network. Theoretically, Google could provide a global TCP load balancer that would have most of the benefits of the Layer 7 one, but traffic would remain encrypted on their backbone. Conveniently, they do not. You can read into that as much as you want to.
Instead, for now, we're doing DNS-level regional resolution. This means our DNS provider is set up to explicitly respond with different IP addresses depending on the region of the requester (ie. which region's DNS server is being hit). This is an alternative to using what's referred to as a "multicast" IP - where app.knudge.com resolve to multiple IP addresses, and Internet Magic™ determines which has the shortest network route to our servers. It's slightly more controlled/predictable, but is missing niceties of load balancing that breaks through to higher network layers, and is able to do things like
This has drawbacks: we're open to more types of DDoS attacks than being behind the wall provided by a cloud provider's higher-network-layer, global load balancer, and we don't benefit from as much edge network routing. Still, we have regional LBs in front of the app — they take care of the more nefarious things, and our throttling takes care of most others, and our regional distribution is solid enough to serve customers where they're at.
This site — my personal site, the one you're on now — is hosted by Deno.
Their politics don't matter. They are terminating SSL in front of the web
server that hosts this site. There are unknown network steps between that
SSL termination, and code I have personally written, where network traffic
flows unencrypted. This is inherently untrustable, yet it is the de-facto
order of business for just about everyone these days. While I haven't
written my very own post/article/essay titled
The Power of Defaults yet, there are plenty out there, and
there's a reason why. I'm getting free web hosting, for which I am
grateful, fronted by a very reliable service. However, the default is that
any traffic to my site is being decrypted by code I do not control,
traversing networks that I do not trust. Convenience that may come at the
cost of true privacy, and in the long run, true freedom.