DIY VPN (part II)

[continued from part I]

Economics of a self-hosted VPN

This section considers the overall cost of operating a VPN server. First, we can rule out software acquisition costs. On the server side, the community edition for OpenVPN is freely available as open-source software. (The commercial dubbed OpenVPN Access Server comes with a per-user licensing model. While adding useful fit-and-finish including configuration GUI, that version does not change the fundamental scaling or security properties of the system.) On the client side, there are freeware options on all major platforms. One exception is when ADCS is leveraged for the certificate authority. Even there one can lease hourly access to a private Windows server from major cloud providers, rolling that into operational expenses instead of paying outright for the software. As far as operational expenses are concerned, what follows is a back of the envelope calculation using AWS as the infrastructure provider. While AWS has a free-tier for new customers, this analysis assumes that discount has already been exhausted, in the interest of arriving at a sustainable cost basis.

Computation

OpenVPN runs on Linux and is not particularly CPU intensive at the scale envisioned here— supporting a handful of users with 2-3 simultaneous connections at peak capacity. A single t3-nano instance featuring a single virtual CPU and half GB of RAM is perfectly up to task for that load. Based on current AWS pricing, these cost roughly half cent per hour when purchased on demand in US regions such as Virginia or Oregon. The operational cost can be reduced by committing to reserved instances. Paying upfront for one-year takes the price down to 0.30¢/hour while a three year term yields 0.20¢/hour. Even greater savings and flexibility may be achieved with what AWS terms  spot instances: current spot-prices for nano instances hover around 0.16-0.20¢ per hour depending on availability zone. However spot instances comes with the risk that the server may be preempted if demand spikes to the point where prices exceed the bid. That level of reliability may still be acceptable for a personal setup, since one can always bid on another spot instance with higher price. Conservatively using the 0.20¢ estimate, we arrive at EC2 instance costs just shy of $1.50 per month.

Storage

It turns out that the cost of running an instance is not the primary consideration, because of the modest computational requirements of openvpn. When using minimal instance types such as t3-nano, disk space becomes a significant contributor to cost. At the time of writing, AWS charges 10¢ per GB/month for entry-level SSD volumes attached to instances. Costs only go up from there for more demanding loads and guaranteed IOPS. (While there are cheaper options such as spinning disks, those can only be used in addition to existing boot volume and not as replacement.) That means simply keeping a 15GB disk around for a month alone exceeds the cost of running the instance for that same period.

Fortunately Linux distributions are also modest in their demands for disk space requirements. Minimum requirements for Ubuntu18 server— not the desktop edition, which is considerably more resource hungry— weight in at a svelte 256MB of memory and 1.5GB of disk space. It is not possible to translate that directly into an AWS image: all popular AMIs offered by Amazon come with minimum 8GB disk space, more than five times the stated baseline. AWS will not allow launching an EC2 instance based on one of these AMIs with a smaller boot volume.

Bring-your-own-image

There is a work around: AWS also allows customers to import their own virtual machine images from common virtualization platforms. That permits crafting a properly slimmed-down Linux image locally and converting it into an AMI.

Fair warning: this process is clunky and error-prone. After uploading an image to S3 and initiating the conversion as batch process, several minutes can elapse before AWS reports a problem (typically having to do with the image format) that takes users back to square one. It turns out only OVF format images accepted. VMware distributions including Fusion on OSX can export to that format using the included ovftool.

With some buffer to accommodate additional software inserted by AWS into the image, log files and future updates, we end up with an Ubuntu18 server AMI that can squeeze into 3GB SSD volume. That comes out to a recurring storage expense of 30¢ per EC2 instance. There is also the storage associated with the AMI itself, one copy per region. That carries the slightly lower price-tag charged to EBS snapshots and can be amortized across multiple instances launched in the same region. Assuming the worst case scenario of a single server, we arrive at an upper bound of 50¢ per month.

Bandwidth

While computation and storage expenses are highly competitive with typical prices charged by commercial VPN services, bandwidth is one area where Amazon is much less favorable for operating 24/7 VPN. Amazon charges 9¢ per GB for outbound traffic, defined as bits heading out of AWS infrastructure in that region. A VPN server has an unusual bandwidth symmetry. Most servers receive a small amount of data inbound—such as a request for a web page— and  respond back with a large amount of data outbound, for example a high-resolution image or video stream. But a VPN server is effectively acting as proxy that shuttles traffic in both directions. Every incoming request from the client is forwarded to its true destination, becoming outbound traffic. Likewise every inbound response from that destination is sent back to the client as outbound traffic. So the total traffic volume for purpose of metering is closely approximated by the upstream and downstream traffic generated by users connected to the VPN, plus a small overhead introduced by the VPN protocol itself.

The average US household consumes 190GB of bandwidth per month according to the latest iGR report from 2017. While that report does not distinguish upstream/downstream, it is reasonable to assume downloading is responsible for the bulk of this figure. Adjusting by 50% for projected growth and including another 50GB for mobile devices with their own data plan yields a number around 300GB month of traffic. If 100% of that traffic is routed over the VPN service hosted at AWS, bandwidth costs alone would exceed all other other operational expenses by a factor of ten.

This cost scales linearly with the amount of VPN usage, or equivalently the number of users sharing the VPN. At large enough scales, the same also holds true for EC2 costs, since a single nano instance can not service an unbounded number of subscribers. But those limits are unlikely to be reached for a system intended for a handful of people, while the impact of additional bandwidth usage will be directly reflected in the monthly bill.

Verdict on self-hosting

AWS turns out to be highly competitive for hosting a VPN solution, with the caveat that only moderate usage is required. Considering that popular VPN services for consumers charge in the neighborhood of $3-$10 per month, a single nano instance can offer a better solution at lower cost, especially when the server is shared by multiple people. Advantages to self-hosting include:

• Eliminating trust in a third-party service, including random client apps. VPN provider is in a position to observe traffic metadata, such as websites visited and frequency For unenlightened websites not using HTTPS, VPN service can even observe the full traffic exchange. Unscrupulous providers have taken advantage of this, most recently Facebook with its Onavo spyware product masquerading as VPN. Hosting your own VPN avoids that dependency, although one must still implicitly trust the infrastructure provider. Amazon has visibility into EC2 traffic which can be used to identify activity: “user coming from IP address 1.2.3.4 connected to the VPN server, and that resulted in VPN server reaching out to websites at address 6.7.8.9” While one can be confident AWS (unlike Facebook) respects privacy and will not randomly mine those logs to spy on their own customers, they can still be compelled to disclose records by law enforcement.
• High-availability with the backing of Amazon infrastructure. Hardware, storage and networking failures are low probability events.
• Geodistributed points of presence, with ability to host VPN servers in the US, Europe, South America and Asia.
• Large selection of IP4 and IP6 addresses from the AWS range, compared to hosting from a residential connection.
• More importantly, freedom to change IP addresses at will. New elastic IP can be requested from AWS and assigned to an instance anytime the service operator wants to discard their association with the existing address
• Unlimited number of devices at no additional cost (within the capacity limits of the EC2 instance)

The self-hosting route also comes with two notable caveats:

• Overhead of maintaining the service, particularly the PKI necessary for issuing/revoking certificates. OpenVPN does not help matters here: it does not directly implement CRL or OCSP for retrieving revocation information. It requires the presence of a CRL file deployed locally. (But one can get the same effect by crafting a custom script to be executed server-side for each connection, to invoke openssl for proper chain validation that pays attention to CDP or OCSP responder links in the certificate. Alternatively a cron job can periodically retrieve CRLs and update the local file openvpn expects to find this information.)
• Steep pricing for bandwidth. This is less an argument against the DIY approach and more a cautionary note about using AWS for hosting, which is the obvious choice along with Azure and GCE. It turns out Azure and GCE are not that different when it comes to egress traffic. More cost-effective plans are available from alternative providers such as Linode and Digital Ocean, featuring upwards of 1 terabyte egress traffic for the fixed monthly price of server hosting.

CP

 

DIY VPN (part I)

Introduction

Net neutrality is not the only casualty of an FCC under the sway of a new administration. In a little noticed development, ISPs have free reign to spy on customers and sell information based on their Internet traffic. Google has been on a quest to encrypt all web traffic using the TLS protocol— engaging in subtle games of extortion by tweaking the default web browser UI to throw shade at sites still using the default unencrypted HTTP protocol. But there is one significant privacy limitation with TLS: it does not hide the identity of the website the user is visiting. Such metadata is still visible, both in the TLS protocol but also in the addressing scheme used for the underlying IP protocol.

This is where virtual private networks come in handy. By encapsulating all traffic through an encrypted tunnel into another connection point, they can also protect metadata from the prying eyes of surveillance-happy ISPs. But That is only the beginning: VPNs also reduce privacy risks from arguably the apex predator of online privacy— advertising-based revenue models. In the grand tradition of surveillance capitalism, every crumb of information about a customer is enlisted into building comprehensive profiles. Even if consumers never login to Facebook or Google, these networks can still build predictive profiles using alternative identifiers that help track consumers over time. Of all the stable identifiers that users carry around like unwanted, mandatory name-tags forced on them— HTTP cookies, their counterparts complements of Flash and creative browser fingerprinting techniques— IP addresses are one of the hardest to strip away. Incognito/private browsing modes allow shedding cookies and similar trackers at the end of the session, but IP addresses are assigned by internet service providers. In many cases they are surprisingly sticky: most residential broadband IPs rarely change.

From clunky enterprise technology to consumer space

In the post-Snowden era, the default meaning of “VPN” also changed. Once an obscure enterprise technology that allowed employees to connect to internal corporate networks from home, it shifted into the consumer space, pitched to privacy-conscious mainstream endusers as a way to protect themselves online. In an ironic twist, even Facebook the apex predator of online privacy infringement got into the act: the company acquired the analytics service Onavo and offered the existing Onavo VPN app to teenagers free of charge, in exchange for spying on their traffic. (When TechCrunch broke the news, Apple quickly put the kibosh on that particular privacy invasion by revoking the enterprise certificate Facebook had misused to create an app for general distribution with VPN privileges.) The result has been a competitive landscape for consumer VPN services, complete with the obligatory reviews and ranking from pundits.

For power users there is another option: hosting your own VPN service. This post chronicles experiences doing that, organized into two sections. First section an overview of what goes into building a functioning VPN server, using common, off-the-shelf open source components. Second half looks at the economics of the DIY approach, comparing the costs against commercial offerings and more importantly, looking at how different usage patterns— number of presence points, simultaneous users, monthly bandwidth consumed— affect those numbers.

Implementation considerations

“The nice thing about standards is that you have so many to choose from”

— Andrew Tannenbaum

Maze of protocols

One of the first challenges in trying to create a VPN solution is the fragmented nature of the standard. Unlike the web where all web-browsers and web-servers have converged on the protocol TLS for protecting communications, there is no comparable “universal” VPN standard with comparable ubiquity. This is either surprising or utterly predictable depending on perspective. It is surprising considering VPNs are 20+ years old. Attempts to standardize VPN protocols are the same vintage: IPSec RFCs were published in 1995. L2TP followed a few years later. With the benefit of two decades to in protocol tinkering— proprietary, standards-based or somewhere in between— you might expect the ecosystem to converge on some least common denominator by now. Instead VPNs remain highly fragmented, with incompatible options pushed by every vendor. The roots of the technology in enterprise computing may have something to do with that outcome. If Cisco, Palo Alto and Fortinet compete on selling pricy VPN gear to large companies with the assumption that every employee will be required to also install a  VPN client from the same vendor who manufactures the hardware, there is little incentive to interoperate with each other.

In this morass of proprietary blackbox protocols, OpenVPN stands out as one of the few options with some semblance of transparency. It is not tied to specific hardware, which makes for convenient DIY setups using cloud hosting: it is much easier— not to mention cost effective— to install software in a virtual server than rack a physical appliance in a datacenter cage. It also has good cross-platform support client side, with a good mixture of freeware and commercial applications on Windows, OSX and Linux flavors— the last one in particular being the black-sheep platform frequently neglected by commercial VPN offerings. (WireGuard is another more recent entry distinguished by modern cryptography, but it does not quite have the same level of implementation maturity.)

OpenVPN on iPad

OpenVPN server-side

In the interest of reach and portability, this examples uses Ubuntu Server 18 LTS, since many commercial hosting services such as AWS and Linode offer virtual hosting for this operating system. There are extensive online tutorials on setting up the server-side online so this blog post will only summarize the steps here:

1. Enable IPv4 & IPv6 forwarding via sysctl configuration
2. Setup iptables rules (again, for IPv4 & IPv6) and make them persistent using iptables-persistent package.
3. Tweak openvpn server configuration file
   • Define a range of private IPv4 and IPv6 addresses to assign clients
   • Specify DNS servers (Google DNS is a natural choice)
   • Define routes to send all traffic through the VPN
4. Provision credentials
   • Generate Diffie-Hellman parameters
   • Use openssl to generate an RSA key-pair for the server & create a certificate signing request (CSR) based on that pair
   • Submit the certificate request to a certificate authority to obtain a server certificate and install this on the server— more on this in the next section

PKI tar-pit

OpenVPN supports authentication based on different models but the most commonly used design involves digital certificates. Every user of the VPN service as well every server they connect to has a private key and associated credential uniquely identifying that entity. There are no shared secrets such as a single password known to everyone, unlike for example in the case of L2TP VPNs where all users have the same “PSK” or preshared key. One crucial advantage of using public-key credentials is easy revocation: if one user loses their device or stops using the VPN service, their certificate can be revoked without affecting anyone else. By contrast, if an entire company is sharing the same PSK for every employee, it will be painful to rotate that secret for every departure. From an operational perspective, distributing credentials is simpler. Passwords have to be synchronized on both sides in order for the server to verify them. With digital certificates, the secret private key only exists on user device and the server does not need to be informed ahead of time whenever a new certificate is issued. (Although they need access to the certificate revocation list or CRL, in order to check for invalidated credentials.)

Public-key credentials avoid many of the security headaches of passwords but come with the operational burden of managing a PKI. That is arguably the most heavy-weight aspect for spinning up a stand alone VPN infrastructure. Large enterprises have an easier time because the overhead is effectively amortized into other use-cases of PKI in their existing IT infrastructure. Chances are any Window shop at scale is already running a certificate authority to support some enterprise use case. For example employees badges can be smart-cards provisioned with a digital certificate. Or every machine can be automatically issued a certificate using a feature called auto-enroll when they are connected to the domain controller, which is then leveraged for VPN. That seamless provisioning experience is difficult to recreate cross-platform once we accept the requirement to support OSX, Linux, Android and iOS platforms. Instead credentials must be provisioned and distributed manually—and that includes both VPN users and VPN servers.

There are open-source solutions for running a certificate authority, notably EJBCA. OpenSSL itself can function as a primitive CA. OpenVPN introduced its own solution called easy-rsa, glorified wrapper scripts around OpenSSL to simplify common certificate management tasks.  But arguably the most user-friendly solution here is Windows Active Directory Certificate Services or ADCS. ADCS features both a GUI for simpler use-cases as well as automation through the command-line via old-school certutil and more modern powershell scriplets.

Active Directory Certificate Services GUI on Windows 2012

ADCS is not free software, but it is widely available as cost-effective, hosted solution from Amazon AWS and MSFT Azure among other providers. Considering the CA need only be powered on for the occasional provisioning and revocation, operational expenses will be dominated by the storage cost for the virtual disk image as opposed to the runtime of the virtual machine. One might expect a CA has to stay up 24/7 to serve revocation lists or operate an OSCP responder— and ADCS can do both of those, when the server also has IIS role installed. But OpenVPN can only parse revocation information from a local copy of the CRL. That rules out OCSP and implies CRLs must be manually synchronized to the server, defeating the point of having a server running around the clock to serve up revocation information. In effect OpenVPN only requires an offline CA.

[continued]

CP

Voting with other people’s wallets: plutocracy, blockchain style

“One person, one vote” may be the rallying cry for electoral reform, but governance for cryptocurrency— to the extent one can speak of governance with a straight face in this ecosystem— often operates on a different and decidedly plutocratic principle. Were it not for the extreme volatility of cryptocurrencies against the US dollar, the corresponding slogan would be “one dollar, one vote.” From the design of proof-of-stake algorithms that grant mining shares based on how much capital the miner has committed to the system to the design of polling mechanism that attempt to gauge community opinion on various proposals, influence is unabashedly measured as a function of account balance.

One example popularizing this notion as something of a Gallup-poll for the Ethereum community is Coinvote. Anyone can create a poll and participants vote yay or nay on the proposal, with their vote weighted by the amount of funds they hold. Voting itself does not involve any exchange of funds. Instead of participants cast their vote by sending a specially crafted message, digitally signed using the same private-key associated with their blockchain address. The signature establishes using cryptographic techniques that the voter is indeed the same person who controls the purse strings for the associated blockchain address. This system has been used for gauging community opinion on contentious questions, ranging from the EIP-999 proposal to rescue funds trapped in the now defunct Parity multi-signature wallet to more speculative questions around changing the Ethereum proof-of-work function.

Publications such as CoinTelegraph blithely cite these poll results as if they were representative of the vaguely-defined “community.” But the design is fundamentally unsound. Even if one buys into the dubious  plutocratic ideal that votes count in direct proportion to the voters bank account, Coinvote and similar polling models are fundamentally broken because they fail to account for the quirks of how funds are managed at scale.

Vote early, vote often?

Let’s start with one design subtlety that Coinvote gets right: preventing multiple votes using the same pool of funds. Votes are not counted when they are initially cast but at the end of the polling period, based on blockchain balances at a specific point-in-time. This is important because funds can move around. Imagine that Alice has a pool of ether. She can cast a vote using this pool of funds, then immediately “loan” them out to Bob— or even another blockchain address she controls herself— who casts another vote from the new address. By looking at a snapshot of the blockchain at one specific moment in time, Coinvote avoids double-counting votes from ether that gets shuffled around.

Contractually disenfranchised

The challenges with Coinvote are more complex. First problem is that by virtue of insisting on signed messages, it can not accommodate funds associated with smart contract. Recall that there are two types of Ethereum accounts: externally-owned and smart contracts. The former are what we traditionally associate with cryptocurrency. There is a private-key, carefully guarded by the owner of those funds and he/she wields control over that pool of money by cryptographically signing messages using that secret. The latter is the notion of “programmable money” pioneered by Ethereum. There is no secret associated with such an address per se, but there is a set of conditions expressed in a programming language— the “contract”— associated with the address which determine under what conditions the money can be spent.

Contracts allow expressing detailed logical restrictions around how money can be spent: for example a contract can be designed to only permit sending funds to a small number of “whitelisted” addresses, only if 2 out of 5 shareholders agree but not before March 2021 and no more than 1000 ether per day. It is precisely their expressiveness that makes contracts ideal for storing large pools of funds, with strict controls around spending conditions to prevent theft and misuse. Three out of the ten largest ethereum balances today are held in smart-contracts including the account with highest balance, storing more than 2% of all ether in existence. But that also means the holders of those funds have effectively become disenfranchised: they are ineligible to cast votes by signed messages, since there is no key to sign with.

One work around is for owners of those funds to temporarily transfer funds to a plain address, cast the vote and transfer them back. But that is an unworkable solution: moving out of the contract obviates the security controls imposed by that contract, jeopardizing the safety of those funds while they are controlled by a single key. This is no longer voting with the wallet— it’s voting by waving wads of cash in the air.

Contracts can be designed to interact with other contracts on the blockchain. In principle the problem can be solved with a better design where votes are cast by sending messages to a “ballot-box” contract on blockchain. Since contract invocations require paying mining fees or “gas” in ethereum terminology, this would have to be in addition to and not a replacement for off-chain voting with signed messages. Otherwise it amounts to a poll-tax. But that only shifts the burden to the sender side. Recall that contracts are immutable. Once a contract is published, it can not be modified, upgraded or even patched for bugs. (The illusion of upgradeability as in the Gemini dollar stable-coin contract, is achieved by using a level of indirection: an immutable “proxy” contract at a fixed address maintains a pointer to a second contract that contains the real implementation.) Suppose the Ethereum Politburo decrees a new standard for vote casting/receiving by contracts. All existing smart-contracts in use would still be incompatible with the standard since they predate its introduction, and all associated funds would still have no way to voice their opinion on polls.

Integrity of votes

Speaking of using a smart contract to count votes, there is another benefit to that approach over Coinvote: at least everyone else can verify that votes were recorded accurately. When signed messages are being submitted to a website, there is no guarantee that they will be counted fairly. The organizer could discard votes if they are trying to tip the outcome one way or another. While every vote recorded can be publicly verified— everyone can check the signature on the signed message and inspect the blockchain for the amount of money that vote speaks for— there is no easy way to prove that a vote was suppressed. Suppose a participant objects by producing a signed message they alleged to have used for voting. The organizer can fire back, arguing that it was not delivered in time before the deadline. Or the organizer may not allow the voter to change their mind: a signed message is valid forever. If the voters cast two conflicting votes, the organizer is free to retain the preferred one. We could try to work around suppression by requiring the organizer to issue a time-stamped and signed receipt for each vote cast. But that only shifts the shenanigans to an earlier: Alice voting against a proposal receives her receipt but Bob keeps running into mysterious problems with the website every time he casts a vote in favor.

Moving the vote recording mechanism onto the blockchain at least introduces some semblance of transparency. To the extent that vote suppression is occurring, it is now controlled by miners instead of a centralized organizer. As long as there is competition in mining, every vote cast has a fighting chance of making it into the permanent record of the blockchain— unless that is, the proposal is one miners are unanimous against. Consider voting on a proposal to reduce mining fees: the rational decision for all miners could be to only accept “no” votes and forgo the short-term revenue from recording “yes” votes, in favor of a strategic play to create the appearance of widespread opposition to the measure.

Voting with other wallets

There is a different problem with using blockchain transactions to cast votes: the assumption that one address speaks for one person is wrong. Custodial services— such as cryptocurrency exchanges— commonly pool customer funds into a shared, omnibus account. Typically every customer still has unique deposit addresses, as it is necessary to distinguish incoming funds to decide which customer to credit. But when customers want to withdraw funds, the transaction can originate from any address controlled by the exchange. Wallet software to tries to solve an optimization problem called unspent-output selection, which involves choosing the right combination of available funds to satisfy a particular request. If Alice deposits 1BTC into an exchange and later Bob wants to withdraw 1BTC, it is entirely possible for Alice’s deposit to be used for that request. This does not mean Alice’s funds are gone— it only means that UTXO was among those selected for the transaction initiated by Bob. What goes on under the hood is that the exchange maintains an internal ledger recording the cryptocurrency balances of every customer. Alice still has 1BTC, while Bob’s balance got debited the amount withdrawn. These ledger updates are not reflected on the blockchain: it would be too slow and wildly inefficient in transaction fees if every time a customer bought or sold bitcoin, the corresponding amount of funds had to be shuffled around on the blockchain. (At one point Bitfinex attempted to perform daily reconciliation among customer accounts in response to a CFTC consent decree, but gave up on that design after a 2016 security breach.)

The bottom line is that customers of custodial services can initiate withdrawals from addresses they do not have full control over. This has implications for voting strategies built around sending funds. For example, CarbonVote conducted a poll to gauge community opinion on the DAO bailout. Participants voted by sending a 0-ether transaction to one of two addresses, indicating a preference in favor of or against the proposed fork. A transaction with zero value is allowed in Ethereum and does not result in a decrease in the balance of the originating account. (However the sender must still pay the ethereum transaction fee in “gas” so the poll-tax criticism applies.) As before, the votes are weighted by the balance of funds in the originating address. Knowing how omnibus wallets work, it is trivial to game this poll: withdraw from your exchange account directly to one of the yes/no addresses. If the customer happens to get lucky, the transaction will originate from a very high-value address that holds funds for thousands of other customers. The resulting vote will have outsized influence, far out of proportion to the actual ether held by the person casting that vote.

It turns out those favorable circumstances do happen for many exchanges in the case of ethereum. For several exchanges, the majority of ether in hot-wallets is concentrated in a small number of blockchain addresses. The initial polls misinterpreted “votes” from those addresses, incorrectly assuming that one transaction represented the preferences of a single person/entity who controlled those funds. CarbonVote later attempted to correct this problem by disregarding some addresses known to be associated with popular exchanges.

Of course ignoring votes does not solve the underlying problem of accurately capturing the preferences of customers using custodial accounts. They can always temporarily withdraw their funds to a personal wallet to cast a vote but this is complicated by the requirement to hold the ether at that address until the poll is closed. (Presumably the customers opted for a custodial solution because they did not want to manage the risk of storing cryptocurrency directly.) Nor does it prevent a determined custodian from voting on behalf of customers: even if one well-known address is blacklisted for the purposes of counting votes, the custodian can always move funds around to another address such as their cold-storage and cast votes from there.

For a demonstration of how easily a single custodian can tip the scales, consider the EIP-999 poll. By virtue of its controversial nature, this poll received more than four times as many votes as the next most popular poll on CoinVote. Votes against the EIP edged out those in favor by a margin of approximately 630K ether. Revisiting the list of ethereum accounts with highest balances, there are a dozen individual addresses with funds exceeding that difference, almost all of them associated with exchanges. A single custodian using assets under management to vote could have tipped the scales from “no” to “yes.”

There is an analog to this problem in traditional finance: proxy voting through mutual funds. Nominally shareholders attending an annual meeting can cast their own votes on questions put before the owners, such as significant changes to the business or compensation plans. But the majority of individual investors in the US do not own shares of companies directly. They own them through a qualified investment managers such as mutual-funds and ETFs. That means the mutual fund gets to vote on behalf of its own members— a vote that may not be representative of how the investors owning shares in that fund actually feel about the issue. In fact critics have pointed out mutual funds typically prefer not to rock the boat, routinely siding with company management on share-holder proposals.

Voting with borrowed money

Weighting votes in proportion to the amount of money they speak for creates additional incentives that can distort the system. Most democratic election systems seek to discourage vote selling and buying; in the US it is a criminal offense. Vote-by-ether is an invitation to do that.

It need not even involve any complicity on the part of the seller. Suppose an investor feels strongly about shaping public opinion on a particular issue that has a running poll. They can temporarily accumulate a large ether position on exchanges, cast a vote and hold the funds until the poll is closed, liquidating the position afterwards. This is less “vote buying” and more a case of “influence buying” since the counter-parties selling ether to the investor are not deliberately trying to give away their vote. Clearly this strategy is expensive and high-risk. If ether prices move in the wrong direction, the investor can end up losing money. Even if prices remain relatively flat— a rarity in the volatile world of cryptocurrency— there are transition fees associated with moving into/out of ether. Not to mention that the act of accumulating a large ether position alone can end up moving the market, making it increasingly more expensive to accumulate more votes.

There are better strategies for gaming the system. Suppose there is a large cryptocurrency holder with no opinion one way or another on a given issue. That person can always auction off their vote to the highest bidder. Alternatively if they wanted to remain at arm’s length from direct vote selling, they can temporarily “loan” out the funds in a risk free manner. Smart contracts make this easy: the seller transfers the ether to a smart-contract that only permits two actions: withdrawals back to the seller after a deadline or 0-ether votes cast by the buyer. The buyer can vote on an unlimited number of polls while the seller is guaranteed to regain control of the funds eventually.

Final tally on blockchain polls

In conclusion then, the current crop of blockchain polls have serious design flaws ranging from disenfranchising large segments of cryptocurrency users to allowing others to easily manipulate the outcome using funds they do not even control— voting with other peoples’ wallets. Until these design issues are properly addresses, they can not be considered anything more than unscientific straw polls, unreliable as an input for important decisions affecting the cryptocurrency ecosystem.

CP

 

QuadrigaCX and the case for regulation

[Full disclosure: this blogger was formerly CSO for a regulated cryptocurrency exchange. All opinions expressed are personal.]

After QuadrigaCX

Two lines of argument predictably follow in the aftermath of any cryptocurrency business failure. First is second-guessing by pundits about their favored silver-bullet technology that could have saved the failed venture— if only they implemented multisig/hardware wallets/Lightning or reformatted all webpages to Comic Sans, this unfortunate situation would not have occurred. Second involves increased calls for regulating cryptocurrency. The aftermath of Canadian exchange QuadrigaCX is following that script so far. Quadriga has entered into bankruptcy protection, with $190M USD in customer assets allegedly stuck in their cold wallet, because the only person with access to that system has unexpectedly passed away. 

Before taking up the question of what type of regulation would have helped, there is a more fundamental question about objectives. What outcomes are we trying to achieve with regulatory oversight? Or stated in the negative, what are we trying to avoid? There are at least three compelling lines of argument in favor of regulating some— not necessarily all— institutions handling digital assets:

1. Consumer protection: Simply put, customers do not want to lose money to security breaches or fraud occurring at cryptocurrency businesses they depend on
2. Stemming negative externalities: Cryptocurrency businesses must not create new avenues for criminal activity such as money laundering while the rest of the financial industry is working to eliminate the same activity in other contexts
3. Market integrity: Pricing of digital assets must reflect their fair value assigned by the market as closely as possible, not artificially manipulated by participants with privileged access.

I. Consumer protection

This one is a no-brainer: when people entrust their money to a custodian, they expect to be able to get those funds back. Quadriga debacle represents only one possible story arc culminating in grief for customers. If their court filings are taken at face value, this incident stemmed from an “honest mistake:” the company lacked redundancy around mission-critical operations, relying entirely on exactly one person to fulfill an essential role. To put it more bluntly, it was incompetence. But that is not the only way to lose customer money. Far more common and better publicized in the cryptocurrency world have been losses due to security breaches, where systems holding digital assets were breached by outside actors and funds taken by force. Meanwhile initial coin offerings (ICOs) have been notorious for dishonest behavior by insiders, culminating in so-called exit scams where ICO organizers abscond with the funds raised, without delivering on the project.

There is no bright line between these categories. If a wallet provider is “0wned” because they failed to implement basic security standards, is that routine incompetence or does it rise to the level of gross negligence? In any case, such distinctions do not matter from a consumer perspective. Customer Bob will not feel any better about losing his savings after learning that a sophisticated a North Korean group was behind the theft. (Not entirely hypothetical, since state-sponsored North Korean attackers have been targeting bitcoin businesses, particularly in South Korea. Nor is it surprising that a rogue country would embrace bitcoin to subvert embargoes when it is cut off from the global financial system, in the same way that criminals & dark-markets embraced bitcoin as early adopters.)

That said we need to spell out he limits of what counts as “unreasonable loss.” All investing comes with risk. Notwithstanding the the tulip-bubble perception created by remarkable run-up in prices in 2017, there is no law of nature that bitcoin prices can only increase. As painful a lesson it may have been for investors who jumped on the HODL bandwagon at the wrong time, it is possible to lose money by investing in cryptocurrencies. For the most part, that type of loss can not be pinned on the cryptocurrency brokerage where the buying/selling occurred. Individual agency for investment decisions is a core assumption of free markets. But even then, there are edge-cases. Suppose an exchange decides to list an obscure altcoin whose value later drops to zero because miners abandon it. Does the exchange share in the responsibility for losses? Even if the exchange did not make patently false representations— “buy this asset, it is the next bitcoin”— customers could argue that supporting trading in that asset constitutes an implicit endorsement. For the purposes of this blog post, we will punt these questions to courts for resolution and only focus on losses due to systemic failures in security and reliability.

II. Negative externalities

A very different line of argument in favor of regulatory intervention involves society’s interest in limiting negative externalities. That interest is compelling because the market participants are not directly harmed and may not have any incentive otherwise to correct this behavior. Consider a cryptocurrency exchange favored by criminals to launder profits from illegal activity— BTCe would have been a good example until its 2017 seizure. Arguably others customers of the exchange were not affected adversely from this behavior, at least not in their role as customers per se. After all, criminals in a hurry are not price sensitive: they could be willing to dump their bitcoin at much lower prices or pay a premium to buy bitcoin at higher price than comparable venues where they would not be welcome. That means in a sense everyone involved in that particular trade is better-off: the crooks are happy because they exchanged their ill-gotten gains into a more usable currency, customers who happened to be counter-parties on the other side of that trade are happy because they got a great deal on their bitcoin and the exchange is happy because they collected commissions on the trade.

Society overall however, is not better off. Easy avenues for converting criminal proceeds into usable cash helps incentivize further criminal activity. Economic activity that looks like a win-win for everyone within the confines of a single cryptocurrency exchange can be harmful in the bigger picture when negative externalities are taken into account. To libertarian sensibilities this may represent an overreach by the state to intervene in free-market transactions between fully informed participants: the criminal trying to dump bitcoin and some willing buyer interested in acquiring those assets. But pragmatically speaking, there is significant precedent for regulating financial services based on this premise and the argument that cryptocurrency is somehow exempt from similar considerations does not pass a sanity check.

III. Market integrity

If price discovery is the central function of an exchange, it is important for those price signals to represent true supply and demand. There are many ways the signal can get out of sync from underlying market conditions. For example, in the early days when Chinese exchanges dominated bitcoin trading— before the government got wise to use of cryptocurrency for evading capital controls— there were frequent allegations that self-reported volume on those exchanges was inaccurate. While the subsequent crackdown by PRBC put a damper on volume, even as late as 2018 observers continued to find compelling evidence that leading exchanges were continuing to misrepresent trade volume.

Even in the absence of complicity by exchange operators, financial markets are subject to manipulation attempts by unscrupulous participants. Deliberately employed strategies such as wash-trading and order-spoofing can distort the price signal that emerges. That distorted view poses a challenge both to participants and third-parties. On the one hand, other customers trading on the same venue are getting a bad deal: they could end up paying too much for their cryptocurrency due to artificially induced scarcity or receive less than fair-market value when selling because of inflated supply. On the other hand, when market data is used for pricing other assets—such as derivatives in the underlying asset or shares in a cryptocurrency fund— the noise added to the signal can have far reaching consequences, affecting people who had no relationship with the exchange where the manipulation occurs.

One of the more interesting cases of alleged market manipulation in bitcoin involves neither exchanges or customers. By some accounts, the stablecoin Tether had an outsized influence on price movements for much of 2017 and 2018. Some cryptocurrency businesses can not handle fiat money—their lack of compliance programs have turned these entities into pariahs of the financial system, with no correspondent bank willing to take their business. This is where Tether comes in. A digital asset designed to have one-to-one relationship to US dollar, tether exists on a blockchains and freely moved around, bypassing traditional channels such as wire transfer. Customers who could not directly deposit US dollars into their Bitfinex account could instead purchase tethers in equivalent amount, move those tethers over and place order on the BTC/USDT order book. In isolation the use of tether is not evidence of attempt to fabricate market demand— although it is arguably a lifeline for exchanges with weak/nonexistent compliance programs, since they are precisely the companies who can not establish banking relationship required to receive fiat currency. But suppose tethers could be printed out of thin air, without receiving the requisite US dollars to back the digital assets? Then Tether (the issuing company) could fabricate “demand” for bitcoin since individuals buying bitcoin with tethers (the currency) are effectively working with free money.  The ballooning amount of tethers in circulation, combined with the opacity of the issuing company— Bloomberg referred to it as the $814 Million Mystery— and lack of independent audits have fueled speculation about whether it is operating a fractional reserve.  At least one research paper found that most of the meteoric rise in bitcoin prices could be attributed to tether issuance. That is an outsized amount of influence for a single company: by issuing less than three billion dollars worth of a digital asset— whether or not those were backed by USD deposits as promised— Tether helped propel bitcoin to a peak market capitalization over 100 times that amount. While the Tether story is not over and the presumption of innocence applies, it underlines another argument for regulation: left unchecked, rogue actors can have outsized influence in distorting the operation of digital currency markets.

CP

 

Proof of funds for cryptocurrency custody: getting by with limited trust (part VI)

[continued from part V]

Design sketch

Here is an overview of an approach that combines private proof-of-assets to trusted examiner with crowd-sourced verification of the ledger. The custodian publishes the ledger as a series of entries, one per customer:

<Pseudonym, balance commitment, range proof>

At a high level:

• Pseudonyms are unique customer IDs generated for this proof and never reused in the future.
• Balances are represented as cryptographic commitments, which hide the actual value from public eyes but allow selective disclosure
• Range proofs are non-interactive zero-knowledge proofs  demonstrating that the committed balance lies in a sane interval, such as zero to 21 million bitcoins.

Pseudonyms must be generated deterministically such that the customer can verify it can only be referring to their account. Generating a random value and emailing that to the customer will not work; if Alice and Bob have the same balance, the custodian could fool both into believing a single entry represents their balance. Likewise a pseudonym can not be derived from an opaque internal identifier only meaningful to the custodian, such as internal database IDs assigned to each customer. While database keys must be unique, customers have no visibility into that mapping and can not detect if the custodian is cheating by assigning the same ID to multiple accounts. One option that avoids these pitfalls is to compute this identifier as a cryptographic commitment to the email address. This protects the email address from public visibility while allowing selective disclosure to that customer when desired.

Speaking of commitments, a similar construction is used for representing account balances. Here the ideal commitment scheme allows doing basic arithmetic on committed values. Specifically: given commitments to two unknown numbers, we want to compute a commitment representing their— still unknown—  sum. That would be very handy in an accounting context: given commitments of individual customer balances, the custodian would be able to produce a new commitment and show that it represents the total balance across all customers. (This is similar to the notion of homomorphic encryption. For example the Paillier public-key encryption algorithm allows working with encrypted data. Given Paillier ciphertexts of two unknown numbers, anyone can craft the Paillier encryption of their sum.) Multiple options from the literature fit the bill here, going back at least two decades including Pedersen and Fujisaki-Okamoto commitment schemes.

Avoiding integer overflows, the cryptography edition

There is still a catch: regardless of the commitment scheme chosen, they all operate in modular arithmetic. There is an upper bound to the values that can be represented, even if that limit happens to be a very large number with hundreds of digits. If we try to use the additive property in a situation where the sum exceeds that limit, the result will overflow and return incorrect results— the cryptographic equivalent of an integer overflow vulnerability. These commitments that act like negative numbers. When combined valid commitments of real accounts, they will end up subtracting from the total balances.

Left unchecked, this allows custodians to cheat. It’s no consolation that such numbers will not occur naturally for real account balances: a dishonest prover can fabricate bogus ledger entries with negative balances. Since the full list of customers is not publicly known, no one will notice the spurious accounts. The imaginary customers will not show up to challenge their misrepresented balance. Meanwhile the negative values reduce the total perceived liability of the custodian because they subtract from total balances expected when summing up the commitments.

This is where the last element of the entry comes in. A range proof (such as Boudot’s result from 2000 using FO commitments) demonstrates that the committed value belongs in a sane interval, without revealing anything more about it. Such range proofs are public: it does not require any secret material to verify. Requiring positive balances reduces any incentive for the custodian to invent bogus customers. Doing so can only inflate the liabilities side of the ledger and require more cryptocurrency on assets side to pass the solvency test. Incidentally, there is a low-tech alternative to range proofs by relaxing the privacy constraint: the custodian can open every commitment for the third-party doing the examination. While the examiner still can’t tell if alice@example.com behind the pseudonym is a real customer, they can at least confirm her alleged balance is positive.

Verification

To prove the integrity of the ledger, commitments in each entry are opened privately for the specific customer associated with that entry. This involves making available to that customer all the random values used in the construction of the commitment. That could communicated via email or displayed on a web page after the customer logs into the custodian website. Armed with this information, every customer:

1. Verify their own balance is represented accurately in the ledger.
2. Rest assured that the ledger entry containing their balance is exclusive to their account. It can not be reused for other customers, because the pseudonym is uniquely tied to identity.
3. Confirm that all entries in the ledger represent positive balances. While other customer balances are hidden by commitments, the associated range proofs are publicly verifiable.
4. Calculate a commitment to total balances across the customer base

That last property achieves consensus around a single committed value for liabilities that everyone agrees on— the custodian, all customers and any independent examiner hired by the custodian. Next the custodian verifiably opens that single commitment for the benefit of the examiner, revealing total liabilities. (Alternatively, the custodian can open it publicly if there is no privacy concern about disclosing total assets under management.)

Next the custodian executes the usual proof of assets on blockchain, by demonstrating knowledge of private keys corresponding to claimed blockchain addresses. This demonstration is not public. Only the trusted examiner gets visibility into addresses. This is where trust in the independent examiner enters the picture. The proof is only convincing to the extent that the examiner is honest and competent. Honest, in that they will not make false assertions if the accounting demonstrates a discrepancy. Competent, in that they are familiar with sound methodologies for proving control over keys. (For example, they will insist on influencing challenge messages to be signed, to avoid being fooled by recycled signatures on ancient messages.) Assuming the proof is carried out to the satisfaction of the examiner, they can produce an attestation to the effect that at a specific point in time custodian assets were approximately equal to liabilities implied in the ledger. Crucially the examiner can look beyond numbers alone and assess the design of the cryptocurrency system. Does it have appropriate physical and logical access controls? Is there enough redundancy in backups? Are there key-person risks where only person can execute critical tasks— looking at you QuadrigaCX?

Summary

To recap: liabilities are verified in a distributed, public fashion with every customer able to check their own balance. This requires no external trust assumptions. Assets on the other hand are verified privately by a trusted third-party, who is given full visibility into distribution of those assets on the blockchain. Unlike BIP127 this approach covers both assets and liabilities. It does not require public disclosure of addresses, and by implication, total assets under custody. Also unlike the Coinfloor approach, individual customer balances are not revealed, not even pseudonymously or to an independent examiner. It is not limited to P2PKH addresses; arbitrary spend scripts can be accommodated. Finally it permits going beyond simple control of addresses and demonstrating higher redundancy. For example with M-of-N multisig, instead of proving control over the minimum quorum of M keys, the custodian can be held to a higher standard and required to prove possession of all N. There is still an element of trust involved in the independent examiner, but less trust required in the custodian for performing the proof. Unlike opaque audits where the examiner can not independently verify ledger integrity, publishing the ledger turns every customer into a potential examiner.

It is easy to accommodate additional requirements with changes to the protocol. For example, if we are willing to place additional trust in the independent examiner, they can also be tasked with reviewing bank statements to check presence of fiat assets. They can reviews internal policies and procedures used by the custodian, looking for red-flags such as key person risk that appears to have plagued QuadrigaCX. We can move in the other direction, reducing trust in the examiner while giving up some privacy for the custodian. Suppose we insist that the exchange publicly open the commitment to its total liabilities and publish the proof of control over keys. That would disclose all blockchain addresses used by the custodian but in return take the examiner out of the trust equation for digital assets.

CP

Proof of funds for cryptocurrency custody: revisiting trust assumptions (part V)

[continued from part IV]

Trust assumptions revisited

Purely cryptographic approaches to proving solvency are constrained by an ambitious dual mandate: achieve full public verifiability and guarantee full privacy for custodian assets. On the one hand, it aims for creating proofs that any one can verify without relying on the word of third parties. On the other hand, it protects the custodian against unnecessary of disclosure proprietary business information such as blockchain addresses or even total assets under management. The first objective is intrinsically incompatible with accommodating fiat balances, while the latter one may not provide as much value as expected.

First consider the nature of the privacy afforded to the custodian: its collection of assets— and by implication total balance— are hidden inside a larger “cover set” of other blockchain addresses chosen during the construction of the proof. Use of zero-knowledge techniques mean the distinction between real vs cover addresses will not be leaked by the proof. But there is no guarantee that outside observers will not be able to distinguish true addresses from the chaff using other signals. Many companies specialize in blockchain analytics: clustering and labeling addresses associated with specific actors and tracing flow of funds. For example, cold-wallet addresses for many exchanges are already well known. The prover could include some these in their cover set, unaware that the addresses have been deanonymized and associated with another actor. While the inclusion of such an address does not invalidate the proof, it provides no privacy benefit. Anyone looking at the proof can immediately eliminate those addresses as belonging to the custodian, leaving a smaller subset for real assets. That problem only gets worse over time: an address may not have been labeled when it was included in the cover set but future work— and new transactions involving that address— can reveal the identity behind it. The anonymity set contracts over time. In fact if Provisions-style approaches were widely deployed, participants may well undermine each other. When the same address appears in multiple proofs from different custodians, it can belong to at most one. For all others, that address can be ruled out as part of the wallet. Similarly, long-term stable addresses used by a custodian would jump out when comparing proofs over time. The cover set may change but real custodian addresses must appear in the proof as long as they carry non-zero balance.

Giving up on blockchain privacy

Another interesting datapoint is the existence of proposed and implemented proof strategies with complete transparency over blockchain addresses. For example a BIP from Blockstream proposes to use “almost-valid” transactions as proof. The custodian creates a single massive transaction consuming all of their valid unspent outputs and one bogus input. The bogus input serves two purposes. By incorporating some fresh information that can not be predicted in advance, it guarantees recency of the proof— since signatures of all the valid inputs must also reference the bogus input, they could not have been precomputed before that last input was known. Second by virtue of being an invalid input, its presence invalidates the entire transaction, preventing its unauthorized broadcast. (Recall that we want to prove control over funds at rest; we do not want to move them in the process.) As far as disclosure goes, this is radical transparency. Not only are all addresses and by implication total assets under custody revealed, the proof also gives away information about the construction of those addresses, such as the set of public-keys behind a multisig script which may never have appeared on the blockchain before.  On the other hand, the proposal makes no attempt to address the liabilities side of the equation. Knowing a custodian can sign for 1000BTC is not helpful without the additional context of how much it was supposed to have in the first place.

Coinfloor

The exchange Coinfloor uses a more comprehensive approach in its Provable Solvency Reports. They publish a pseudonymized version of the internal ledger, listing balances for every customer. These are not cryptographic commitments; actual amounts appear in cleartext. The only piece of information obscured is the identity of the account, which uses hash of API keys. Coinfloor then executes an actual bitcoin transaction moving the total amount on the blockchain. This approach reveals even more information than the Blockstream version. Individual balances are visible to everyone, not just that specific customer who is the only person in a position to verify if it is represented correctly. Pseudonyms help but can not prevent partial leaks. If the exchange is known to have customers with high balances (for example, hedge funds or accredited investors) their balances may stand out since the distribution is highly skewed. For example in the June 2018 report from Coinfloor, just ten customers account for close to 30% of all bitcoin under custody. Luckily the identifiers change over time by incorporating a timestamp in irreversible manner. Consistent identifiers would have provided even less privacy by allowing investors balances to be tracked over time, allowing others to observe which customers are building up or exiting their bitcoin positions.

Finding a middle ground

Merits of these specific proposals aside, there is a clear disconnect between the privacy and trust assumptions of solvency protocols that are deployed and those appearing in the literature. Relying on an independent third-party seems inescapable for two reasons: dealing with fiat currency balances and looking under the hood at the soundness of the overall storage system. (QuadrigaCX was in control of its keys, until one day when it was not not.) On the other hand, ledger verification is best outsourced to customers. They are both ideally positioned to check if their balance is recorded correctly and motivated to do so when their own funds are at stake. Building on this observation, we can explore other design options: proofs that are partially verifiable while still calling on trusted third-parties for specific tasks.

[continued]

CP

Proof of funds for cryptocurrency custody: public verifiability (part IV)

[continued from part III]

Zero-knowledge proofs

Borrowing on a principle from blockchain ideologues that that trusted intermediaries introduce vulnerabilities, we can instead fallback on a purely cryptographic approach to constructing a proof of solvency. The first attempt at this in the literature dates to a paper from CCS 2015 introducing a scheme called Provisions. The authors independently tackle both sides of the accounting book.

• Liabilities are publicly made available as a set of cryptographic commitments, with one commitment for the each customer balance. Commitments are private in the sense that the number is not publicly visible, but it can be selectively disclosed. Each commitment is opened only for the specific customer, for example by emailing them with the information required to verify the committed value. This part effectively crowd-sources verification, by allowing every customer to confirm that their balance is represented accurately. More importantly, the commitment scheme uses a construction that also allows committing to a total balance which is verifiably equal to the sum of individual commitments. That figure represents the total amount of customer claims to cryptocurrency stored by the customer. While the proof implies a public commitment to the sum, that number is not publicly disclosed. As such the amount of assets under custody can remain confidential as a proprietary business metric.
• Proof of assets is done using zero-knowledge proofs, over a cover set that includes the actual custodian addresses as a proper subset. The main contribution of the paper is a provably secure protocol for doing that demonstration without disclosing specific addresses or for that matter total assets. True custodian addresses are obscured among a larger cover set, which could be all addresses on the blockchain if one wanted maximum privacy. The prover uses zero-knowledge proofs over the entire set to demonstrate that the smaller subset for which she knows the private keys— in other words, custodian keys— collectively have enough bitcoin balance to exceed the liabilities committed to in the first step. Again this proof is made public for anyone interested to verify.

Falling short of the goal

At a high level, this model certainly succeeds in removing trust in a third-party accounting firm, allowing anyone including those who have no business relationship with the custodian to verify its financial health. But looking closer at the paper reveals a number of limitations:

1. Proving that liabilities were represented accurately now depends on the diligence of individual customers. This is really the same problem faced by trusted third-party accountant  in a different disguise: ledger integrity. Only this time there is a fighting chance for discrepancies to be uncovered: each customer can verify if their own balance was represented accurately. So the assurance provided by this “proof” is only as good as customer diligence in validating the ledger. It remains an article of faith that such participation will materialize. (Much like the belief that open source software must be getting more secure over time, because if there were any vulnerabilities someone, somewhere looking at it surely would have said something.) Consider that the level of effort involved is non-trivial and involves running random software to verify the commitment.
2. Limitations in the protocol, most importantly the lack of support for multi-signature addresses. This is a deal-breaker in the current incarnation: if there are few things cryptocurrency community can agree on, one of them is that the additional security and redundancy provided by multisig addresses is crucial for risk management when dealing with large amounts. This limitation affects core functionality, not just privacy. Provisions can not be used if the custodian itself is using multisig addresses, even if every other address in the cover set were P2PKH. (In other words, even if we are willing to give up privacy by exposing P2SH custodian addresses among the P2PKH decoys.)
3. Compatibility with hardware wallets. There is a more subtle implementation problem with incorporating a Provision-type design into an existing cryptocurrency storage system. The zero-knowledge proofs require using private key material in non-standard constructions. By “non-standard” we do not mean that they are dangerous or incorrect, only that these are not garden variety algorithms such as ECDSA signature, ECDH key-exchange or ECIES encryption that are widely supported. This matters because of another tried-and-true risk management strategy: using dedicated cryptographic hardware to store high value keys. Off the shelf hardware such as PCs or phones are not ideal for keeping keys due to their large attack surface. Special purpose devices such as hardware-security modules, smart cards and USB tokens are designed and validated to stringent threat models for the safekeeping of secrets. Their simplicity is an advantage for security becomes a disadvantage when contemplating fancy new protocols: these devices are deliberately not extensible. In other words, they support a fixed array of algorithms such as ECDSA and they can not be easily programmed— short of the manufacturer introducing an update— to perform arbitrary new computations using secret keys.
4. Not applicable to fiat currencies. Bank account balances are not verifiable by cryptographic techniques and will remain that way, short of all financial institutions agreeing on a new standard for digitally signed statements. That is not a problem for custodians only handling cryptocurrency. But in every other situation where fiat currency coexists with digital assets, it means the proof of solvency is incomplete. Recall the original reason why proofs are carried out independently for each asset class: to prevent the custodian from shifting balances around between currencies. Putting on the tinfoil hat, a paranoid customer could argue the custodian completed the solvency proof by temporarily using fiat deposits to purchase cryptocurrency.
   Without independent verification of fiat assets against liabilities, there is no way to disprove that line of conspiratorial thinking. (As an aside: stablecoins at first glance offer a solution. For the purpose of the proof, custodian converts its dollar assets into a stablecoin pegged one-to-one against the US dollar. Since stablecoins are tokens represented on a blockchain, the usual cryptographic proofs could then be carried out over control of private-keys in that setting. But the idea that this switch will dispense with third-party trust is illusory: the integrity of the stablecoin depends on its issuer. Transposing the problem from an old-fashioned attestation about bank statements into stablecoin balances introduces a host of new trust assumptions. Specifically that the issuer is carefully maintaining that 1:1 ratio by only minting new coins in proportion to fiat deposits and will allow timely conversion of the stablecoin back into USD on demand— just ask Tether customers how well that is working out.)

Internal controls vs externally verifiable proofs

As the state of the art advances, new protocols may well overcome some of these limitations. But there are more fundamental problems with a public cryptographic proof: possession of keys alone does not equate to sane procedures and risk management. Recent bankruptcy of the Canadian exchange QuadrigaCX is a great example of this. In court filings, company representatives identifies the root cause of $130M+ USD in customer funds being stuck: Quadriga used a single laptop for cold storage, with the password only known to the company founder who recently passed away. That is a staggering failure of internal controls: failure to identify the founder as key-person risk, failure to identify laptop as single point of failure. (There are many other ways this could have gone sideways, including disk corruption on that device.) But as long as one person at Quadriga had access to that cold wallet, they could have produced Provisions-style proofs and passed the solvency test with flying colors.

That is the intrinsic limitation of externally verifiable cryptographic proofs: they can demonstrate that the custodian controls the necessary secret keys at a point in time, but as the saying goes past performance is no guarantee of future results. By contrast a trusted-third party examination need not be limited to quantifying assets/liabilities. They can look into the actual design of cryptocurrency storage— is it a USB drive under the mat or geographically distributed HSMs? They can review policies & procedures around how funds are moved, interview select employees, ask for evidence showing that written procedures are being followed faithfully in day-to-day operations. None of that can guarantee the custodian will not experience a security breach or commit a serious blunder resulting in lost funds. But knowing an organization has consistently applied sound internal controls is at least as valuable a signal as seeing public proof that all the keys are accessible; unlike a point-in-time demonstration, organizational culture around risk management is indicative of future behavior.

The good news is that these two approaches are not mutually exclusive: we can combine third-party audits with partially-public proofs. The starting point is to revisit some of the privacy assumptions behind Provisions: from the perspective of the custodian, what are the zero-knowledge techniques trying to protect and how realistic is that expectation?

[continued]

CP

Updated 2/17: Additional comment on stablecoins.