How Coinbase became a leader in bitcoin transactions with the help of AWS (Amazon Web Sevices)?
About Coinbase:
Cryptocurrencies like Bitcoin have seen a blizzard of headlines over the past few years. These digital tokens share some of the qualities of hard currency and can be bought, traded and spent. In fact, an entire market has grown around the trading of digital currencies, with investors and speculators keeping close tabs on every fluctuation.
At the center is Coinbase, a growing bitcoin wallet and exchange service headquartered in San Francisco, is the largest consumer bitcoin wallet in the world and the first regulated bitcoin exchange in the United States. Bitcoin is a form of digital currency that is created and stored electronically. The company, which supports 3 million global users, facilitates bitcoin transactions in 190 countries and exchanges between bitcoin and flat currencies in 26 countries. In addition to its wallet and exchange services, Coinbase offers an API that developers and merchants can use to build applications and accept bitcoin payments.
Challenges faced by Coinbase before AWS:
1. Security:
Security is the most important of those tenets, according to Rob Witoff ( the director at coinbase) . “We control hundreds of millions of dollars of bitcoin for our customers, placing us among the largest reserves in our industry,” says Witoff . “Just as a traditional bank would heavily guard its customers’ assets inside a physical bank vault, we take the same or greater precautions with our servers.”
2. Scalability:
Scalability is also critical because Coinbase needs to be able to elastically scale its services globally without consuming precious engineering resources. “As a startup, we’re meticulous about where we invest our time,” says Witoff . “We want to focus on how our customers interact with our product and the services we’re offering. We don’t want to reinvent solutions to already-solved foundational infrastructure.” Coinbase also strives to give its developers more time to focus on innovation,” says Witoff .
3. Stability:
To support its goals, Coinbase decided to deploy its new bitcoin exchange in the cloud. “When I joined Coinbase in 2014, the company was bootstrapped by quite a few third-party hosting providers,” says Witoff . “But because we’re managing actual value and real assets on our machines, we needed to have complete control over our environment.”
4. Big data analytics:
Coinbase sought a better data analytics solution. “We generate massive amounts of data from the top to the bottom of our infrastructure that would traditionally be stored in a remote and dated warehouse. But we’ve increasingly focused on adopting new technologies without losing a reliable, trusted core,” says Witoff . “At the same time, we wanted the best possible real-time insight into how our services are running.”
How AWS helped Coinbase?
Coinbase evaluated different cloud technology vendors in late 2014, but it was most confident in Amazon Web Services (AWS). In his previous role at NASA’s Jet Propulsion Laboratory, Witoff gained experience running secure and sensitive workloads on AWS. Based on this, Witoff says he “came to trust a properly designed AWS cloud.”
The company began designing the new Coinbase Exchange by using AWS Identity and Access Management(IAM), which securely controls access to AWS services. “Cloud computing provides an API for everything, including accidentally destroying the company,” says Witoff . “We think security and identity and access management done correctly can empower our engineers to focus on products within clear and trusted walls, and that’s why we implemented an auditable self-service security foundation with AWS IAM.” The exchange runs inside the Coinbase production environment on AWS, powered by a custom-built transactional data engine alongside Amazon Relational Database Service (Amazon RDS) instances and PostgreSQL databases. Amazon Elastic Computer Cloud(Amazon EC2) instances also power the exchange.
The organization provides reliable delivery of its wallet and exchange to global customers by distributing its applications natively across multiple AWS Availability Zones.
Coinbase created a streaming data insight pipeline in AWS, with real-time exchange analytics processed by an Amazon Kinesis managed big-data processing service. “All of our operations analytics are piped into Kinesis in real time and then sent to our analytics engine so engineers can search, query, and find trends from the data,” Witoff says. “We also take that data from Kinesis into a separate disaster recovery environment.” Coinbase also integrates the insight pipeline with AWS CloudTrail log files, which are sent to Amazon Simple Storage Service (Amazon S3) buckets, then to the AWS Lambda compute service, and on to Kinesis containers based on Docker images. This gives Coinbase complete, transparent, and indexed audit logs across its entire IT environment.
Every day, 1 TB of data — about 1 billion events — flows through that path. “Whenever our security groups or network access controls are modified, we see alerts in real time, so we get full insight into everything happening across the exchange,” says Witoff . For additional big-data insight, Coinbase uses Amazon Elastic MapReduce (Amazon EMR), a web service that uses the Hadoop open-source framework to process data, and Amazon Redshift, a managed petabyte-scale data warehouse. “We use Amazon EMR to crunch our growing databases into structured, actionable Redshift data that tells us how our company is performing and where to steer our ship next,” says Witoff .
All of the company’s networks are designed, built, and maintained through AWS CloudFormation templates. “This gives us the luxury of version-controlling our network, and it allows for seamless, exact network duplication for on-demand development and staging environments,” says Witoff . Coinbase also uses Amazon Virtual Private Cloud (Amazon VPC) endpoints to optimize throughput to Amazon S3, and Amazon WorkSpaces to provision cloud-based desktops for global workers. “As we scale our services around the world, we also scale our team. We rely on Amazon WorkSpaces for on-demand access by our contractors to appropriate slices of our network,” Witoff says.
Coinbase launched the U.S. Coinbase Exchange on AWS in February 2015, and recently expanded to serve European users.
The benefits Coinbase received using AWS:
Coinbase is able to securely store its customer’s funds using AWS. “I consider Amazon’s cloud to be our own private cloud, and when we deploy something there, I trust that my staff and administrators are the only people who have access to those assets,” says Witoff . “Also, securely storing bitcoin remains a major focus area for us that has helped us gain the trust of consumers across the world. Rather than spending our resources replicating and securing a new data center with solved challenges, AWS has allowed us to hone in on one of our core competencies: securely storing private keys.”
“In three years, our bitcoin wallet base has grown from zero to more than 3 million. We’ve been able to drive that growth by providing a fast, global wallet service, which would not be possible without AWS,”
Rob Witoff
Director, Coinbase
Additionally, the company has better visibility into its business with its insight pipeline. “Using Kinesis for our insight pipeline, we can provide analytical insights to our engineering team without forcing them to jump through complex hoops to traverse our information,” says Witoff . “They can use the pipeline to easily view all the metadata about how the Coinbase Exchange is performing.” And because Kinesis provides a one-to-many analytics delivery method, Coinbase can collect metrics in its primary database as well as through new, experimental data stores. “As a result, we can keep up to speed with the latest, greatest, most exciting tools in the data science and data analytics space without having to take undue risk on unproven technologies,” says Witoff .
As a startup company that built its bitcoin exchange in the cloud from day one, Coinbase has more agility than it would have had if it created the exchange internally. “By starting with the cloud at our core, we’ve been able to move fast where others dread,” says Witoff . “Evolving our network topology, scaling across the globe, and deploying new services are never more than a few actions away. This empowers us to spend more time thinking about what we want to do instead of what we’re able to do.” That agility is helping Coinbase meet the demands of fast business growth. “Our exchange is in hyper-growth mode, and we’re in the process of scaling it all across the world,” says Witoff . “For each new country we bring on board, we are able to scale geographically and at the touch of a button launch more machines to support more users.”
“Machine learning helps us balance risks for Coinbase, with flexibility for customers where we want them to have the best experience possible.”
Soups Ranjan
Director of Data Science
Coinbase
However, since Coinbase operates in a highly regulated environment, the company takes extra measures to ensure customer data is protected — even from its own data scientists and engineers. Any code that runs on Coinbase production servers has been code reviewed and looked at by multiple sets of people before it goes into production. “One of our core tenets is that we are a security-first company because we are storing cryptocurrencies on behalf of our customers,” says Ranjan.
At the end of the day, digital cryptocurrencies rely on trust for their existence. And companies like Coinbase rely on AWS to build and maintain that trust by working to constantly stay ahead of risks.
Adoption of new Architecture using AWS Step Functions to Securely Deploy to AWS in Seconds:
Reports of cryptocurrency thefts to the FBI’s Internet Crime Complaint Center (IC3) totaled $182 million in 2018, a 212 percent increase over 2017 and likely just a fraction of the actual volume of worldwide cryptocurrency losses to fraud. With an aim of operating as the trusted, safe, and legal center of the crypto-economy, Coinbase strives to be world class at security, compliance, technology, customer support, design, and more. This is why Coinbase has used Amazon Web Services (AWS) as its primary infrastructure provider since 2015 — and why it recently incorporated additional AWS technologies as part of improving its software deployment processes.
In his search for ways to further strengthen security at Coinbase, Jenson identified an opportunity to improve the company’s automated deployment pipelines. “We had multiple deployers, each with different interfaces and complexities,” says Jenson. “I wanted a common framework that would enable us to rapidly build deployers that could validate user input, securely release code to AWS, and stay out of the way of our engineers.”
In the course of researching solutions, Jenson learned of AWS Step Functions. “As soon as I reviewed the documentation, I realized that Step Functions was exactly what I needed,” says Jenson. “Step Functions can maintain state for up to a year, is highly scalable, and makes it easy to describe how to automatically handle and retry after specific errors.”
Choosing to build its new class of deployers with a framework based on AWS Lambda and AWS Step Functions — and using AWS Identity and Access Management (AWS IAM) and Amazon Simple Storage Service (Amazon S3) — put Coinbase on a fast path to implementation, with only a brief initial learning curve. The first deployer Jenson’s team built, an open-source AWS deployer called Odin, takes a description of a project release and safely and securely launches it into AWS using Amazon Elastic Compute Cloud (Amazon EC2) Auto Scaling groups.
“From idea to working implementation to migrating Odin onto the solution — all told, it was six months from conception to production,” says Jenson. “But because we can reuse the code and framework Odin is built on, it took just a few weeks to put our next two deployers into production. It’s only going to get faster from here on out.”
“We have seen a significant reduction in trouble tickets. This is due to the visibility that AWS Step Functions gives our engineers.”
Graham Jenson, Senior Infrastructure Engineer, Coinbase
Simplified Architecture with AWS Step Functions:
The new approach substantially reduced the complexity of Coinbase’s architecture, which in turn improves visibility for Jenson’s team.
“Our previous deployers all had different web hooks, callbacks, Amazon S3 layouts, buckets, and AWS IAM roles and used different means of communicating and polling. It was really difficult to get the visibility we needed,” says Jenson. “Now, with all of our deployers based on the same AWS Lambda and AWS Step Functions foundation, we operate and interact with them all the same way. We can actually watch the data flow through the Step Function, identify failures along particular paths, and take action to fix them.”
This simplicity speeds the process of adding AWS accounts and improves security. “Using AWS Lambda with the AWS IAM assume role, we can onboard an AWS account with a single AWS IAM role, as opposed to an entire service with its own individual configuration,” says Jenson. “With AWS Lambda and AWS IAM, we reduced the time needed to add new AWS accounts from days to seconds.”
The new solution also simplifies auditability. “We can enable multiple accounts going through a single Step Function, which gives us a single audit trail for all deployments,” says Jenson. “That makes it easy to understand what happened in all of the accounts and lets us enable new accounts with high security without having to re-implement the audit trail.”
By empowering engineers to overcome obstacles independently, the solution is also reducing demands on the infrastructure team. “We have seen a significant reduction in trouble tickets about failed deploys,” says Jenson. “This is due to the visibility that AWS Step Functions gives our engineers to let them diagnose and resolve their own issues.”
What all of these internal, technical benefits add up to is stronger security and faster response to customer requests. “With deployers built on AWS Step Functions and AWS Lambda, our engineers can move code into production safely,” says Jenson. “The upshot is that we can release new features more often, respond quicker to security threats, and more easily achieve our SLAs. This adds up to an even better, more secure, customer experience.”
AWS services used by Coinbase:
1. Amazon Step Functions:
AWS Step Functions is a serverless function orchestrator that makes it easy to sequence AWS Lambda functions and multiple AWS services into business-critical applications. Through its visual interface, you can create and run a series of checkpointed and event-driven workflows that maintain the application state. The output of one step acts as an input to the next. Each step in your application executes in order, as defined by your business logic.
2. Amazon S3:
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.
3. AWS Lambda:
AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume — there is no charge when your code is not running.
4. AWS Identity and Access Management:
AWS Identity and Access Management (IAM) enables you to manage access to AWS services and resources securely.
I hope you get some good insights of the AWS-coinbase casestudy.
THANK YOU FOR READING!!
A special thanks to World Record Holder, Vimal Daga sir for his extraordinary teaching skills and to provide such a platform where we can develop ourselves and learn new technologies, their integration, etc. from his years of hard work and research. I consider myself really lucky to be a part of his trainings where I get to improve myself and learn new & exciting things every day.
