Engineering Challenges of a Scale-Up: Transitioning Amberdata from Product-Market Fit to a Growth Company

I love startups. They embody the essence of entrepreneurship, innovation, and disruption. It’s fascinating how a simple idea can change the world. However, transforming an idea into a successful business requires tremendous work, many iterations, persistence, and luck.

If you’ve experienced the startup journey, you know how difficult it can be to create a product or service that customers are willing to pay for. Not surprisingly, poor product-market fit remains the top reason startups fail. As a startup moves beyond its early stages, scaling and expanding operations add another significant step in its journey and introduce considerable complexity.

Amberdata is a scale-up that is going through that growth journey. Two years ago, I joined the startup as its VP of Engineering. We are building a global financial data infrastructure for digital assets and providing various products, such as enterprise data, analytics and predictive insights, research, and market intelligence.

Building data products presents unique challenges, particularly for startups that must stay ahead of their competition, move quickly, and iterate while keeping their cost in check. Anyone who has copied a few TBs of data knows how painfully slow that process can be. Data gravity makes collecting, processing, and distributing data time-consuming and expensive.

When I joined, we processed approximately $1 billion in daily notational transaction value. Today, that figure exceeds $500 billion, and our data lake has significantly expanded, growing from 120 terabytes to over 1.7 petabytes of highly compressed data. Every day, we ingest terabytes of data from centralized crypto exchanges and blockchains, offering hundreds of raw and analytical datasets.

Our primary technical objective is continuously delivering timely, accurate, and relevant digital asset data and insights. Despite doubling our clientele last year, our biggest scaling challenge from product and engineering perspectives has been managing the exponential growth in daily data intake and processing, leading to several technical challenges:

1. System reliability and scalability. Digital asset markets never sleep, and volatility can cause massive spikes in system load. Specific CEX datasets are ephemeral, meaning the services that collect that data must be up without error.

2. Big data. Processing, moving, and storing big data is costly and time-consuming - two luxuries startups don’t have enough of.

3. Data quality, integrity, and freshness. Our customers rely on our data to make critical financial decisions. Data accuracy, trustworthiness, and freshness are fundamental. Most of the work needs to happen in-stream.

4. Technical Debt. Due to the gravity of the data, the interest rate on technical debt is much higher, making it a more complex challenge.

5. Data delivery. Historical data is essential for financial modeling. Some CEX datasets are enormous, and customers want to quickly and easily retrieve these large historical, one-off datasets.

6. Real-time analytics. Real-time insights provide a competitive edge. Batching isn’t an option.

7. Compute and storage cost. Processing, storing, and serving big data is expensive. For a scale-up like Amberdata, it is crucial to manage costs and build a system where computing and storage costs stay relatively flat as more customers and data are added.

One thing is almost certain: the system that leads a startup to product-market fit won’t be scalable. Early-stage startups typically do not build systems for scalability. It wouldn’t make sense, as the primary objective is to find product-market fit. Consequently, startups accumulate technical debt. The pressing question is how much. This can become a considerable burden during growth as systems struggle to manage the increased load, requiring valuable resources to be allocated to keep things running.

Furthermore, growth is not linear. As a startup, you cannot determine when or how rapidly growth occurs. I experienced that growth challenge with my last startup, which underwent hyper-growth. Our customer base grew from 300,000 to over 3 million in about a year, and the engineering team grew from ten to over 160 engineers. Once you gain traction in the market, growth becomes unpredictable, putting significant pressure on the product and engineering teams.

While the technical challenges between my last scale-up and Amberdata differed, the overall scaling objectives were remarkably similar:

1. Align the engineering organization with the business priorities.

2. Quick delivery of new products and features and hitting deadlines.

3. Building the right products and features and not wasting cycles.

4. Reliable systems that can scale and have little technical debt.

5. A scalable product and engineering organization and ensure their high performance.

6. Efficient communication across many teams.

7. Effective system operations, observability, and a fast Mean Time to Repair (MTTR).

The path to resolving scaling challenges is rarely straightforward due to competing priorities and limited resources and time. To compete, capture market share, and expand the product offering, scale-ups frequently prioritize new features over scaling improvements.

Yet features don’t matter if they don’t meet customers' needs. Therefore, a close feedback loop is essential. Engineering cycles are expensive, and startups can’t afford to waste them. Cheap experimentation, frequent software releases, and rapid iteration are vital. Earlier-stage startups often receive ample qualitative feedback; gathering those insights automatically is crucial as you grow. Understanding product usage and costs can provide invaluable insights for adapting and creating what customers desire and avoiding waste. Those insights can then be used to prioritize ruthlessly.

Last but not least, change occurs much more rapidly in startups. Sometimes, the market demands change; other times, it’s the competition, and sometimes, you might make the wrong bets. This puts a significant strain on both teams and leadership. Everyone needs to be flexible and adaptable. Nonetheless, a company should never forget that its number one asset is its employees.

Having a clear vision and strategy is essential to counteract challenges. The vision is the north star you eventually try to reach, and the strategy is the guide to getting there, even if it changes over time.

Scaling can be challenging, but it is also gratifying. It’s easy to lose sight of how far you’ve come when you're in the thick of it. Over the last two years, the engineering organization at Amberdata has relentlessly pursued two high-level objectives:

1. Rapidly develop products that solve customers' problems, exceed SLAs, and are shipped continuously.

2. Build a scalable, high-performing product and engineering organization that pursues clear goals and thrives in an empowering culture.

We break down those goals into objectives and key results for every quarter to build towards where we want to be: excellence.

Over the next few months, we will explore the challenges we faced, break them down, and discuss how we addressed them.

Here are some topics we will cover over the next few months:

1. As Bitcoin hit $100k for the first time, we inserted over a million events per second into our real-time analytics data store.

2. We collect whole blockchains in hours and add new exchange datasets for spots, options, and futures in a week at a fraction of the cost we used to spend.

3. Most datasets can now be pulled 17 times faster via our REST endpoints at high reliability.

4. Managing Petabytes of data effectively.

5. We reduced our infrastructure cost by half while making our systems highly available.

6. Teams reliably deploy to production 8+ times/day.

Growing up is daunting. Scaling up doesn’t happen overnight, and startups will make many mistakes on the way. It takes time to expand an organization and its processes, hire the right team, change practices, and ensure reliable and scalable products and services. Many trade-offs will have to be made during that journey. The number one goal is customer delight. Every scale-up is different. It’s essential to understand your challenges and have a clear north star that is pursued relentlessly, but also be open to change if needed.

Our scale-up journey will continue for a while, and there is plenty to improve. I am proud of how far we have come as an organization. Stay tuned for the following article.