For nearly three years, I deployed code to what is arguably the most adversarial production environment in the tech industry: public blockchains. I wrote code for projects managing anywhere from $1M to a peak of $20B+. When over $11B has been lost as a result of hacks and exploits, I don’t think it’s unfair to call it a cutthroat environment. With the stakes this high, I saw developers and companies forced to optimize for very different variables than what we see in traditional software development. This post is a retrospective on my relatively brief stint in the space, both from technical and later more philosophical perspectives.

“You can sleep well when the markets close…”

…which, in the blockchain industry, is never.

This was one of the first things I was told by a mentor when I started working in the space — somewhat tongue-in-cheek, but as I would quickly learn, very true. This was what really knocked into me that the space forces you to optimize for different factors than traditional software development (in theory, at least!). The environment is always “live”, things are rarely revertable, and the stakes are always high. My #1 maxim approaching development, both as a developer, project lead, and later as a founder, was to optimize for a good night’s sleep.

Unfortunately, I’ve experienced this firsthand in the most nerve-wracking way possible, noticing an active exploit in a margin trading system at 1AM, leading to a several-hour impromptu bug bash, tracing down the faulty line of code, whilst frantically trying to raise the (majority Europe-based) team to pause the markets and deploy a fix.

Writing traditional software comes with well-known failure modes, and well-known ways to avoid them. Unfortunately, as is all-too-common in the startup space, “move fast and break things” does, surprisingly, tend to break things from time to time. Breaking things is simply not an option. A minor exploit like the margin trading system cost hundreds of thousands of dollars in lost funds. Here’s a (not too) short list of the software engineering practices I found to be most effective.

Tools Should Protect You From Yourself

A good majority of aviation accidents trace back to human error. This is even more so true in the software industry. Picking tools that make it impossible for entire classes of bugs to manifest is crucial. Ironically, a common theme here is reducing the surface area that you, a human, can have an impact on. While this does imply tools like static analyzers, sanitizers, and various linters, it’s the more foundational choices that end up mattering most. In the blockchain world, this means picking a programming language that’s designed to be safe, and a framework that minimizes the risk of common pitfalls.

And here’s where the frustrations begin. Solidity, the most used smart contract DSL, also happens to have an insane number of footguns, unintuitive hidden behaviors, and misleading syntax. Not to mention reentrancy, a bug that is solely responsible for billions lost. No “future of finance” should be built on such poor foundations, and having been acquainted with the space for so long, it’s hard to put any trust in all but the most battle-proven codebases.

Reentrancy attacks apply to reentrant functions: Where a function calls another function, which may call back into the original function (A() -> B() -> A() function patterns). If a reentrant function has improper state management, it can result in incorrect behavior:

function withdraw(amount) {
	require(balance[msg.sender] >= amount);
	send(msg.sender, amount); // send() invokes a callback on msg.sender
	balance[msg.sender] -= amount; // This line is executed *after* the call
}

function callback(...) {
	if (depth < 10) withdraw(amount); // Calls withdraw() reentrantly
}

In this example, send triggers a malicous function (here: callback) before decrementing the balance. If send is called again, the require() check continues to succeed, allowing more funds to be withdrawn than intended.

Rust is frequently cited as a great alternative, designed with many safety guarantees, and appearing in many frameworks. Leveraging a compiler to automatically check and rid your code of nontrivial classes of bugs is a massive advantage. My favorite example is encoding higher-level information into the type system, such as U128<USD> over U128 to represent currencies. New(er) frameworks like CosmWasm completely eliminate reentrancy and other common pitfalls by design, again, making it impossible for a stupid human to make a mistake.

Perhaps most interestingly, the Blockchain boom has indirectly led to a renaissance in formal verification, a field that has been around for decades, but never really got the attention and investment it deserved until relatively recently. Formal Verification can help be the “final boss” of lifting the human factor out of the equation, exhaustively proving that your code behaves as intended. I’ll personally be keeping an eye out for continued progress in formal verification tools/languages and their application to the software industry at large.

Verification tools have already seen some uptake in certain projects, primarily in the theoretical distributed systems space, but also notably at large tech companies such as Amazon.

Safety-Critical Code is Nuclear Waste

This is obvious: The most secure code is no code at all. The less code you have to deal with, the less surface area there is for bugs to manifest. In the blockchain world, the most security-critical code is that which lives on the blockchain. Critical code here should prioritize simplicity above all else, with any extra functionality moved out of the danger-zone. Whilst tempting to bundle extra functionality, this separation of concerns is a great way to minimize the risk of bugs, or at least, bugs with potentially catastrophic consequences. Here’s where things like the UNIX philosophy come into play. It’s fundamentally easier to reason about, test, and verify the API design and implementation of single-purpose modules. In blockchain, another (more tangible) consequence of minimizing the critical code mass is that it makes reviews and audits easier and less time consuming (and thus, less expensive!).

Speaking of audits — I found the audit industry built around blockchain to be extremely predatory, often with very little value provided. Audits can cost upwards of $100k, and from personal experience, fail to find issues that later manifested in production. In fact, given the highly pseudonymous and adversarial environment, some folks I’ve worked with suspected auditors of malpractice, intentionally cataloguing bugs, or using privileged codebase access to later exploit code that they had audited. Not a great look, especially when auditors push the narrative that their audits are the only thing protecting end-users.

When handling safety-critical code, adopting a safety-first code style is also something that can pay dividends. This is something I struggled with, but ultimately found to be effective for the future version of myself who was figuring out what the hell past-me was thinking. Be boring, be explicit, be defensive, be consistent, and be paranoid. I highly recommend TigerBeetle’s excellent TigerStyle guide or NASA’s Power of Ten, both of which outline great safety-first coding practices.

Words scrounged from a note I wrote to myself who-knows-when:

Be boring. Your code should never be clever and concise at the sake of simplicity.

Be explicit. Hidden logic — abstractions, operator overloading, implicit behaviors, or inheritance — can hide dangerous footguns.

Be defensive. Constantly validate your assumptions. Assume that any data you receive or load could be malformed or malicious, even if you wrote it yourself.

Be consistent. Write similar logic the same way every time. If something feels wildly new, maybe it isn’t a good idea. Don’t force poor abstractions into multiple contexts.

Be paranoid. Write tests traversing every conceivable code path. Assume edge cases and worst-case runtimes as the common case, and test well beyond your expected operational envelope.

Shortcuts Are Never Worth It

Finally, I’ve seen and experienced the strong temptation to take shortcuts in the name of shipping fast. From services monitoring and managing $1B+ being hosted on relatively insecure AWS boxes, to pushing unreviewed code to master, to bus-factor = one key management and deployment practices — shortcuts like these are the slipperiest slopes of all. I’ve flagged slack messages containing secrets that should’ve been protecting hundreds of thousands; experienced well-timed exploits, designed to land when the entire team was asleep; and even retained access to sensitive production systems for months after leaving the company.

Lax security practices are never ideal, especially so when nation-state actors like North Korea’s Lazarus Group actively target blockchain developers. In fact, I’ve seen firsthand whilst interacting with a developer for an external team with suspiciously bad skills, broken English, and who ended up deploying and exploiting a backdoor. I myself have been guilty of becoming complacent and impatient with long-winded development/deployment practices. Unfortunately, the surface area of any project grows with each person working behind it. Supply chain attacks extend to the human supply chain. When your company is globally distributed, hiring a bad actor is a real and everpresent risk that further erodes my trust in the space.

A Retrospective

For two years I fell in love with the blockchain space, not necessarily for the products, but for the absolutely fascinating technology, much of which is fresh off the research presses. I loved the challenges of building in a space where so many foundational blocks are still being questioned, where any contribution could have a massive impact, and where there’s a fantastic community of extremely talented and passionate engineers.

Don’t get me wrong, I believe that there’s a lot of extremely talented people continuing to work towards the holy grail of a secure, scalable, and open financial system to replace the current one. In fact, there’s been a lot of progress in doing so! There are some incredibly powerful things that can be established when you have an open source financial ecosystem, with strong and safe primitives built and ready for composition and integration into a combinatorial explosion of new products and services. The fact that there’s things that make me grind my teeth in the traditional finance space, making me wish I was using a blockchain-based alternative, is a testament to the potential of the technology.

Ultimately, at least at the moment, it seems like the lay of the land is solidly against the status quo of the blockchain industry. Rampant value extraction and fraud from the recent memecoin craze (going as far up the ladder as the sitting POTUS!), the continued delegitimization of the space by constant exploits, fraud, and scams, as well as the lack of truly safe security practices, from user experience to development practices — altogether, these have continued to fuel my skepticism and were the primary motivators behind my decision to step away from the space.

An aside on UX: It’s mind-boggling that one misstep still results in a literal game-over scenario! There’s so much friction to using blockchain wallets, and yet they’re still so dangerous to work with.

As-is, blockchain systems simply won’t scale to the level of trust required for a better financial system (let’s forget about the technical consequences of using an strictly ordered blockchain at global scale), and no (western?) government seems committed to establishing a regulatory framework that would allow the space to mature and grow in a healthy way. I’ve seen legitimate projects backed by extremely well-meaning and talented people fail, or move to foreign countries, simply because no legal framework exists to allow them to operate safely. Meanwhile the insider-trading rampant, fraudulent, and frankly useless memecoin (and earlier, the NFT scene) thrives without consequence.

There are a lot of ideas, both technical and financial, that I think we could stand to learn from the blockchain space, and I hope that a niche of the space continues to grow and thrive. There’s some seriously powerful technology that has already come from blockchain development, forged by the unique challenges and pressures of the space (in Cryptography, Distributed Systems, Languages/Compilers, etc.), and I would love to continue seeing it applied to other domains.

My advice to anyone that’s in the industry right now? Take a step back. Are you doing something that truly is making the world better? Or are you simply a hospice nurse for systems saddled with technological, financial, and social debt? I think you’ll find that you already know the answer deep in your heart, whether or not in the affirmative. I absolutely admire the work that many of my old colleagues and friends are doing, but for me? It’s time to move on.