Ontonathon: Our Unfiltered Failure Log

Velocity is one of our core values at Onton. This week, we're stress-testing it.

4 days. As many features as we can ship. Because of the time constraint, corners will be cut and mistakes will happen. So we're documenting all of it.

Why? Because the sanitized version of building software is boring and unhelpful. The real learning is in the chaos.

Follow along. Updates daily.

Day 1

Goal: ship two major features in two days. 1) Saved searches with async multi-search support and 2) a chat interface for product Q&A and comparisons.

The Wins

• Reranking worked on first try. Julia service and Clojure code developed separately just... worked together. Magic.

• Aditri set up her first Julia service (future sprints will be faster)

• UI framework made some components surprisingly easy e.g. search bar reuse

• Saved search results feature only took 15 minutes

• Team division of labor was solid

The Fails

• What seemed "doable at first glance" turned into a paradigm shift (supporting multiple concurrent searches vs. one at a time)

• Planning ate the day. Design decisions and architecture took way longer than actually writing code

• GitHub broke (because of course it did)

• Sophia's dev container died, lost an hour to restart

• Chat API setup was a mess - so confusing we didn't think we'd ship in time. Gemma figured it out by evening.

• Simple SQL query turned into 30 lines of code

• Frontend AND backend both needed changes for something we thought was frontend-only (search history)

Biggest Lesson

We're better at dividing work than at deciding what work to do. Should've shipped a rougher v0 faster instead of trying to design the perfect solution upfront. Plan less, ship more.

Day 2

We launched! Got both features live. And then everything broke.

The Wins

• Chat interface launched and actually worked in production

• Fixed the search disaster in 1.5 hours

• Both features are live. Imperfect, but live.

The Fails

• New service deployed but didn't start up properly in prod. Worked perfectly locally.

• Search broke completely when we pushed to prod. Frontend and backend were tested individually and both worked fine. Integration? Not so much.

• Dev environment for search was a pain to set up - needed test data and indexing processes running before we could even test locally. Hit unexpected issues along the way.

• Chat session management needs work - figuring out how to handle context between conversations

• Data format conversion issues caused bugs we didn't anticipate

Biggest Lesson

We would have caught it if we had a working dev environment. The search bug that killed 1.5 hours would've been obvious if we'd prioritized dev setup earlier. Testing components individually doesn't mean you've tested integration.

Day 3

With launch behind us, we're improving performance, fixing bugs, and making things actually feel good to use.

The Wins

• Improved search reranking performance by sending requests in parallel

• Fixed broken product pages - you can click on them again!

• Text search is working locally now

• Made progress on loading screen animation with product colors

• Identified and started fixing the similar products bug

The Fails

• Tried to build Docker image for a new, more powerful GPU. Python version was incompatible with the hardware. Now rebuilding everything with updated requirements.

• Moving to a more powerful GPU somehow made things worse. The irony...

• Similar products completely broke. Image search uses websockets differently than text search, and frontend rehydration loses the backend ordering.

• Search result ranking quality is off after implementing saved searches

• Loading screen animation is finicky - trying to make product images animate in a "random but intentional" way that feels whimsical. Lots of timing issues. Cursor keeps breaking its own code.

Biggest Lesson

More powerful hardware does not mean better results. When you change one thing (GPU, Python version, search implementation), expect everything downstream to break in unexpected ways. Latency optimization is extremely gnarly. So many variables to control for, and it's incredibly sensitive to small changes.

Day 4

The final push.

The Wins

• Search reranking went from 4-7 seconds down to 0.3 seconds. That's 13-23x faster. Expected maybe 2x improvement, got 5-8x better than that.

• Finally got new Python versions working on our GPUs after days of fighting with it

• Animation launched at 11:58pm (technically made the deadline)

• Made progress on lazy loading - hydrating images as you scroll instead of all at once.

• Fixed the similar items bug on product pages that broke earlier

The Fails

• The 0.3 second reranking approach only came together at midnight. So many complications and dead ends before getting there.

• Benchmarking test for latency didn't reflect prod - test image was way smaller than real images.

• Animation has jankiness - full screen pop-up happening afterward. Either async search is retriggering it, or the animation itself is buggy.

• Images look flickery on mobile

• Animation doesn't fully cover bigger screens yet

• Potentially broke mobile search

• Dev environments kept crashing. Sophia couldn't perform searches in her own branch. Had to push code up for Gemma to test. Flying blind is not an effective developer experience.

• Image search doesn't work in dev environments right now

Biggest Lesson

If you feel like there's a better way, fight for it. Alex and Aditri had a working 1.5-2.5 second solution but pushed for better and got to 0.3 seconds. Also, you can't ship effectively without stable dev environments. When you can't test your own changes locally, you're flying blind. Context switching to test for each other kills momentum, especially during crunch time.

Stay Tuned

That's a wrap on the Ontonathon. But this is just a small taste of it all. We're shipping a lot more in the coming weeks and months - new features, improvements, and probably a few more things that will break spectacularly before they work beautifully. To stay up to date, follow us on Instagram, LinkedIn, X, or join our Discord community.