PostHog abandons DuckDB warehouse after five months, builds custom Trino
The analytics company switched to DuckDB to escape ClickHouse scaling issues but underestimated customer workload complexity.
What to know
- PostHog's DuckDB warehouse failed at scale in five months despite initial promise; single-node architecture required massive instance sizes for real workloads, and the DuckLake extension suffered severe write performance issues (3+ minute commits).
- The company is migrating to a Trino-based system with its own lakehouse format (HogLake), but the announcement offers no proof the new architecture will succeed where DuckDB failed.
- The move raises questions about premature technology adoption and architecture selection—PostHog skipped ClickHouse due to perceived limitations, tried DuckDB without stress-testing at scale, and is now building custom infrastructure instead of optimizing existing solutions.
The dispute Whether PostHog's architecture decisions reflect pragmatic iteration or poor engineering discipline—are they learning as they scale, or repeatedly making the same mistakes of inadequate pre-migration testing? · positions read across 17 posts and comments
PostHog should have tested for scale before switching; repeated migrations signal poor planning.
-
“OK, but I think I would have tested 'will it scale?' before switching to DuckDB in the first place.”
insane_dreamer · Hacker News ↗
The post is incomplete and reads like recruitment; PostHog hasn't shown the new system actually works.
-
“The article then ends before saying if the new setup solved their problems.”
Aurornis · Hacker News ↗
PostHog should have explored improved ClickHouse or Iceberg before building custom infrastructure.
-
“I wonder if they have considered moving back to ClickHouse. ClickHouse Iceberg support has improved a lot over the past few months.”
hasyimibhar · Hacker News ↗
PostHog Analytics SaaS companyDuckDB Open-source OLAP database
How it unfolded 1 development · click the chart to see its coverage articlespostscomments
-
1
PostHog details scaling failures with DuckDB, announces migration to custom Trino
PostHog published a detailed postmortem explaining why DuckDB failed at scale. Single-node architecture required oversized instances for large workloads; the DuckLake lakehouse extension suffered from poor write performance (3+ minute commits due to full catalog re-reads); and maintaining a fork of DuckLake prevented upstream contribution. The company is migrating to a Trino derivative with HogLake, its own open table format.
“We love DuckDB. It's probably the best single-node tool for analyzing heaps of data… The same features that are great locally gave us problems in the cloud.”
— PostHog -
Sometimes a company just isn't able to manage a data warehouse; I've worked for a few.I even phone screened with this one but couldn't cover their crazy quilt of requirements; at that time they were also looking for someone to rewrite the clickhouse optimizer, and they were stuck trying to reshard their cluster, which should take an afternoon.I…
2 more of the top 3 · 17 posts in this stretch
-
H
Moving our warehouse off DuckDB Link: https:// posthog.com/blog/moving-off-du ckdb Discussion: https:// news.ycombinator.com/item?id=5 0338747
-
Things like this are a part of why I push back on people telling me we can do important things later. I don’t want to swap the engine out of a machine that’s moving that fast. There’s no time to learn about how it behaves with real versus synthetic traffic and what the failure modes are when you’re doing many thousands of transactions per…
-
-
background
PostHog rebuilds warehouse on DuckDB, abandons ClickHouse — PostHog migrated its data warehouse from a shared ClickHouse cluster to DuckDB, with each customer getting their own instance. The move was motivated by ClickHouse's inability to handle long-running modeling queries and lack of cost-based query optimization.
What people are saying 13 voices from 1 site · best of 17 · verbatim
- Did PostHog test DuckDB at scale before committing the entire warehouse to it?
- Will the custom Trino + HogLake setup actually solve the problems, or is this another temporary solution?
- Why build custom infrastructure instead of fully exploring ClickHouse's improved Iceberg support or other established distributed options?
- Nov 2
-
Listen, lad. I built this database up from nothing. When I started here, all there was was swamp. Other kings said I was daft to build a database on a swamp, but I built it all the same, just to show 'em. It sank into the swamp. So, I built a second one. That sank into the swamp. So, I built a third one. That burned down, fell over, then sank into…
-
>We rebuilt everything around our own flavor of Trino and created our own open table format, HogLake.Ah yes, making the same mistake again.
-
OK, but I think I would have tested "will it scale?" before switching to DuckDB in the first place.
-
>The article then ends before saying if the new setup solved their problems.This reads to me like a recruiting post; thats why they are open if it solved their problems. I'm not a fan of the way they framed it, it seems they have made their architecture decision and now need help implementing this.I'd imagine if you are the kind of person that…
-
> The article then ends before saying if the new setup solved their problems.They got to put ClickHouse, DuckDB and Trino on their resume & LinkedIn. Problem solved!
-
Founder here -- this is about our warehouse product which is separate from our main analytics DB. Our DuckDB warehouse was used by a handful of pilot customers. Our clickhouse database is serving all of our analytical queries.
-
Now they have their "own flavor" of Trino and a brand new open table format called "HogLake". This gives the impression of making excuses with little introspection.It would be great to see their ducklake fork, their patches and github issues so the community can learn.
-
I wonder if they have considered moving back to ClickHouse. ClickHouse Iceberg support has improved a lot over the past few months. You can easily run stateless ClickHouse on top of Iceberg. This solves the tenancy issue, and it can easily handle petabytes scale [1].[1]
-
The key insight I am getting from this is that as migrations get easier to do with agents, there will be a lot more of them.But more importantly a lot of them will happen because we will all get more trigger happy to start one. On the other hand I also don't know if at scales like PostHog, a successful migration can be tested without doing all of…
-
It reminds of people who use ORMs, never pop the hood to analyze the queries, indexes, etc - then decide it's the database that's the problem.
-
> Five months ago, we rebuilt our warehouse on DuckDB instead of ClickHoueThey switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative?The article then ends before saying if the new setup solved their problems.It’s great that we can now make these migrations with little downtime (I assume?) and…
-
> Five months ago, we rebuilt our warehouse [1] on DuckDB instead of ClickHouse.Why did they write this post if they had not even allowed things to settle. It sounds like someone wanted to claim victory (and promo?)
-
Am I missing something here, or is the only technical detail in the post "a trillion rows is a lot of data, maybe trying trinio will work this time"?It feels like a lot of churn and not a lot of measurements or engineering.