📢 Hivemind: Connection-Pool Exhaustion Fix Released

Date: 2026-08-01
Images: steemit/hivemind:latest, steemit/hivemind:latest-c89b5f5
PR: https://github.com/steemit/hivemind/pull/375 (c89b5f5)

A new Hivemind image is out that resolves a recurring availability issue some operators may have noticed: under heavy load, the API server periodically degraded with elevated HTTP 4xx rates and had to be restarted by the load balancer. This post explains what was happening and how to upgrade.

What was wrong

Hivemind serves API requests from a fixed-size PostgreSQL connection pool (default 20, often raised to 25 in production). A single bridge.get_discussion call on a deeply nested thread could hold several connections for a long time — each depth level issues a sequential child-id lookup, and a wide/old thread can have hundreds of levels. When enough of these ran at once, the pool was exhausted.

The /health endpoint also needed a connection from that same pool. Once the pool was drained by slow discussion queries, /health could no longer acquire a connection, the load balancer marked the instance unhealthy, and it was replaced. Users saw intermittent "Internal error" responses on get_profile / get_discussion, and on steemit.com, "Sorry! This page doesn't exist." Service recovered on its own after instance replacement (typically 15–30 minutes).

The worst observed query ran for 160 seconds — far longer than any legitimate request should take.

What changed

PR #375 addresses this from four angles, all aimed at guaranteeing that no single request can monopolize the pool:

1. Per-query execution timeout (statement_timeout=30s). PostgreSQL now cancels any query that runs longer than 30 seconds and immediately returns the connection to the pool. Previously, a query could run indefinitely — the pool's acquire timeout only governs how long you wait for a free connection, not how long a query runs.

2. Isolated health-check engine. /health and /head_age now run on a dedicated single-connection engine, separate from the main pool. This means health checks stay responsive even when the main pool is fully saturated, so the load balancer no longer kills instances that are merely busy — not actually down.

3. Bounded discussion recursion. _load_discussion is now capped at MAX_DEPTH=50 depth levels and MAX_THREAD_POSTS=500 total posts. A thread that previously walked thousands of comments now stops at a safe bound. Threads larger than the cap return a partial tree (the root + first 500 posts) instead of putting the server at risk.

4. Cached hide-status lookups. Two per-request DB lookups (_get_author_hide_id, _check_posts_hide_id) are now cached for 300s. This removes two connections from every get_discussion call.

Verified

The fix was deployed to a dev environment and to production beta-hivemind and validated end to end:

• /health returns 200 with normal head-block age.
• bridge.get_discussion works correctly on real posts.
• An 850-comment thread returns 497 posts (correctly capped) in ~2s — previously this was an unbounded walk.
• With the main pool saturated by 15 concurrent heavy requests, /health still responds in well under a second and never hits the acquire timeout.

How to upgrade

Pull the new image and restart your Hivemind server container:

docker pull steemit/hivemind:latest      # or :latest-c89b5f5 to pin this build
# then restart your hive server container/process

No database migration is required. DB_VERSION is unchanged.

Notes for operators

• statement_timeout is set to 30s. If you run custom, intentionally long analytical queries against the Hivemind DB through this server, they will now fail fast after 30s. This is by design — such queries are exactly what endangered availability. For ad-hoc analytics, connect to PostgreSQL directly (the timeout only applies to connections from the API server).
• Very large threads return partial results. Discussions with more than 500 posts are truncated. This is an acceptable trade-off vs. the previous behavior of risking pool exhaustion; the root post and the first 500 comments are always returned.
• One extra DB connection per instance. The isolated health engine uses maxsize=1, so total connections go from N to N+1. Negligible in practice.
• The underlying indexes that the slow queries relied on were already present and correct — the 160s query was caused by contention, not a missing index. No schema or index changes were needed.

Thanks

Thanks to everyone who reported the intermittent errors. If you still see unexpected 4xx responses or instance restarts after upgrading, please open an issue at https://github.com/steemit/hivemind/issues with the time and the API method involved.