Unlocking a Decade of Knowledge: Introducing the Steem Search API by bangla.witness

in Steem Dev15 days ago

Unlocking a Decade of Knowledge: Introducing the Steem Search API by bangla.witness

86.png

The Steem blockchain is fundamentally a content-based social blogging platform, and without a doubt, its greatest asset is the content itself. Over its nearly 10+ year history, Steem has hosted an incredible amount of high-quality, informative, and creative posts. From tutorials and crypto analysis to beautiful photography and personal stories, there is a wealth of knowledge stored on this chain.

However, despite this massive accumulation of data, there has been a significant barrier for users and developers alike: there is no effective way to search and discover this older content.

Currently, we are mostly limited to seeing only the most recent posts. Countless brilliant, timeless articles remain unread simply because there hasn't been a robust search API available for the Steem blockchain. Furthermore, the built-in post search functionality on steemit.com has been disabled or non-functional for a long time.

Because of this, users and developers cannot fully utilize the vast knowledge base the Steem blockchain has accumulated over the last decade.

The Solution: A Comprehensive Search API

To solve this long-standing issue, on behalf of @bangla.witness, we are thrilled to announce the development of a powerful and comprehensive Search API!

We have successfully indexed all posts, comments, accounts, and communities on the Steem blockchain using a technology called Elasticsearch.

What is Elasticsearch and How Does It Work?

Before we dive into the API features, let's briefly talk about the engine powering it: Elasticsearch.

  • What is it? Elasticsearch is a highly scalable, open-source full-text search and analytics engine. It is designed to store, search, and analyze huge volumes of data quickly and in near real-time. It is the same technology used by major tech companies to power their search bars.
  • How does it work? Instead of searching through text directly line-by-line (which is very slow), Elasticsearch uses a structure called an "inverted index"—similar to the index/glossary at the back of a book. When we feed Steem posts into Elasticsearch, it breaks the text down into individual words and maps exactly which posts contain those words. When you search for a term, it instantly looks up this index and retrieves the relevant posts in milliseconds. This makes our new API incredibly fast, efficient, and accurate!

Exploring the Posts API

Our new API offers various endpoints to help developers and users find exactly what they are looking for. The base URL for the API is https://api.amarbangla.blog.
and all api endspoint documentation is found on https://api.amarbangla.net/doc

Here is a detailed breakdown of the posts endpoints available in our API and how they work:

1. Full-Text Post Search (/posts/search)

  • Description: This is the core search feature. It performs a powerful full-text search across post titles, bodies, and tags.
  • Method: GET
  • Parameters:
    • q (required): Your search query (e.g., bitcoin).
    • sort (optional): Sort the results by relevance (default) or newest.
    • page & limit (optional): Used for pagination to load more results (default limit 50).

2. Recent Posts Feed (/posts/recent)

  • Description: Fetches a chronological feed of the absolute latest posts published on the blockchain.
  • Method: GET
  • Parameters: page, limit.

3. Posts by Author (/posts/author/:author)

  • Description: Retrieve a paginated list of all posts published by a specific Steem author.
  • Method: GET
  • Parameters:
    • author (required): The username of the author (e.g., /author/steemitblog).
    • page, limit (optional).

4. Posts by Tag (/posts/tag/:tag)

  • Description: Fetch a feed of posts associated with a specific Steem tag.
  • Method: GET
  • Parameters:
    • tag (required): The specific tag you want to browse.
    • page, limit (optional).

5. Posts by Category (/posts/category/:category)

  • Description: Fetch a feed of posts belonging to a specific primary category.
  • Method: GET
  • Parameters:
    • category (required): The category name.
    • page, limit (optional).

6. Search Suggestions & Autocomplete (/posts/suggest)

  • Description: An incredibly useful autocomplete endpoint that fetches title prefixes. Developers can use this to show search suggestions to users as they type in a search bar!
  • Method: GET
  • Parameters:
    • q (required): The prefix search string.

7. Fetch a Single Post (/posts/:author/:permlink)

  • Description: Fetch the full data of a single, specific post using the author's name and the post's unique permlink.
  • Method: GET
  • Parameters:
    • author (required): The author's username.
    • permlink (required): The post's permlink string.

Deep Dive: Beyond Just Posts

While searching for posts is critical, a true blockchain search tool needs to cover the entire ecosystem. Our API extends far beyond just posts. Here is a breakdown of all the other powerful endpoints developers and users can now access:

Comments API

Discussions are the heart of the Steem blockchain. We have made it incredibly easy to index, search, and map comment threads.

  • 1. Fetch a Single Comment (/comments/:author/:permlink)
    • Description: Retrieve the full data for any specific comment on the blockchain.
  • 2. Search Inside Comments (/comments/search)
    • Description: Performs a full-text search specifically inside the body of comments. Perfect for finding specific discussions!
    • Parameters: q (required search term), sort (relevance or newest).
  • 3. Comments by Author (/comments/by-author/:author)
    • Description: Get a paginated list of all comments made by a specific user.
  • 4. Fetch Direct Replies (/comments/replies/:author/:permlink)
    • Description: Easily fetch all direct replies to a parent post or a specific comment.

Accounts API

Finding users and tracking account creation history is now lightning-fast.

  • 1. Single Account Metadata (/accounts/:name)
    • Description: Retrieve the full metadata profile of any Steem account.
  • 2. Account Search (/accounts/search)
    • Description: A fuzzy and prefix search tool for Steem accounts. If you only remember part of a username, this will help you find the right account instantly!
    • Parameters: q (required search term).
  • 3. Recently Created Accounts (/accounts/recent)
    • Description: Get a chronological list of the newest accounts joining the Steem blockchain.
  • 4. Accounts by Creator (/accounts/by-creator/:creator)
    • Description: List all accounts that were created by a specific Steem user/creator.
  • 5. Accounts by Creation Method (/accounts/by-source/:source_op)
    • Description: Filter accounts based on how they were created (e.g., account_create, account_create_with_delegation, create_claimed_account).

Delegations API

Tracking Steem Power delegations just got a lot easier.

  • 1. Outgoing Delegations (/delegations/outgoing/:account)
    • Description: View a list of all active vesting delegations originating from a specific account to others.
  • 2. Incoming Delegations (/delegations/incoming/:account)
    • Description: View a list of all active vesting delegations targeted at (received by) a specific account.

Top & Stats API (Analytics)

We have built custom algorithms to rank and provide analytical data on the Steem ecosystem's performance.

  • 1. Top Content (/top/content)
    • Description: Retrieves ranked posts filtered by our custom quality score model over a specific time window.
    • Parameters: window (optional: '24h', '7d', '30d', or 'all').
  • 2. Top Authors (/top/author)
    • Description: See the aggregated author quality rankings based on their post output over a specific time window.
  • 3. Top Categories & Tags (/top/category & /top/tag)
    • Description: Provides lifetime aggregated statistics tracking how frequently specific categories and tags are used across the blockchain.
  • 4. Top Communities (/top/community)
    • Description: A ranked list of Steem communities based on their total size and overall activity.

Communities API

Communities are the hubs of the Steem platform, and our API makes exploring them a breeze.

  • 1. Community Profile (/communities/:name)
    • Description: Retrieve the metadata and profile details for a specific Steem community (e.g., hive-123456).
  • 2. Community Search (/communities/search)
    • Description: Find specific communities by matching search keywords against their titles or descriptions.
  • 3. Browse All Communities (/communities/browse)
    • Description: A paginated list of all tracked communities on Steem, sorted by their creation date.
  • 4. Fetch Community Posts (/communities/:name/posts)
    • Description: Get a chronological feed of all posts published within a specific community.

System API

  • Health Check (/health)
    • Description: A simple endpoint for developers to check if the API and the underlying Elasticsearch engine are up, running, and healthy.

Beta Version & The Road Ahead

Please note that this Search API is currently in its Beta version. As with any massive database indexing project, there may be a few limitations or slight delays as we continue to sync and optimize nearly 10 years of blockchain history.

However, this is just the beginning! We have big plans for our next updates, where we will be introducing even better features, faster indexing speeds, and more advanced filtering options. We are constantly monitoring the system and tweaking our Elasticsearch configuration to ensure the best possible performance.

See It In Action: amarbangla.net

To prove the power of this API and give the community a ready-to-use tool right out of the box, we haven't just built the backend—we’ve built the frontend too!

We are proud to introduce amarbangla.net, a brand new, lightning-fast frontend built entirely on top of our custom Search API. You don't need to be a developer to experience how fast Steem can be. Head over to amarbangla.net right now to search for your favorite old posts, browse communities, and experience a modern, ultra-responsive Steem interface.


Conclusion

By providing full access to Posts, Comments, Accounts, Delegations, and detailed Analytics, we are handing developers the keys to build the next generation of Steem DApps.

Whether you want to build a lightweight blogging front-end, a community analytics dashboard, or a specialized search engine, the bangla.witness Search API has everything you need.

We are incredibly excited to see what the community builds using this tool. Let's unlock the true potential of the Steem blockchain together!

If you appreciate our ongoing development for the Steem ecosystem, please show your support by voting for @bangla.witness as your Steem Witness!

Sort:  

Really impressive work — a full Elasticsearch-powered search layer covering posts, comments, accounts, delegations, and communities is exactly what Steem has been missing for years.
Quick question for @bangla.witness: is this open source? Would love to see a public repo where devs could PR in new endpoints and dig deeper into specific use cases — the possibilities here feel genuinely infinite (trending detection, cross-account analytics, custom filters, etc.). Curious what other devs here would want to build on top of this too! E.g, json_metadata covers various possibilities to link in advanced features.

yes, this would be open source within 2 weeks.

 15 days ago 

Very Useful for Botting !!

thanks for your feedback.

great work !!!

https://chat.amarbangla.net/ says I'm busy.

https://chat.amarbangla.net/ now , working,
thanks for your feedback.

Great! 👍
In frontend markdown is not rendering properly.

 14 days ago (edited)

thanks for your feedback.

In frontend markdown is not rendering properly.

i will contact with you personally, need your help.

Ok. Anytime