MyGlobalWebsite

Building an independent web search engine using open-source components and substantial custom-developed software.

The Original Research Question

Could an independent developer use an existing complete open-source search engine to build a practical alternative to Google or Bing?

We investigated available projects and technologies to determine whether they could provide a suitable foundation for a competitive search service.

Why We Cancelled the Evaluation

In October 2026, we decided to discontinue the comparison of existing open-source search engines.

We did not identify a complete open-source solution that we believed could satisfy our objectives without substantial additional development.

Some technologies provided useful individual components, while others had limitations involving architecture, project activity, crawling, indexing, or scalability.

Our conclusion does not mean that capable open-source search technologies do not exist. It means that none of the solutions we evaluated met our requirements as a complete foundation.

Evaluation Conclusion

The open-source search engine comparison has been discontinued.

Development of MyGlobalWebsite itself is continuing independently.

Our Successful Demonstration

The evaluation produced a functional search engine using Apache Nutch and Apache Solr, with a custom frontend written in Go.

Apache Nutch

Crawls webpages, discovers links, and retrieves content for processing.

Apache Solr

Indexes the collected documents and processes full-text search requests.

Go Frontend

Provides the search interface and presents matching results to users.

Docker & Nginx

Provide application deployment and web server infrastructure.

The system successfully retrieved webpages, indexed their contents, and returned matching search results.

We intend to retain this implementation as a demonstration of what can be accomplished using freely available software.

A working demonstration, however, is not equivalent to a large-scale search engine with extensive coverage and highly relevant results.

Developing Our Own Search Engine

Rather than continue searching for a complete existing solution, we decided to develop MyGlobalWebsite using a combination of open-source code and original software.

We will use existing libraries, scripts, and components where appropriate.

However, a significant portion of the system will consist of custom code developed specifically for MyGlobalWebsite.

Current Development Architecture

Rust Web Spider

Discovers webpages, retrieves content, follows links, and manages crawling.

PostgreSQL

Stores crawled pages, discovered URLs, domain information, and search-related data.

Go Search Backend

Will process search requests and retrieve matching data from PostgreSQL.

Rust Frontend

Will present search results and support additional result-processing algorithms.

Artificial Intelligence and Search

We are investigating AI-assisted processing of collected webpages.

Potential uses include identifying the language of a webpage, categorizing content, determining geographic relevance, and producing useful search descriptions.

These capabilities could help us build a search engine serving users across different languages and countries.

They remain development objectives, not completed features.

Search Ranking and Result Diversity

We intend to develop our own algorithms for selecting and presenting search results.

We want to investigate methods that give relevant and informative websites an opportunity to appear without relying exclusively on conventional popularity rankings.

Relevance, freshness, user interaction, and source diversity are among the factors we plan to investigate.

Documenting Our Progress

But Is It True? Lab will publish updates explaining the technologies selected, the software developed, and the problems encountered.

We intend to document successful approaches as well as unsuccessful experiments.

Components may be changed or replaced as the search engine develops.

Visit MyGlobalWebsite

Our working search engine demonstration is available at MyGlobalWebsite.

Search coverage remains limited while development continues.

Visit MyGlobalWebsite

Long-Term Objective

We recognize that competing with established search providers requires extensive infrastructure, development, and resources.

We do not claim that MyGlobalWebsite currently matches Google or Bing.

Our objective is to build a useful independent search engine, measure its actual capabilities, and share what we learn during development.

Return to Research Projects