Testing my own web research tool against a coding agent's built-in one
I built a web research tool for pages my coding agents couldn't read. Tested blind, it tied the built-in one and was slower, but read the hard pages better.
In August I built my own web research tool for coding agents. Agents like GitHub Copilot CLI come with a built-in fetch, the tool that downloads a web page so the agent can read it. On some sites it gets next to nothing. In a test that month, an agent’s built-in fetch got an error back from Reddit, Quora and Stack Overflow, while a real browser got text from all three. So my tool reads pages in a real browser.
In September I tested whether it was worth keeping, against the research built into Copilot CLI. On general research questions, scored blind, the answers tied and mine was slower on every question. What kept my tool was the second half of the test, on the kind of page I built it for.
Two ways to look something up
Both tools answer a question the same way. They search the web, read some of the pages they find, and write an answer that cites only pages they actually read. A search result’s snippet doesn’t count.
Mine searches through SearXNG, an open-source search engine I host at home that asks other search engines and merges what they return. It opens the pages it picks in a real browser and saves their text. Many sites that turn away a plain download still let a real browser in.
Copilot’s built-in research uses its own search and its built-in fetch. It can also call some sites’ APIs, like GitHub’s, to read code without loading web pages.
A first comparison that wasn’t fair
To decide whether to keep it, I first ran both tools on four questions. Copilot’s answers came out more complete on all four, and mine was faster on three.
That test wasn’t fair, and my write-up of it now opens by saying so. Copilot ran its questions at the same time and used many more searches and pages, and none of that was matched. Two of the questions leaned on GitHub, where Copilot can use GitHub’s API. And my tool caps the requests it sends to each site per hour, a cap every agent on my machine shares. It hit its GitHub cap on the first question, which broke two later ones. Three of its four losses had one cause.
The fair test: same budget, scored blind
So I ran it again with four new questions about running servers at home. Two asked how a backup program locks its storage and what changed in the current Docker release. The other two asked how fast a Mac can copy files to a home file server, and how to get smart home devices working with a home hub.
Both tools got the same questions, the same date to treat as today, and the same budget for each answer: three searches, eight pages, ten minutes and 1,200 words. Each question ran three times on each tool. Before scoring, the answers were stripped of anything that named their tool and shuffled in pairs, so the scoring was blind.
Each answer was scored out of 100, mostly for being accurate, current and complete. Over 12 answers each, mine averaged 90.00 and Copilot’s 89.75, a quarter of a point apart. At around 90 an answer was mostly right and current, and the points it lost were mainly for a page it missed or a fact that had gone stale. Both tools did both.
Speed wasn’t close. Copilot was faster on every question, from 1.1 to 1.9 times at the median.
| Measure | My tool | Copilot CLI |
|---|---|---|
| Average score out of 100, 12 answers | 90 points | 89.75 points |
| Backup question, median time | 4.3 min | 3.8 min |
| Docker question, median time | 3.7 min | 1.9 min |
| File copy question, median time | 6.6 min | 4.1 min |
| Smart home question, median time | 5.1 min | 3.3 min |
I’m not sure the tie would hold with more runs. Three runs per question, each scored once, aren’t conclusive, and the two tools ran inside different agent setups.
The pages I built it for
The general questions mixed easy pages with hard ones. My tool was built for the hard ones, so the test had a second part. Six fixed pages were each read once by my tool and once by Copilot’s built-in fetch on its own.
Copilot’s fetch gave cleaner text on plain documentation, though both got it. On each of the other five it got only part of the page. That meant the first section of a long reference, one reply from a forum thread, or a code file’s details without the code. Mine got more from all five. The Reddit thread was the starkest: Copilot’s fetch returned one word, “Reddit”, and mine got the post and its nested replies.
That part tested the fetch on its own. Copilot’s full research can also read some forums through their APIs, which this test left out.
Finding the hard pages is another matter. Asked for Reddit posts from 2026, my search returned ten relevant threads and none of them was from 2026. It ranked threads from two to three years earlier. Copilot’s search returned nothing at all.
What I learned
My first comparison pointed the wrong way twice. It had Copilot ahead on completeness and mine ahead on speed. With matched budgets and blind scoring, quality came out level and Copilot was faster on every question.
The average hid what my tool was for. Over twelve general answers the scores tied. Tested on hard pages alone, my tool got more from five of the six.
Where it stands
I kept my tool for the pages it reads better, and it’s still what my agents use first. Since 31 October, when its own search fails, the agent finds pages with Copilot’s built-in search and my tool still reads them. It’s still slower, and when a question asks for posts from this year, its search still ranks old threads first.