Web scraping is usually a two-step pipeline: fetch raw HTML, then parse it into structured data. Adding AI to the mix changes the second step. Instead of writing brittle XPath or CSS selectors for every site, you can clean the page, convert it to Markdown, and let a language model answer questions about the content directly.
This guide builds that pipeline in a Jupyter Notebook using standard Python libraries and a small OpenAI model. The total setup time is under ten minutes.
What You Need
Install the following packages:
The stack is intentionally lightweight:
- requests fetches the page
- BeautifulSoup removes noisy elements like scripts, styles, and comments
- markdownify converts the cleaned HTML to Markdown
- ftfy fixes broken text encoding
- openai answers the user query
- python-dotenv keeps the API key out of the notebook
Create a .env file in the same directory as the notebook and add your key:
For this task, a small model like gpt-5.4-nano is sufficient. The goal is not reasoning or creativity. It is reading comprehension over a single document.
Fetching and Cleaning
The first cell loads the environment, initializes the client, and defines the model name. The second cell fetches the target URL and parses it with BeautifulSoup. The cleaning step is aggressive by design: it strips script tags, style tags, comments, and navigation elements that add no semantic value. What remains is the article body, headings, and links.
The cleaned HTML is then passed to markdownify, which produces plain Markdown. This format is more token-efficient than raw HTML and easier for the model to read.
Querying the Content
With the Markdown content in hand, the final cell constructs a simple prompt:
The model receives the full page context and the user question in a single call. Because the content is already cleaned and compressed, even long articles fit comfortably within the context window of a small model.
When to Use This Pattern
This approach works best for question-answering over single pages: product documentation, blog posts, news articles, and documentation sites. It fails when the answer requires clicking through multiple pages, interacting with JavaScript-heavy applications, or extracting structured tabular data with precise formatting requirements.
For those cases, combine this pipeline with Playwright or Selenium for rendering, and add structured output parsing if you need JSON rather than free text. But for the common case of "what does this page say about X," the AI scraper replaces dozens of lines of fragile parsing logic with one clean API call.