ScrapeGraphAI 是一个 Python 库，它利用大型语言模型（LLM）和直接图形逻辑为网站、文档和 XML 文件创建抓取管道。这个库的特点是，用户只需要描述他们想要提取的信息，库就会自动完成抓取任务。
安装 ScrapeGraphAI
通过 pip 安装 ScrapeGraphAI：
pip install scrapegraphai
安装 Playwright，用于基于 JavaScript 的抓取：
playwright install
建议在虚拟环境中安装库，以避免与其他库发生冲突。
使用 ScrapeGraphAI
ScrapeGraphAI 提供了三种主要的抓取管道：
SmartScraperGraph：单页面抓取器，只需要用户提示和输入源。
SearchGraph：多页面抓取器，从搜索引擎的前 n 个搜索结果中提取信息。
SpeechGraph：单页面抓取器，从网站提取信息并生成音频文件。
示例用例
使用本地模型的 SmartScraperGraph：
确保已安装 Ollama 并使用 ollama pull 命令下载模型。
示例代码展示了如何创建 SmartScraperGraph 实例并运行它，以获取项目列表及其描述。
使用混合模型的 SearchGraph：
使用 OpenAI 的 SpeechGraph：
输出示例
注意事项
ScrapeGraphAI 库通过简化抓取过程，使得用户无需深入了解网页结构或编写复杂的抓取逻辑，就能够从网站中提取所需信息。这对于需要从多个来源收集数据的用户来说，是一个非常有用的工具。
? Quick install
The reference page for Scrapegraph-ai is available on the official page of pypy: pypi.
pip install scrapegraphai
you will also need to install Playwright for javascript-based scraping:
playwright install
Note: it is recommended to install the library in a virtual environment to avoid conflicts with other libraries ?
? Demo
Follow the procedure on the following link to setup your OpenAI API key: link.
? Documentation
The documentation for ScrapeGraphAI can be found here.
Check out also the docusaurus documentation.
? Usage
There are three main scraping pipelines that can be used to extract information from a website (or local file):
SmartScraperGraph: single-page scraper that only needs a user prompt and an input source;
SearchGraph: multi-page scraper that extracts information from the top n search results of a search engine;
SpeechGraph: single-page scraper that extracts information from a website and generates an audio file.
It is possible to use different LLM through APIs, such as OpenAI, Groq, Azure and Gemini, or local models using Ollama.
Case 1: SmartScraper using Local Models
Remember to have Ollama installed and download the models using the ollama pull command.
scrapegraphai.graphs
graph_config {
: {
: ,
: ,
: ,
: ,
},
: {
: ,
: ,
},
: ,
}
smart_scraper_graph (
prompt,
source,
configgraph_config
)
result smart_scraper_graph.()
(result)The output will be a list of projects with their descriptions like the following:
{'projects': [{'title': 'Rotary Pendulum RL', 'description': 'Open Source project aimed at controlling a real life rotary pendulum using RL algorithms'}, {'title': 'DQN Implementation from scratch', 'description': 'Developed a Deep Q-Network algorithm to train a simple and double pendulum'}, ...]}Case 2: SearchGraph using Mixed Models
We use Groq for the LLM and Ollama for the embeddings.
scrapegraphai.graphs
graph_config {
: {
: ,
: ,
:
},
: {
: ,
: ,
},
: ,
}
search_graph (
prompt,
configgraph_config
)
result search_graph.()
(result)The output will be a list of recipes like the following:
{'recipes': [{'name': 'Sarde in Saòre'}, {'name': 'Bigoli in salsa'}, {'name': 'Seppie in umido'}, {'name': 'Moleche frite'}, {'name': 'Risotto alla pescatora'}, {'name': 'Broeto'}, {'name': 'Bibarasse in Cassopipa'}, {'name': 'Risi e bisi'}, {'name': 'Smegiassa Ciosota'}]}Case 3: SpeechGraph using OpenAI
You just need to pass the OpenAI API key and the model name.
scrapegraphai.graphs
graph_config {
: {
: ,
: ,
},
: {
: ,
: ,
:
},
: ,
}
speech_graph (
prompt,
source,
configgraph_config,
)
result speech_graph.()
(result)The output will be an audio file with the summary of the projects on the page.
粤ICP备14082021号
免费POC，
零成本试错
