微信扫码
与创始人交个朋友
我要投稿
之前介绍了文本切分五个层级,本文方法是第三个层次:
Level 1: Character Splitting - 简单的字符长度切分
Level 2: Recursive Character Text Splitting - 通过分隔符切分,然后递归合并
Level 3: Document Specific Splitting - 针对不同文档格式切分 (PDF, Python, Markdown)
Level 4: Semantic Splitting - 语义切分
Level 5: Agentic Splitting-使用代理实现自动切分
基本概念和环境
分块通常旨在将具有共同上下文的文本放在一起。考虑到这一点,我们可能希望特别尊重文档本身的结构。例如,markdown 文件按标题组织。在特定标题组中创建块是一种直观的想法。为了解决这一挑战,我们可以使用MarkdownHeaderTextSplitter。这将按指定的一组标题拆分 markdown 文件。
本文用到的安装包如下:
pip install langchain-text-splitters
切分实现
markdown_document = "# Foo\n\n## Bar\n\nHi this is Jim\n\nHi this is Joe\n\n ### Boo \n\n Hi this is Lance \n\n ## Baz\n\n Hi this is Molly"
headers_to_split_on = [
("#", "Header 1"),
("##", "Header 2"),
("###", "Header 3"),
]
markdown_splitter = MarkdownHeaderTextSplitter(
headers_to_split_on)
md_header_splits = markdown_splitter.split_text(
markdown_document)
print(md_header_splits)
结果如下:
[Document(page_content='Hi this is Jim\nHi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}), Document(page_content='Hi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}), Document(page_content='Hi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on, strip_headers=False)md_header_splits = markdown_splitter.split_text(markdown_document)print(md_header_splits)
[Document(page_content='# Foo\n## Bar\nHi this is Jim\nHi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}), Document(page_content='### Boo\nHi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}), Document(page_content='## Baz\nHi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]
如何将 Markdown 行返回为单独的文档
默认情况下,MarkdownHeaderTextSplitter根据headers_to_split_on中指定的标题聚合行。我们可以通过指定return_each_line来禁用此功能,使得一行就是一条内容:
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on,return_each_line=True,)md_header_splits = markdown_splitter.split_text(markdown_document)print(md_header_splits)
[Document(page_content='Hi this is Jim', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}), Document(page_content='Hi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}), Document(page_content='Hi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}), Document(page_content='Hi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]
如何限制块大小:
然后,我们可以在每个 markdown 组中应用任何我们想要的文本分割器,例如RecursiveCharacterTextSplitter,它允许进一步控制块大小。
markdown_document = "# Intro \n\n## History \n\n Markdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9] \n\n Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files. \n\n ## Rise and divergence \n\n As Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for \n\n additional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks. \n\n #### Standardization \n\n From 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort. \n\n ## Implementations \n\n Implementations of Markdown are available for over a dozen programming languages."
headers_to_split_on = [
("#", "Header 1"),
("##", "Header 2"),
]
# MD splits
markdown_splitter = MarkdownHeaderTextSplitter(
headers_to_split_on=headers_to_split_on, strip_headers=False
)
md_header_splits = markdown_splitter.split_text(markdown_document)
# Char-level splits
from langchain_text_splitters import RecursiveCharacterTextSplitter
chunk_size = 250
chunk_overlap = 30
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=chunk_size, chunk_overlap=chunk_overlap
)
# Split
splits = text_splitter.split_documents(md_header_splits)
splits
颠覆传统OCR轻松搞定复杂PDF的工具
53AI,企业落地应用大模型首选服务商
产品:大模型应用平台+智能体定制开发+落地咨询服务
承诺:先做场景POC验证,看到效果再签署服务协议。零风险落地应用大模型,已交付160+中大型企业
2024-10-06
Anthropic RAG: 上下文检索技术
2024-10-06
AI检索黑马获2000万美元融资,推进RAG系统精准化,破解AI幻觉难题
2024-10-06
OpenRAG:全面增强RAG推理,超越Self-RAG、RAG 2.0、Command R+
2024-10-06
如何引入chunk层级优化RAG性能:从树结构RAPTOR到GEM-RAG加权图方案解读
2024-10-06
KAG:RAG已经不够了,知识增强生成才是王道,提升朴素RAG一倍性能
2024-10-05
打造自己的RAG解析大模型:Windows下部署OCR应用服务(可商业应用)
2024-10-05
基于RAG的to B智能体(Agent)应用实践
2024-10-03
RAG实战篇:将用户输入转换为精确的数据库查询语言
2024-07-18
2024-07-09
2024-07-09
2024-07-08
2024-07-09
2024-06-20
2024-07-07
2024-05-05
2024-07-07
2024-05-19
2024-09-30
2024-09-26
2024-09-26
2024-09-20
2024-09-16
2024-09-12
2024-09-11
2024-09-10