{"id":5721,"date":"2024-02-24T13:51:46","date_gmt":"2024-02-24T05:51:46","guid":{"rendered":"https:\/\/aict.nkust.edu.tw\/digitrans\/?p=5721"},"modified":"2024-12-13T20:37:30","modified_gmt":"2024-12-13T12:37:30","slug":"%e5%88%a9%e7%94%a8-langchain-%e5%af%a6%e4%bd%9c%e5%a4%9a%e6%a8%a1%e6%85%8b%e6%a8%a1%e5%9e%8b%e7%9a%84-rag%ef%bc%9a%e9%99%a4%e4%ba%86%e8%ae%80%e6%96%87%e7%ab%a0%e4%b9%9f%e8%83%bd%e7%9c%8b%e5%9c%96","status":"publish","type":"post","link":"https:\/\/aict.nkust.edu.tw\/digitrans\/?p=5721","title":{"rendered":"\u5229\u7528 LangChain \u5be6\u4f5c\u591a\u6a21\u614b\u6a21\u578b\u7684 RAG\uff1a\u9664\u4e86\u8b80\u6587\u7ae0\u4e5f\u80fd\u770b\u5716\u7b54\u984c"},"content":{"rendered":"\n

2024-02-24 | AIF \u6280\u8853\u767c\u5c55\u4e2d\u5fc3-\u6797\u828a<\/p>\n\n\n\n

\u76f8\u4fe1\u4e0d\u5c11\u4eba\u5df2\u7d93\u77e5\u9053 ChatGPT \u9019\u985e\u7684\u5927\u578b\u8a9e\u8a00\u6a21\u578b\uff08LLM,Large Language Model\uff09\uff0c\u96d6\u7136\u5c0d\u8a71\u80fd\u529b\u5f37\uff0c\u537b\u4e5f\u5e38\u4e82\u63a5\u8a71\u3002\u800cRAG\uff08Retrieval Augmented Generation\uff09\u7684\u505a\u6cd5\u4fbf\u662f\u8b93 LLM \u5728\u56de\u7b54\u554f\u984c\u6642\u80fd\u5920\u53c3\u8003\u76f8\u95dc\u6587\u4ef6\uff0c\u6709\u6548\u907f\u514d\u4e86\u56e0\u77e5\u8b58\u4e0d\u8db3\u800c\u7522\u751f\u7684\u5e7b\u89ba\u73fe\u8c61\uff08hallucination\uff09\uff0c\u4f8b\u5982\u57fa\u91d1\u6703\u8207\u5929\u4e0b\u96dc\u8a8c\u5408\u4f5c\u63a8\u51fa\u7684\u300c\u5b6b\u4e3b\u4efb AI \u52a9\u6559\u300d\uff0c\u6b63\u662f\u5229\u7528\u6b64\u6280\u5de7\uff0c\u8b93 LLM \u53ef\u4ee5\u6839\u64da\u300a\u5b6b\u4e3b\u4efb\u7684\u7d93\u6fdf\u7b46\u8a18\u300b\u9019\u672c\u66f8\u7684\u5167\u5bb9\uff0c\u63d0\u4f9b\u8f03\u6b63\u78ba\u3001\u9069\u7576\u7684\u56de\u61c9\u3002<\/p>\n\n\n\n

\u7576\u7136\uff0c\u65e5\u5e38\u751f\u6d3b\u4e2d\u9664\u4e86\u6587\u5b57\uff0c\u9084\u5305\u542b\u4e86\u5716\u7247\u3001\u8a9e\u97f3\u7b49\u4e0d\u540c\u985e\u578b\u7684\u8cc7\u8a0a\u5167\u5bb9\u3002\u8fd1\u671f\uff0cLLM\u7684\u767c\u5c55\u4e5f\u9081\u5411\u591a\u6a21\u614b\u6a21\u578b\uff08multi-modal\uff09\u7684\u9818\u57df\uff0c\u4f7f\u5f97\u6a21\u578b\u80fd\u5920\u66f4\u5168\u9762\u5730\u7406\u89e3\u5305\u62ec\u5716\u7247\u5728\u5167\u7684\u591a\u7a2e\u8cc7\u8a0a\u3002\u672c\u6587\u5c07\u5206\u4eab\u5982\u4f55\u5c07\u5716\u50cf\u6574\u5408\u5230 RAG \u4e2d\u7684\u5be6\u4f5c\u65b9\u6cd5\uff0c\u4f7f\u61c9\u7528\u5834\u666f\u80fd\u66f4\u52a0\u8c50\u5bcc\u3002<\/p>\n\n\n\n

\n

\u5982\u679c\u4f60\u5c0d RAG \u9084\u4e0d\u592a\u719f\u6089\u7684\u8a71\uff0c\u5efa\u8b70\u53ef\u4ee5\u5148\u53c3\u8003\u4e0a\u4e00\u7bc7\u6587\u7ae0\u300a\u5229\u7528LangChain\u5be6\u4f5cChatPDF\uff1a\u554f\u500b\u554f\u984c\uff0c\u8f15\u9b06\u627e\u51fa\u6587\u4ef6\u91cd\u9ede\u300b\uff0c\u9019\u7bc7\u6587\u7ae0\u4ecb\u7d39\u4e86\u5982\u4f55\u5c07RAG\u52a0\u5165\u5c0d\u8a71\u6a5f\u5668\u4eba\u4e2d\u3002<\/p>\n<\/blockquote>\n\n\n\n

\u4e00\u3001\u8b80\u53d6PDF\u6a94<\/h3>\n\n\n\n

\u9996\u5148\uff0c\u6211\u5011\u4ee5\u300aSwin Transformer\uff1a Hierarchical Vision Transformer using Shifted Windows\u300b\u9019\u7bc7\u8ad6\u6587\u7684 PDF \u6a94\u6848\u70ba\u4f8b\uff0c\u5229\u7528 unstructured \u5957\u4ef6\u4e2d\u7684 partition_pdf \u5de5\u5177\uff0c\u5c07PDF\u6587\u7ae0\u7d30\u5206\u70ba\u300c\u6587\u5b57\u300d\u3001\u300c\u8868\u683c\u300d\u548c\u300c\u5716\u50cf\u300d\u4e09\u7a2e\u5143\u7d20\uff0c\u5f8c\u7e8c\u4f9d\u5143\u7d20\u985e\u578b\u5c07\u6709\u4e0d\u540c\u7684\u8655\u7406\u65b9\u5f0f\u3002<\/p>\n\n\n\n

import os\nimport uuid\nimport base64\nfrom IPython import display\nfrom unstructured.partition.pdf import partition_pdf\nfrom langchain.chat_models import ChatOpenAI\nfrom langchain.embeddings import OpenAIEmbeddings\nfrom langchain.chains import LLMChain\nfrom langchain.prompts import PromptTemplate\nfrom langchain.schema.messages import HumanMessage, SystemMessage\nfrom langchain.schema.document import Document\nfrom langchain.vectorstores import FAISS\nfrom langchain.retrievers.multi_vector import MultiVectorRetriever\n\nos.environ['OPENAI_API_KEY'] = \"sk-*****\"\noutput_path = \".\/images\/Swin\"\n\n# Get elements\nraw_pdf_elements = partition_pdf(\n    filename=\".\/PDFs\/Swin.pdf\",\n    extract_images_in_pdf=True,\n    infer_table_structure=True,\n    chunking_strategy=\"by_title\",\n    max_characters=4000,\n    new_after_n_chars=3800,\n    combine_text_under_n_chars=2000,\n    extract_image_block_output_dir=output_path,\n)<\/code><\/pre>\n\n\n\n
\u4e8c\u3001\u5c07\u6587\u5b57\u3001\u8868\u683c\u53ca\u5716\u50cf\u8f49\u63db\u6210\u6587\u5b57\u6458\u8981<\/h3>\n\n\n\n
\u5f97\u5230\u300c\u6587\u5b57\u300d\u3001\u300c\u8868\u683c\u300d\u548c\u300c\u5716\u50cf\u300d\u4e09\u7a2e\u5143\u7d20\u5f8c\uff0c\u7dca\u63a5\u8457\u8981\u601d\u8003\u5982\u4f55\u5c07\u5b83\u5011\u5b58\u5230 Vectorstore \u4e2d\u3002\u6211\u5011\u9078\u64c7\u5c07\u4e09\u7a2e\u5143\u7d20\u90fd\u8f49\u63db\u6210\u6587\u5b57\u6458\u8981\uff0c\u518d\u5c07\u6587\u5b57\u6458\u8981\u8f49\u63db\u6210\u5411\u91cf\uff0c\u4ee5\u4fbf\u9032\u884c\u5f8c\u7e8c\u641c\u7d22\uff0c\u9019\u9ebc\u505a\u7684\u539f\u56e0\u4e3b\u8981\u662f\u56e0\u70ba\u6587\u4ef6\u6aa2\u7d22\u6642\uff0c\u901a\u5e38\u6703\u4f7f\u7528\u6587\u7ae0\u7684\u5411\u91cf\u76f8\u4f3c\u5ea6\u9032\u884c\u641c\u5c0b\u3002\u6b64\u5916\uff0c\u5728\u751f\u6210\u6458\u8981\u7684\u904e\u7a0b\u4e2d\uff0c\u4f7f\u7528\u201dgpt-3.5-turbo\u201d\u6a21\u578b\u8655\u7406\u6587\u5b57\u548c\u8868\u683c\u90e8\u5206\uff0c\u4e26\u4ee5\u201dgpt-4-vision-preview\u201d\u6a21\u578b\u8655\u7406\u5716\u50cf\u90e8\u5206\u3002\uff08\u5982\u4e0b\u5716\u6240\u793a\uff09<\/p>\n\n\n
\n
\"\"
\u6587\u4ef6\u8f49\u63db\u6d41\u7a0b<\/figcaption><\/figure>\n<\/div>\n\n\n
# Get text summaries and table summaries\ntext_elements = []\ntable_elements = []\n\ntext_summaries = []\ntable_summaries = []\n\nsummary_prompt = \"\"\"\nSummarize the following {element_type}: \n{element}\n\"\"\"\nsummary_chain = LLMChain(\n    llm=ChatOpenAI(model=\"gpt-3.5-turbo\", max_tokens=1024), \n    prompt=PromptTemplate.from_template(summary_prompt)\n)\n\nfor e in raw_pdf_elements:\n    if 'CompositeElement' in repr(e):\n        text_elements.append(e.text)\n        summary = summary_chain.run({'element_type': 'text', 'element': e})\n        text_summaries.append(summary)\n\n    elif 'Table' in repr(e):\n        table_elements.append(e.text)\n        summary = summary_chain.run({'element_type': 'table', 'element': e})\n        table_summaries.append(summary)\n\n# Get image summaries\nimage_elements = []\nimage_summaries = []\n\ndef encode_image(image_path):\n    with open(image_path, \"rb\") as f:\n        return base64.b64encode(f.read()).decode('utf-8')\n    \ndef summarize_image(encoded_image):\n    prompt = [\n        SystemMessage(content=\"You are a bot that is good at analyzing images.\"),\n        HumanMessage(content=[\n            {\n                \"type\": \"text\", \n                \"text\": \"Describe the contents of this image.\"\n            },\n            {\n                \"type\": \"image_url\",\n                \"image_url\": {\n                    \"url\": f\"data:image\/jpeg;base64,{encoded_image}\"\n                },\n            },\n        ])\n    ]\n    response = ChatOpenAI(model=\"gpt-4-vision-preview\", max_tokens=1024).invoke(prompt)\n    return response.content\n    \nfor i in os.listdir(output_path):\n    if i.endswith(('.png', '.jpg', '.jpeg')):\n        image_path = os.path.join(output_path, i)\n        encoded_image = encode_image(image_path)\n        image_elements.append(encoded_image)\n        summary = summarize_image(encoded_image)\n        image_summaries.append(summary)<\/code><\/pre>\n\n\n\n
\u4ee5\u4e0b\u662f  LLM\u5c0d\u8868\u683c\u8cc7\u6599\u53ca\u5716\u50cf\u8cc7\u6599\u505a\u7684\u6458\u8981\uff0c\u660e\u986f\u53ef\u770b\u51fa\u7121\u8ad6\u662f\u8907\u96dc\u7684\u6a21\u578b\u67b6\u69cb\uff0c\u6216\u662f\u7f3a\u4e4f\u6587\u5b57\u89e3\u8aaa\u7684\u6d41\u7a0b\u5716\uff0cLLM \u90fd\u5c55\u73fe\u51fa\u5177\u4e00\u5b9a\u6c34\u6e96\u7684\u7406\u89e3\u80fd\u529b\u3002\u7dca\u63a5\u8457\uff0c\u5c31\u8981\u5229\u7528\u9019\u4e9b\u6458\u8981\u5167\u5bb9\u5c55\u958b\u9032\u4e00\u6b65\u641c\u5c0b\u3002<\/p>\n\n\n
\n
\"\"
\u6587\u4e2d\u8868\u683c<\/figcaption><\/figure>\n<\/div>\n\n\n
\"\"\/
\u8868\u683c\u6458\u8981<\/figcaption><\/figure>\n\n\n
\n
\"\"
\u6587\u4e2d\u5716\u50cf<\/figcaption><\/figure>\n<\/div>\n\n
\n
\"\"
\u5716\u50cf\u6458\u8981<\/figcaption><\/figure>\n<\/div>\n\n\n
\u4e09\u3001\u5c07\u6587\u5b57\u6458\u8981\u8f49\u63db\u6210\u5411\u91cf<\/h3>\n\n\n\n
\u63a5\u8457\uff0c\u6211\u5011\u900f\u904e Embedding \u6a21\u578b\u5c07\u6458\u8981\u8f49\u63db\u6210\u5411\u91cf\u4e26\u5132\u5b58\u5230\u5411\u91cf\u8cc7\u6599\u5eab\uff08vectorstore\uff09\u4e2d\u3002\u7531\u65bc\u5411\u91cf\u8cc7\u6599\u5eab\u63d0\u4f9b\u4e86\u5f37\u5927\u7684\u6f14\u7b97\u6cd5\uff0c\u91dd\u5c0d\u5411\u91cf\u5feb\u901f\u9032\u884c\u76f8\u4f3c\u5ea6\u641c\u5c0b\uff0c\u56e0\u6b64\uff0c\u6211\u5011\u5c07\u5411\u91cf\u8207\u539f\u59cb\u8cc7\u6599\u9023\u7d50\u5728\u4e00\u8d77\uff0c\u4ee5\u4fbf\u53ef\u4ee5\u53ea\u91dd\u5c0d\u6458\u8981\u9032\u884c\u6aa2\u7d22\uff0c\u4e26\u53d6\u51fa\u672a\u7d93\u8f49\u63db\u7684\u539f\u59cb\u8cc7\u6599\u3002<\/p>\n\n\n\n
# Create Documents and Vectorstore\ndocuments = []\nretrieve_contents = []\n\nfor e, s in zip(text_elements, text_summaries):\n    i = str(uuid.uuid4())\n    doc = Document(\n        page_content = s,\n        metadata = {\n            'id': i,\n            'type': 'text',\n            'original_content': e\n        }\n    )\n    retrieve_contents.append((i, e))\n    documents.append(doc)\n    \nfor e, s in zip(table_elements, table_summaries):\n    doc = Document(\n        page_content = s,\n        metadata = {\n            'id': i,\n            'type': 'table',\n            'original_content': e\n        }\n    )\n    retrieve_contents.append((i, e))\n    documents.append(doc)\n    \nfor e, s in zip(image_elements, image_summaries):\n    doc = Document(\n        page_content = s,\n        metadata = {\n            'id': i,\n            'type': 'image',\n            'original_content': e\n        }\n    )\n    retrieve_contents.append((i, s))\n    documents.append(doc)\n\nvectorstore = FAISS.from_documents(documents=documents, embedding=OpenAIEmbeddings())<\/code><\/pre>\n\n\n\n
\u56db\u3001\u5c0d\u5411\u91cf\u9032\u884c\u641c\u7d22\u4e26\u56de\u7b54\u554f\u984c<\/h3>\n\n\n\n
\u6700\u5f8c\uff0c\u7576\u6211\u5011\u8f38\u5165\u4e00\u500b\u554f\u984c\u6642\uff0c\u6a21\u578b\u53ef\u4ee5\u5148\u5f9e\u5411\u91cf\u5eab\u4e2d\u4f9d\u64da\u6458\u8981\u53d6\u51fa\u6700\u76f8\u95dc\u7684n\u500b\u5167\u5bb9\uff0c\u518d\u5c07\u300c\u6587\u5b57\u8207\u8868\u683c\u7684\u539f\u59cb\u8cc7\u6599\u300d\u53ca\u300c\u5716\u50cf\u7684\u6587\u5b57\u6458\u8981\u300d\u653e\u5165 LLM \u4e2d\u505a\u70ba\u53c3\u8003\uff0c\u9019\u6a23\u7684\u56de\u7b54\u6a21\u5f0f\uff0c\u4f7f LLM \u9664\u4e86\u53ef\u4ee5\u53c3\u8003\u6587\u7ae0\u4e2d\u7684\u6587\u5b57\uff0c\u4e5f\u80fd\u5f9e\u5716\u50cf\u8207\u8868\u683c\u4e2d\u5f97\u5f97\u66f4\u591a\u8cc7\u8a0a\u3002\u4ee5\u4e0b\u7a0b\u5f0f\u78bc\u4fbf\u662f\u6a21\u578b\u5c0d\u65bc\u554f\u984c\u300cWhat is the difference between Swin Transformer and ViT\uff1f\u300d\u7684\u56de\u7b54\u4ee5\u53ca\u5176\u6240\u53c3\u8003\u7684\u5716\u50cf\u3002<\/p>\n\n\n\n
answer_template = \"\"\"\nAnswer the question based only on the following context, which can include text, images and tables:\n{context}\nQuestion: {question} \n\"\"\"\nanswer_chain = LLMChain(llm=ChatOpenAI(model=\"gpt-3.5-turbo\", max_tokens=1024), prompt=PromptTemplate.from_template(answer_template))\n\ndef answer(question):\n    relevant_docs = vectorstore.similarity_search(question)\n    context = \"\"\n    relevant_images = []\n    for d in relevant_docs:\n        if d.metadata['type'] == 'text':\n            context += '[text]' + d.metadata['original_content']\n        elif d.metadata['type'] == 'table':\n            context += '[table]' + d.metadata['original_content']\n        elif d.metadata['type'] == 'image':\n            context += '[image]' + d.page_content\n            relevant_images.append(d.metadata['original_content'])\n    result = answer_chain.run({'context': context, 'question': question})\n    return result, relevant_images<\/code><\/pre>\n\n\n
\n
\"\"
\u56de\u7b54\u7d50\u679c<\/figcaption><\/figure>\n<\/div>\n\n\n
\u7d50\u8a9e<\/h3>\n\n\n\n
\u672c\u6587\u85c9\u7531\u4e00\u500b\u7c21\u55ae\u7bc4\u4f8b\u5be6\u4f5c LLM \u61c9\u7528\u9818\u57df\u4e2d\uff0c\u4f7f\u7528\u591a\u6a21\u614b\u6a21\u578b\u5b8c\u6210\u7684 RAG \u5de5\u5177\u3002\u9996\u5148\u5c07\u4e0d\u540c\u5f62\u5f0f\u7684\u6587\u4ef6\u5167\u5bb9\u8f49\u63db\u6210\u6587\u5b57\u6458\u8981\uff0c\u518d\u5c07\u6587\u5b57\u6458\u8981\u8f49\u63db\u6210\u5411\u91cf\u9032\u884c\u641c\u5c0b\uff0c\u4f7f\u5f97\u6a21\u578b\u5728\u56de\u7b54\u7684\u904e\u7a0b\u4e2d\u65e2\u53c3\u8003\u6587\u5b57\u5167\u5bb9\u3001\u53c8\u80fd\u5920\u5f9e\u5716\u50cf\u548c\u8868\u683c\u4e2d\u7372\u5f97\u66f4\u591a\u8cc7\u8a0a\u3002\u5f9e\u904e\u53bb\u5c0d\u55ae\u4e00\u6a21\u614b\u7684\u56de\u7b54\u5230\u73fe\u5728\uff0c\u6211\u5011\u770b\u5230 LLM \u6280\u8853\u4e0d\u65b7\u5730\u6f14\u9032\uff0c\u5176\u61c9\u7528\u751a\u81f3\u5df2\u64f4\u5c55\u81f3\u591a\u6a21\u614b\u5834\u666f\uff0c\u4f7f\u6a21\u578b\u80fd\u66f4\u5168\u9762\u7684\u7406\u89e3\u6587\u672c\u4e2d\u4e0d\u540c\u7a2e\u985e\u7684\u8cc7\u8a0a\u3002\u5047\u4f7f\u8b80\u8005\u624b\u908a\u6709\u4e00\u4e9b\u5305\u542b\u6587\u5b57\u53ca\u5716\u50cf\u7684\u591a\u6a21\u614b\u8cc7\u6599\uff0c\u4e0d\u59a8\u5617\u8a66\u5229\u7528\u9019\u500b\u65b9\u6cd5\uff0c\u5be6\u969b\u8a66\u505a\u770b\u770b\u3002<\/p>\n\n\n\n
\u53c3\u8003\u8cc7\u6599<\/h3>\n\n\n\n