Summarization Chain vs. Summarization Checker Chain: A Comprehensive Comparison

10 min readAug 12, 2024

Learn the Pros and Cons of the different types of summarization chains, and determine the best use cases.

LLMSummarizationCheckerChain VS load_summarize_chain

The LLMSummarizationCheckerChain and load_summarize_chain in LangChain serve different purposes and are used for different aspects of summarization with large language models (LLMs). Below is a comparison of the two:

Purpose and Functionality

load_summarize_chain:

General Summarization Chain: This is a utility to load a predefined summarization chain, which is typically used for generating summaries of text. It is designed to provide a straightforward summarization process, which can handle various types of documents.
Versatile and Configurable: Depending on how it’s set up, this chain can be adapted to different summarization needs, whether summarizing short texts, long documents, or specific sections of a document.
However, its main focus is on producing a summary rather than checking or refining it.

LLMSummarizationCheckerChain:

Summarization Quality Assurance:

This chain is specifically designed to validate, check, and refine summaries generated by LLMs.
It focuses on ensuring the accuracy, coherence, and completeness of the summary.
Iterative Refinement and Validation: It can detect issues or inaccuracies in the initial summary and iteratively refine the output. This makes it especially useful when the accuracy of the summary is critical, such as in professional or academic settings.

2. Workflow and Process

load_summarize_chain:

Simple and Direct: The process involves generating a summary based on the input text, often in one or a few steps. It doesn’t typically include a built-in mechanism for checking or refining the summary.
Faster Execution: Because it involves fewer steps and focuses purely on summarization, it’s generally faster and more straightforward to use.

LLMSummarizationCheckerChain:

Validation and Refinement: This chain involves a more detailed process where the initial summary is evaluated, checked for errors, and then refined. This ensures that the final summary is both accurate and contextually appropriate.
Comprehensive Summary Quality Control: It may involve additional steps like comparing the summary against the original text, ensuring that key points are not missed, and correcting any inaccuracies.

3. Use Cases

load_summarize_chain:

Quick Summarization Needs: Ideal for scenarios where a quick, general summary is needed without the need for detailed accuracy or error-checking.
Versatility: Suitable for a broad range of documents and summarization tasks, making it a good choice for general-purpose use.
LLMSummarizationCheckerChain:
High-Accuracy Requirements: Best used when the accuracy, coherence, and completeness of the summary are crucial. This includes professional reports, academic papers, legal documents, or any context where errors in the summary could have significant consequences.
Complex or Long Documents: Particularly useful for summarizing large or complex documents where an iterative refinement process helps ensure that the summary captures all critical information accurately.

4. Advantages

load_summarize_chain:

Ease of Use: Simple to implement and quick to run, making it ideal for straightforward summarization tasks.
Speed: Because it doesn’t involve additional checking or refinement steps, it’s generally faster.

LLMSummarizationCheckerChain:

Reliability: Provides a higher level of assurance that the summary is accurate and reliable, thanks to its iterative checking and refinement process.
Detailed Summarization: Better suited for tasks where the summary must be comprehensive, accurate, and free from errors.

5. When to Use Which

Use load_summarize_chain:

When you need a quick, general summary without complex requirements.
When working with simpler or shorter documents where accuracy is important but not critical.

Use LLMSummarizationCheckerChain:

When summarizing long, complex documents that require a high level of accuracy and completeness.
When you need to ensure that the summary is free from errors and accurately reflects the original content.In professional, academic, or legal contexts where precision is paramount.

In summary, load_summarize_chain is a more straightforward tool for generating summaries quickly, while LLMSummarizationCheckerChain is a more sophisticated tool designed for ensuring the accuracy and quality of summaries through iterative refinement and checking.’’’

Let’s go hands-on with the code.

We will set up the environment, here we will use the hugging face model API calls which provide better token limits than the OpenAi Api.

First login to the Hugging face and generate the API key(Access Token)


#######################################################
#Step up the LLM Environment 
#######################################################

from langchain_community.llms.huggingface_endpoint import HuggingFaceEndpoint
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
from langchain.text_splitter import CharacterTextSplitter
from langchain.chains.mapreduce import MapReduceChain

##################################################
#Model API call
repo_id = "mistralai/Mistral-7B-Instruct-v0.2"
llm = HuggingFaceEndpoint(
    repo_id=repo_id,
    max_length=128,
    temperature=0.5,
    huggingfacehub_api_token= "hf_yourkey")

#Split the text


#Split the text

text_splitter = CharacterTextSplitter()

# load the document
with open('page2.txt',encoding="utf8") as f:
    page= f.read()
texts = text_splitter.split_text(page)

#load the document object without saving it into the vector-db


#document format
#load a document object from something you just want to copy and paste. 
#In this case, you don't even need to use a DocumentLoader, but rather can just construct the Document directly.


from langchain.docstore.document import Document
docs = [Document(page_content=t) for t in texts[:4]]

1. Summarizing with ‘map_reduce’ Chain type

The MapReduce method in LangChain, implemented as the MapReduceDocumentsChain, involves processing each chunk of data individually before combining the results. For summarization, each chunk is summarized first, and then these summaries are merged.
For question-answering tasks, each chunk provides an answer, which is then combined.

Pros:
Scalability: Can handle larger documents or more documents compared to StuffDocumentsChain.
Parallelization: Individual document processing can be done in parallel, making the process faster.
Cons:
Increased LLM Calls: Requires significantly more calls to the LLM, which can be resource-intensive.
Information Loss: Some details may be lost during the final combination of the outputs.


from langchain.chains.summarize import load_summarize_chain
import textwrap

chain = load_summarize_chain(llm, 
                             chain_type="map_reduce", verbose=True,)


output_summary = chain.run(docs)
wrapped_text = textwrap.fill(output_summary, width=100)
print(wrapped_text)

# View Prompt Template for summarizing each part of Map Reduce
chain.llm_chain.prompt.template

# for combining the parts
chain.combine_document_chain.llm_chain.prompt.template

# with Prompt template --------------------
prompt_template = """Write a summary in bullets for the following:

{text}

SUMMARY IN BULLETS:"""

BULLET_POINT_PROMPT = PromptTemplate(template=prompt_template, 
                        input_variables=["text"])

chain = load_summarize_chain(llm, 
                             chain_type="map_reduce",
                             return_intermediate_steps=False, # if True use chain.invoke()
                             map_prompt=BULLET_POINT_PROMPT, 
                             combine_prompt=BULLET_POINT_PROMPT)

#directly set
# chain.llm_chain.prompt= BULLET_POINT_PROMPT
# chain.combine_document_chain.llm_chain.prompt= BULLET_POINT_PROMPT

output_summary = chain.run(docs)
wrapped_text = textwrap.fill(output_summary, 
                             width=100,
                             break_long_words=False,
                             replace_whitespace=False)
print(wrapped_text)

here are the outputs

chain.llm_chain.prompt.template | chain.combine_document_chain.llm_chain.prompt.template

2. Summarizing with the ‘stuff’ Chain type

Stuffing is the simplest approach, where all relevant data is packed into the prompt to provide context for the language model. In LangChain, this method is implemented as the StuffDocumentsChain.

Pros:
Single LLM Call: Requires only one call to the LLM.
Comprehensive Context: The LLM processes all the data at once, which can be beneficial for generating text.
Cons:
Context Length Limitation: Most LLMs have a context length limit, so this method may not work for large or numerous documents, as the prompt might exceed the allowable length.
The main drawback of this approach is that it is only suitable for smaller datasets. When dealing with larger amounts of data, this method becomes impractical, and more advanced methods are needed.


#intialize the chain
chain = load_summarize_chain(llm, chain_type="stuff")

prompt_template = """Write a summary in bullets for the following:
{text}
SUMMARY IN BULLETS:"""

BULLET_POINT_PROMPT = PromptTemplate(template=prompt_template, 
                        input_variables=["text"])

chain = load_summarize_chain(llm, 
                             chain_type="stuff", 
                             prompt=BULLET_POINT_PROMPT,verbose=True)

output_summary = chain.run(docs)

wrapped_text = textwrap.fill(output_summary, 
                             width=100,
                             break_long_words=False,
                             replace_whitespace=False)
print(wrapped_text)

len(output_summary)

here are the output

3. Summarizing with the ‘refine’ Chain type

The Refine method begins with an initial prompt on the first chunk of data, which generates an output. For subsequent documents, this output is combined with the next document, and the LLM is asked to refine the output based on the new information.

Pros:
Enhanced Contextual Relevance: Can incorporate more relevant context and may retain more information compared to the MapReduceDocumentsChain.
Cons:
Increased LLM Calls: Requires more LLM calls than the StuffDocumentsChain.
Sequential Dependency: The calls are not independent, meaning they cannot be parallelized like in the MapReduceDocumentsChain. There is also a potential dependency on the order in which documents are processed.

chain = load_summarize_chain(llm, chain_type="refine",)

output_summary = chain.run(docs)
wrapped_text = textwrap.fill(output_summary, width=100)
print(wrapped_text)

With refine prompt template


#with Refine Template
prompt_template = """Write a concise summary of the following extracting the key information:

{text}

CONCISE SUMMARY:"""

PROMPT = PromptTemplate(template=prompt_template, 
                        input_variables=["text"])

refine_template = (
    "Your job is to produce a final summary\n"
    "We have provided an existing summary up to a certain point: {existing_answer}\n"
    "We have the opportunity to refine the existing summary"
    "(only if needed) with some more context below.\n"
    "------------\n"
    "{text}\n"
    "------------\n"
    "Given the new context, refine the original summary"
    "If the context isn't useful, return the original summary."
)

refine_prompt = PromptTemplate(
    input_variables=["existing_answer", "text"],
    template=refine_template)

chain = load_summarize_chain(llm, 
                             chain_type="refine", 
                             return_intermediate_steps=True, 
                             question_prompt=PROMPT, 
                             refine_prompt=refine_prompt)

output_summary = chain({"input_documents": docs}, return_only_outputs=True)

wrapped_text = textwrap.fill(output_summary['output_text'], 
                             width=100,
                             break_long_words=False,
                             replace_whitespace=False)
print(wrapped_text)
len(output_summary["output_text"])

print(wrapped_text) | len(output_summary[“output_text”])

DONE THAT IT !!!!!!!!!!!!!

NOW WE WILL CHECK THE “SUMMARIZATION CHECKER”

As as have already seen earlier the pros and cons of summarization checker chain “LLMSummarizationCheckerChain” let’s quickly go through its implementation


#dummy text
article = '''Coinbase, the second-largest crypto exchange by trading volume, released its Q4 2022 earnings on Tuesday, giving shareholders and market players alike an updated look into its financials. In response to the report, the company's shares are down modestly in early after-hours trading.In the fourth quarter of 2022, Coinbase generated $605 million in total revenue, down sharply from $2.49 billion in the year-ago quarter. Coinbase's top line was not enough to cover its expenses: The company lost $557 million in the three-month period on a GAAP basis (net income) worth -$2.46 per share, and an adjusted EBITDA deficit of $124 million.Wall Street expected Coinbase to report $581.2 million in revenue and earnings per share of -$2.44 with adjusted EBITDA of -$201.8 million driven by 8.4 million monthly transaction users (MTUs), according to data provided by Yahoo Finance.Before its Q4 earnings were released, Coinbase's stock had risen 86% year-to-date. Even with that rally, the value of Coinbase when measured on a per-share basis is still down significantly from its 52-week high of $206.79.That Coinbase beat revenue expectations is notable in that it came with declines in trading volume; Coinbase historically generated the bulk of its revenues from trading fees, making Q4 2022 notable. Consumer trading volumes fell from $26 billion in the third quarter of last year to $20 billion in Q4, while institutional volumes across the same timeframe fell from $133 billion to $125 billion.The overall crypto market capitalization fell about 64%, or $1.5 trillion during 2022, which resulted in Coinbase's total trading volumes and transaction revenues to fall 50% and 66% year-over-year, respectively, the company reported.As you would expect with declines in trading volume, trading revenue at Coinbase fell in Q4 compared to the third quarter of last year, dipping from $365.9 million to $322.1 million. (TechCrunch is comparing Coinbase's Q4 2022 results to Q3 2022 instead of Q4 2021, as the latter comparison would be less useful given how much the crypto market has changed in the last year; we're all aware that overall crypto activity has fallen from the final months of 2021.)There were bits of good news in the Coinbase report. While Coinbase's trading revenues were less than exuberant, the company's other revenues posted gains. What Coinbase calls its "subscription and services revenue" rose from $210.5 million in Q3 2022 to $282.8 million in Q4 of the same year, a gain of just over 34% in a single quarter.And even as the crypto industry faced a number of catastrophic events, including the Terra/LUNA and FTX collapses to name a few, there was still growth in other areas. The monthly active developers in crypto have more than doubled since 2020 to over 20,000, while major brands like Starbucks, Nike and Adidas have dived into the space alongside social media platforms like Instagram and Reddit.With big players getting into crypto, industry players are hoping this move results in greater adoption both for product use cases and trading volumes. Although there was a lot of movement from traditional retail markets and Web 2.0 businesses, trading volume for both consumer and institutional users fell quarter-over-quarter for Coinbase.Looking forward, it'll be interesting to see if these pieces pick back up and trading interest reemerges in 2023, or if platforms like Coinbase will have to keep looking elsewhere for revenue (like its subscription service) if users continue to shy away from the market.
'''

wrapped_text = textwrap.fill(article, 
                             width=100,
                             break_long_words=False,
                             replace_whitespace=False)
print(wrapped_text)

len(article)

fact_extraction_prompt = PromptTemplate(
    input_variables=["text_input"],
    template="Extract the key facts out of this text. Don't include opinions. \
    Give each fact a number and keep them short sentences. :\n\n {text_input}"
)

fact_extraction_chain = LLMChain(llm=llm, prompt=fact_extraction_prompt) # here we used LLMChain

facts = fact_extraction_chain.run(article)
len(facts) #observe the difference

wrapped_text = textwrap.fill(facts, 
                             width=100,
                             break_long_words=False,
                             replace_whitespace=False)
print(wrapped_text)  

#Observe the difference
from langchain.chains import LLMSummarizationCheckerChain

checker_chain = LLMSummarizationCheckerChain(llm=llm, 
                                             verbose=True, 
                                             max_checks=2
                                             )

final_summary = checker_chain.run(article)
final_summary

len(final_summary) #Observe the difference

#View the templates
checker_chain.create_assertions_prompt.template
checker_chain.check_assertions_prompt.template
checker_chain.revised_summary_prompt.template
checker_chain.are_all_true_prompt.template

checker_chain.create_assertions_prompt.template
checker_chain.check_assertions_prompt.template
checker_chain.revised_summary_prompt.template
checker_chain.are_all_true_prompt.template

Thanks for your time. i hope you enjoyed this. I tried my best to gather details across and simply as much as possible i could.

In the next article, we will explore Pros & Cons and ways to Evaluate our models using Rouge & BLEU metrics

Until then feel free to reach out. Thanks for your time, if you enjoyed this short article there are tons of topics in advanced analytics, data science, and machine learning available in my medium repo. https://medium.com/@bobrupakroy

Some of my alternative internet presences are Facebook, Instagram, Udemy, Blogger, Issuu, Slideshare, Scribd, and more.

Also available on Quora @ https://www.quora.com/profile/Rupak-Bob-Roy

Let me know if you need anything. Talk Soon.

Check out the links i hope it helps.