AQuaMUSE
Query-Based Multi-Document News Summarization with Keyword-Guided Abstractive Summarization
An experimental NLP comparative study investigating whether statistical (RAKE) and semantic (KeyBERT MiniLM) keyword extraction methods provide contextual guidance that improves abstractive text summarization quality using BART-large-cnn.
"European energy transition policies are driving significant infrastructure shifts toward renewable sources. While structural investments face grid synchronization challenges, key renewable initiatives accelerate regional carbon reduction targets and foster technological independence."
Investigating Contextual Guidance in News Summarization
The primary objective of AQuaMUSE is to investigate whether keyword extraction methods can provide contextual guidance that improves abstractive text summarization quality.
The project compares four distinct input configurations — ranging from a non-query baseline to full context, RAKE statistical keywords, and KeyBERT semantic MiniLM keywords — using the BART-large-cnn architecture across multiple hyperparameter settings.
RESEARCH MOTIVATION & CORE QUESTION
Addressing a critical gap in query-guided text summarization research through a structured ablation study.
Limited Investigation in News Domains
Most existing work on query-based summarization focuses heavily on dialogue systems or conversational search engines. There has been limited investigation into how different keyword extraction strategies — specifically statistical versus semantic transformer-based methods — influence abstractive summarization quality on multi-document news articles.
Comparative Experimental Study
This project addresses this gap through a systematic ablation study comparing four input configurations under controlled experimental conditions. Rather than claiming a groundbreaking discovery, this work presents a rigourous experimental investigation into model behavior under keyword guidance.
“How does keyword extraction method affect the performance of query-guided abstractive multi-document news summarization?”
GOOGLE RESEARCH AQuaMUSE & DATA SCRAPING
Hybrid web scraping with fallback handling, URL caching, and rigorous filtering criteria.
Trafilatura Main Parser
Serves as the primary extraction library for retrieving clean body text, news article content, and structural metadata from scraped web links.
BeautifulSoup Fallback
Handles edge cases where primary parsing fails or encounters unconventional DOM layouts, ensuring high extraction coverage across news sources.
MD5 URL Caching System
Implements an MD5-hashed URL caching mechanism to avoid re-scraping identical web pages during iterative experimental runs.
DATASET CLEANING & FILTERING NUMBERS
Note: Numbers represent dataset filtering, not model accuracyDATASET CLEANING CRITERIA IMPLEMENTED:
EXPLORATORY DATA ANALYSIS FINDINGS
Statistical profiling of document length, query distribution, length correlation, and abstractiveness.
Average document length in the cleaned dataset before truncation.
Maximum document length requiring sentence-level selection.
Consistently compact user queries providing summary target focus.
Approximately zero correlation between query, document, and target summary lengths.
Unigram Overlap Ratio (Abstractiveness Indicator)
Statistical analysis revealed approximately 57% unigram overlap between reference summaries and source document content.
Key EDA Insight: The ~57% unigram overlap indicates that the AQuaMUSE dataset possesses a balanced hybrid nature — retaining key factual terminology from source documents while requiring abstractive rephrasing for fluent summary generation.
TECHNICAL PROCESSING PIPELINE
End-to-end 9-step workflow from data acquisition to metric comparative analysis.
Dataset Preparation — Google Research AQuaMUSE
Loading query, multi-document, and ground-truth reference summary triplets from the Google AQuaMUSE dataset.
RAKE vs KEYBERT MINILM COMPARISON
Evaluating statistical co-occurrence vs transformer semantic embedding keyword extraction.
RAKE
Rapid Automatic Keyword Extraction- Co-occurrence Graph: Evaluates word frequency and degree ratios in text adjacency graphs.
- Zero Pretraining: Does not require a pretrained language model or neural network weights.
- High Velocity: Extremely fast compute execution time suitable for high-volume pipelines.
- Focus: Captures statistically frequent phrase combinations.
KeyBERT MiniLM
paraphrase-MiniLM-L6-v2 Embeddings- Contextual Embeddings: Uses
paraphrase-MiniLM-L6-v2sentence transformer model. - Cosine Similarity: Ranks keywords by semantic vector proximity to the full document representation.
- Semantic Awareness: Identifies conceptual synonyms even without direct word repetitions.
- Lightweight Footprint: Efficient transformer encoder tailored for sentence embeddings.
BART-large-cnn & TOKEN LIMIT CHALLENGE
Overcoming BART's 1,024-token bottleneck when processing multi-document news articles.
1,024 Token Capacity vs 4,705 Word Articles
BART-large-cnn has a maximum positional input constraint of 1,024 tokens. However, the average news document in the dataset spans ~972 words, with maximum lengths reaching 4,705 words across multi-document clusters.
Directly truncating long documents destroys critical query-relevant facts. To solve this, a Query-Guided Sentence Selection strategy extracts the most query-focused sentences to construct a dense, high-signal payload within BART's token limit.
8 EXPERIMENTS ABLATION MATRIX
4 Input Configurations × 2 Hyperparameter Settings = 8 Experimental Runs.
No-Query Baseline (Doc Only)
[Document Text Only] → BART-large-cnnNo-Query Baseline (Doc Only)
[Document Text Only] → BART-large-cnnFull-Context Baseline
Query: {query}
Doc: {full_document} → BARTFull-Context Baseline
Query: {query}
Doc: {full_document} → BARTRAKE Keyword Guidance
Query: {query}
Keywords [RAKE]: {keywords}
Doc: {selected_sentences}RAKE Keyword Guidance
Query: {query}
Keywords [RAKE]: {keywords}
Doc: {selected_sentences}KeyBERT MiniLM Guidance
Query: {query}
Keywords [KeyBERT]: {keywords}
Doc: {selected_sentences}KeyBERT MiniLM Guidance
Query: {query}
Keywords [KeyBERT]: {keywords}
Doc: {selected_sentences}QUANTITATIVE EVALUATION FRAMEWORK
Measuring n-gram overlap, harmonic precision-recall, and contextual embedding similarity.
ROUGE Metrics
Evaluates n-gram recall and precision overlap between generated summaries and ground-truth references.
METEOR Score
Evaluates summary quality based on the harmonic mean of unigram precision and recall, incorporating exact matches, stemming, synonyms, and paraphrasing.
BERTScore
Computes similarity scores using contextualized token embeddings from BERT models rather than exact surface-level matches.
COMPARATIVE ASSESSMENT FRAMEWORK
Evaluating how keyword guidance impacts summary quality across 8 experimental configurations.
Query Focus & Relevance
Measuring whether keyword-guided inputs prevent model drift when generating summaries from multi-document clusters.
Statistical vs Semantic Impact
Comparing whether statistical co-occurrence (RAKE) or transformer semantic embeddings (KeyBERT) yield better contextual guidance.
Hyperparameter Sensitivity
Analyzing beam search width and length penalty interactions across different keyword input payloads.
ENGINEERING CHALLENGES OVERCOME
Key technical hurdles in data collection, token constraints, and model evaluation.
1,024-Token Context Limit Bottleneck
Multi-document news clusters often exceed 4,000 words. Feeding uncompressed text truncates essential facts. Solved via query-guided sentence selection to prioritize high-signal content before feeding into BART.
Web Scraping Reliability & Fallbacks
News websites feature non-standard DOM structures, paywalls, and broken links. Engine built using trafilatura as primary extractor with BeautifulSoup fallback handling.
Dataset Cleaning & Paywall Removal
Raw scraping contained 404 pages, login forms, duplicate queries, and empty payloads. Designed a 7-stage filter pipeline cleaning 5,020 raw items down to 4,703 high-quality training instances.
Extractive vs Abstractive Balance
Navigating the dataset's ~57% unigram overlap to ensure keyword extractors guide BART without forcing verbatim copying or hallucinated information.
PROJECT CONTRIBUTIONS
End-to-end engineering deliverables implemented in this comparative NLP project.
TECHNOLOGY ARCHITECTURE
Explore the AQuaMUSE Codebase
View the repository, preprocessing scripts, scraping pipeline, and experiment notebooks on GitHub.