Automating Legal Text Summarization with Multi-Agent Systems and Human-Centered Evaluation

Abstract

This research introduces an automated multi-agent system aimed at enhancing the summa rization and evaluation of legal documents, specifically judgments. Legal professionals often struggle to condense complex legal texts while retaining essential details. To address this, we developed a system using CrewAI that incorporates specialized agents for document analysis, summarization, and evaluation. The system uses predefined templates to extract key legal in formation, such as case numbers, involved parties, and court decisions, ensuring structured and consistent summaries. The summarization process involves agents like the Senior Document Analyst, responsible for extracting critical points using the template, and the Judicial Assis tant, which compiles comprehensive summaries. Two automated evaluation metrics, ROUGE score and BERT score, along with human evaluation, are employed to assess the summaries’ quality, focusing on clarity, language quality, and accuracy. The human evaluations by lawyers guarantee that the summaries are practical and useful for legal professionals. By combining template-based extraction with both automated and human evaluation, this multiagent system provides an efficient approach to legal judgment processing. Further, it also ensures that the high standards of legal precision and coherence are upheld.