In modern IT businesses, the effective management of service provision is crucial for maintaining operational efficiency. IT Service Management (ITSM), a subset of Service Science, focuses on service-related aspects of IT business. The most widely used framework within ITSM is the Information Technology Infrastructure Library (ITIL), which guides IT service processes. As noted by Gulap et al. [1], ITSM, as defined in ITIL, serves as both a glossary for standardizing terminology and a set of processes designed to implement IT best practices. This report focuses on Incident Management (IM), the process of addressing and resolving user-reported issues. IM is integral to ensuring efficient service delivery, from issue identification to resolution, as outlined by Gulap et al. [1] and Baresi et al. [2].

In practice, a user contacts the service department through email or phone, and a unique ticket is generated for each issue. This ensures that multiple issues can be managed simultaneously without confusion [3]. According to the ITIL framework, tickets are first addressed by "1st line support" (Service Desk) agents who categorize the issue, after which it is escalated to the appropriate "2nd line support" agent for resolution [3,4]. ITIL defines a series of support levels, resembling a decision tree, to streamline the process and manage large volumes of tickets efficiently [5]. This structure is essential for maintaining service quality while improving operational efficiency and generating valuable data for continuous improvement [1].

The rising volume of service tickets, combined with repetitive categorization tasks, has drawn attention to the potential benefits of Artificial Intelligence (AI) in ITSM. AI’s application in automating ticket classification could alleviate the manual workload, thereby improving operational efficiency [3]. Commercial AI-assisted ticketing solutions have been developed, but the full integration of AI within ITSM remains an underexplored area, particularly in smaller IT businesses. While larger IT service providers benefit from AI models tailored to their extensive service desks, smaller firms face challenges in adopting AI due to resource constraints and the lack of a primary focus on Incident Management [4,5].

Research in the AI-ITSM domain has introduced automated methodologies to enhance AI models’ precision in ticket classification and resolution [3]. However, smaller IT businesses may lack the resources to implement these extensive solutions. This study addressed this gap through a case study of a niche IT service provider operating in the nautical tourism sector. This sector faces high issue diversity and resource limitations that hinder the implementation of comprehensive AI-driven ticket quality improvement efforts [2,3]. The research explored how small-scale IT service providers can overcome these challenges by employing cost-effective AI solutions for improving ticket classification and streamlining service processes.

AI implementation in ITSM is highly dependent on the quality of the data used to train the models. Batini et al. [5] highlight that data quality is a fundamental aspect of information systems research, as poor-quality data leads to significant operational costs. Each ticket generated under the ITIL framework provides critical data for continuous service improvement [1]. However, this data is often incomplete or inaccurate, limiting its effectiveness in AI model training [4]. Reinhard et al. [3] emphasize that support agents often provide insufficient descriptions of issues due to time pressures or convenience, undermining the data quality that AI models require for accurate ticket classification. Thus, enhancing data quality is essential for the successful deployment of AI in ITSM.

The quality of the data directly influences the AI model’s ability to perform accurate ticket classification [4]. Heinrich et al. [4] demonstrated that low-quality data, particularly incomplete or inaccurate ticket descriptions, negatively impacts AI model performance in the context of recommender systems. This study addressed this issue by evaluating cost-effective methods for improving data quality, exploring how small businesses with limited resources can enhance the training datasets used for AI classification. By improving data quality, this research provided practical solutions for the implementation of AI in ITSM, enabling smaller IT businesses to effectively utilize AI-driven tools to optimize service delivery and incident resolution, even under resource constraints [5].

This research addressed the gap in the AI-ITSM literature by proposing simple yet effective data quality improvement strategies for small businesses. These strategies include inexpensive methods to enhance the accuracy and completeness of ticket data, which is crucial for AI model performance. By testing and evaluating these methods, this study provided insights into how small and medium-sized IT service providers can implement AI-driven solutions without the need for extensive resources. The outcomes of this case study aimed to assist smaller enterprises in utilizing AI to improve operational efficiency, reduce costs, and enhance customer satisfaction through more accurate and efficient incident management [4].

Statement of the Problem

Effective incident management is a crucial aspect of IT Service Management (ITSM), ensuring service quality and operational efficiency. Incident Management (IM), which covers the entire lifecycle of a user-reported issue, has traditionally relied on the manual categorization of service tickets by IT agents, following the guidelines set by the Information Technology Infrastructure Library (ITIL). However, this traditional approach has become increasingly inefficient, particularly as ticket volumes rise, leading to time-consuming processes and a higher likelihood of errors [3-5]. The advent of Artificial Intelligence (AI) presents a transformative opportunity to automate the ticket categorization and handling process, potentially eliminating the need for manual intervention and enhancing operational efficiency [1,2] While the potential of AI in ITSM, particularly in incident management, has been widely acknowledged, its practical application has remained complex and underexplored [3,5]. Despite advancements in AI-assisted ticketing systems, smaller IT service providers in niche sectors face unique challenges in adopting and implementing AI-driven solutions due to resource limitations and a lack of comprehensive strategies. Therefore, this study addressed these gaps by examining the feasibility and effectiveness of implementing AI for incident management within resource-constrained environments, offering insights into scalable AI solutions for smaller enterprises.

Research Questions

To address the challenges and explore the potential of AI in improving IT incident management, this study aims to answer the following research questions:

1. What dataset enhancement methodologies are known in prior research on AI-Incident Management that can improve the AI classification of tickets?
2. How can these methodologies be translated for small and medium-sized businesses with limited resources to improve ticket AI classification?

By answering these questions, the study aims to provide practical insights and solutions for small to medium-sized IT service providers looking to leverage AI technologies for enhanced incident management.

Academic Relevance

Research on Artificial Intelligence (AI) within the IT Service Management (ITSM) domain is currently sparse, particularly concerning the challenges posed by datasets in need of enhancement. Existing literature, such as that by Cai & Zhu [6], emphasizes the critical role of data quality in AI applications across various domains. Heinrich et al. [4] further highlight how data quality impacts AI performance specifically within ITSM contexts. This study aimed to address and enhance data quality issues prevalent in imperfect datasets, echoing methodologies explored by Baresi et al. [2] and Reinhardt et al. [3] through statistical analyses. However, there remains a gap in exploring simpler yet effective methodologies to enhance data quality specifically tailored for the ITSM field.

Practical Relevance

In this case study, the primary objective was to leverage AI technologies to alleviate the manual workload associated with Incident Management processes. While many aspects of IT businesses are scalable, the handling of service requests often requires intensive human intervention. Solutions like FAQ lists and AI chatbots offer potential for reducing this workload in generic IT service management scenarios but may fall short in specialized sectors. The niche IT service provider in this study operates within the nautical tourism sector, where resolving technical issues demands highly specialized expertise, contributing to high operational costs.

The focus within Incident Management is on ticket categorization, a task critical for generating essential data under the ITIL framework. Each ticket must be meticulously categorized to facilitate efficient IT service management and continuous improvement processes. The current manual categorization process, amidst a high volume of incoming tickets, underscores the need for automation through AI. However, the effectiveness of AI in accurately categorizing tickets hinges on the quality of the underlying data. In this case study, the dataset suffers from data quality issues, which poses a significant barrier to successful AI implementation. Systematic improvements aimed at enhancing data quality are therefore paramount to enabling an effective AI model capable of reducing manual workload through accurate ticket categorization.

The relevance of this research extends to exploring methodologies for enhancing data quality specifically tailored for AI implementation in IT incident management. The resource constraints faced by smaller IT service providers, such as limited budgets for outsourcing AI initiatives, add complexity to the adoption of sophisticated AI-driven solutions. This constraint is particularly pertinent in sectors like nautical tourism, where project-based service delivery may circumvent comprehensive ITIL practices, further limiting investment in AI implementation efforts. Thus, this study aimed to bridge the gap between theoretical advancements in AI-driven ITSM and practical implementation in resource-constrained environments.

The primary objective of this case study was to illuminate the foundational challenges associated with AI implementation within service delivery processes governed by the ITIL framework, while also evaluating enhancement methodologies proposed in the AI-ITSM research domain. To achieve this, a comprehensive review of current theoretical frameworks and empirical research across IT Service Management (ITSM) and Artificial Intelligence (AI) domains is imperative. This review encompasses defining and exploring dimensions of data quality, methodologies for empowering AI through data quality enhancement, and techniques for evaluating AI model performance.

It Service Management (ITSM)

IT Service Management (ITSM) is defined as a strategic approach to managing IT operations with a focus on delivering IT services that meet the needs of customers and align with business goals [7]. Central to ITSM is the Information Technology Infrastructure Library (ITIL), a comprehensive framework comprising best practices and processes aimed at optimizing service delivery and enhancing operational efficiency through systematic data collection. ITIL integrates closely with ticketing systems tailored for IT service management processes, facilitating multitasking by service agents and enabling the collection of vital metrics for service improvement.

Figure 1: Overview of ITSM Processes (ISO/IEC 20000 1, 2005).

Figure 1 provides an overview of the structured ITSM processes, illustrating the systematic approach advocated by standards such as ISO/IEC 20000 1 (2005). This case study specifically focuses on implementing AI within Incident Management (IM) processes, aiming to streamline resolution procedures and enhance service delivery.

The Importance and Challenges of Ticket Classification

Ticket classification stands as a foundational process within the ITIL framework's incident resolution process, serving as the initial step to translate service requests into actionable data for continuous service improvement. This critical stage not only supports efficient incident management but also significantly impacts the efficacy of AI-driven enhancements aimed at automating and streamlining IT service operations. In IT service management (ITSM), ticket classification plays a crucial role in organizing and prioritizing incoming service requests based on their nature and severity [7]. By categorizing tickets accurately, IT organizations can ensure timely responses and resolutions, thereby enhancing customer satisfaction and operational efficiency. This process is essential for generating structured data that informs decision-making and supports the continuous improvement of service delivery processes within the ITIL framework.

The quality of data generated through ticket classification is paramount not only for manual service management but also for training AI models designed to automate and optimize incident resolution processes. Research underscores that AI models heavily rely on high-quality data inputs to achieve accurate classification and predictive capabilities [4]. Therefore, any inaccuracies or inconsistencies in ticket classification can propagate through AI training datasets, leading to compromised model performance and reduced effectiveness in handling real-time service incidents.

AI-driven ticket classification encounters several inherent challenges rooted in the variability and complexity of textual data submitted by service requesters. Unstructured inputs, often plagued by spelling errors, abbreviations, and vague descriptions, pose significant hurdles for AI algorithms tasked with interpreting and categorizing issues [2]. The lack of specificity in initial service requests requires IT agents to engage proactively, seeking additional information to refine ticket classifications accurately. For example, a service requester reporting "laptop not working properly" presents a broad issue that could stem from hardware malfunctions, software glitches, network connectivity issues, or authentication failures [7]. Without detailed clarification from the requester, IT agents must rely on their expertise to deduce the underlying cause and categorize the ticket accordingly. This process demands not only technical proficiency but also effective communication skills to ensure precise ticket classifications that align with ITIL standards and service level agreements (SLAs).

Effective AI empowerment in incident management hinges on robust strategies to enhance data quality throughout the ticket classification process. Pre-processing techniques, such as text normalization and error correction, play a crucial role in standardizing textual inputs before they are fed into AI models (Reinhardt et al., 2023). By cleansing and structuring unstructured data, IT organizations can improve the signal-to-noise ratio in AI training datasets, thereby enhancing the accuracy and reliability of automated ticket classification systems. Furthermore, continuous validation and refinement of AI training sets are essential to mitigate the impact of human errors and inconsistencies in manual ticket classifications. Iterative feedback loops allow IT organizations to identify and rectify misclassifications, ensuring that AI models evolve and adapt to changing service demands and operational environments [4].

Artificial Intelligence Categories

Artificial Intelligence (AI) plays a pivotal role in enhancing IT incident management by automating processes and improving customer experience through streamlined service delivery. Within AI, text data classification is fundamental for efficiently categorizing service tickets and prioritizing responses. This section explores three key classification algorithms-Bayesian classifier, decision tree classifier, and artificial neural networks (ANN)-and their relevance to AI-driven incident management.

a. The Bayesian Classifier

The Bayesian classifier applies probabilistic principles to classify text data based on the occurrence and weights of features (words) within the text [7]. For instance, in the context of IT incident management, the classifier analyzes words like "network issue" or "software malfunction" to probabilistically determine the most likely category of the service ticket. By calculating conditional probabilities and updating probabilities as new evidence (textual features) emerges, the Bayesian classifier adapts to classify issues accurately. In practice, the Bayesian approach allows IT service desks to automate the initial categorization of tickets, reducing manual effort and accelerating response times [4]. This automation is particularly effective in handling large volumes of service requests, where quick and accurate classification is crucial for maintaining service levels and enhancing operational efficiency.

b. The Decision Tree Classifier

The decision tree classifier operates like a flowchart, using decision points based on the features of the text data to classify issues (ISO/IEC 20000-1, 2005). It can handle both categorical and numerical data, making decisions based on the presence, absence, or specific values of certain features. For example, if a ticket mentions slow internet speeds but not complete connectivity loss, and this issue affects multiple users, the decision tree might classify it as a "network congestion" problem if the speed falls below a predefined threshold. However, the effectiveness of decision tree classifiers in AI-driven incident management hinges on the quality and completeness of textual data. Unclear or incomplete descriptions can lead to misclassification, highlighting the importance of data preprocessing and feature selection to optimize decision tree performance [3].

c. Artificial Neural Networks (ANN)

Artificial Neural Networks (ANNs) are versatile classifiers inspired by biological neural networks, designed to learn and recognize patterns in data [2]. ANNs consist of interconnected nodes organized in layers-input, hidden, and output-where each connection (synapse) carries a weight that adjusts during training to optimize classification accuracy. In the context of IT incident management, ANNs excel in handling complex and unstructured textual data, leveraging deep learning techniques to extract meaningful patterns from service tickets [7]. However, the complexity of ANNs also presents challenges, including the "black box" nature where the internal decision-making process isn't easily interpretable. Despite this, ANNs offer flexibility in handling diverse datasets and improving classification accuracy over time through continuous learning and adaptation.

d. Data Quality

Data quality stands as a pivotal factor influencing the efficacy of AI models in IT incident management. In this research context, data quality refers to the fitness for use of the datasets used to train AI models, crucial for predicting and resolving IT issues promptly and accurately. According to Cai and Zhu [6], accurate and high-quality data are essential for deriving meaningful insights and value from big data analytics. Wang and Strong [8] define data quality as the usability and reliability of data, highlighting its critical role in the effectiveness of AI applications within IT service management frameworks.

Figure 2: Two-Layer Big Data Quality Standard by [6].

Figure 2 illustrates a comprehensive conceptual overview of data quality dimensions proposed by Cai and Zhu [6], encompassing aspects such as completeness, accuracy, relevance, and timeliness. In the realm of IT incident management, not all dimensions are equally applicable, with a specific focus on reliability and relevance. These dimensions dictate that AI models trained on IT service tickets must prioritize data entries that demonstrate high reliability in issue descriptions and relevance to incident management processes. Ensuring these criteria are met enhances AI model accuracy and efficiency in incident resolution.

e. AI Empowerment Methodologies

In the realm of AI-driven IT incident management, the efficacy of AI models hinges significantly on the quality of the data they are trained on. Effective methodologies for enhancing AI capabilities, pioneered by Baresi et al.[2] and Reinhardt et al. [3], offer valuable frameworks for improving data quality and subsequently optimizing AI-ITSM (IT Service Management) workflows. However, these methodologies often require extensive data science resources, posing challenges for smaller enterprises with limited budgets.

Reinhardt et al. [3] introduce a systematic ticket analytics pipeline designed specifically to enhance AI-ITSM models within incident management frameworks. This pipeline, articulated through five sequential data analytical steps (DR1-5), serves to refine data quality and enhance the overall performance of AI models deployed in incident resolution.

DR1 initiates the process by defining and characterizing key data quality metrics tailored to issue descriptions and resolution narratives. This stage ensures that the training dataset for AI models is populated with entries that meet stringent reliability and relevance criteria. By focusing on these metrics, Reinhardt et al. [3] underscore the importance of starting with high-quality data inputs to foster accurate model predictions and actionable insights in real-world IT incident scenarios.

Following DR1, DR2 involves preprocessing textual data extracted from incident reports. This crucial step involves cleaning and standardizing input data by removing irrelevant elements such as hyperlinks, email signatures, and special characters. By streamlining the textual data, DR2 prepares the dataset for subsequent analysis and model training, mitigating noise and ensuring consistency in data inputs.

DR3 leverages topic clustering techniques to categorize and organize incident tickets based on thematic relevance. This clustering process enhances dataset coherence by identifying and retaining only those tickets that contribute meaningfully to AI model training. Redundant or irrelevant tickets are systematically removed, optimizing the dataset's utility and relevance in enhancing AI-driven incident management capabilities.

In DR4, Reinhardt et al. [3] introduce a scoring mechanism that evaluates each ticket across multiple dimensions of data quality. Metrics such as accuracy, completeness, and timeliness are assessed to assign scores that prioritize tickets with high-quality data inputs. This iterative process ensures that AI models are trained on datasets enriched with reliable and pertinent information, thereby enhancing their predictive accuracy and operational effectiveness in handling IT incidents.

Finally, DR5 concludes the pipeline by assigning normalized scores to each ticket, reflecting its overall contribution to enhancing AI model performance. By quantifying the usefulness of each ticket based on aggregated data quality scores, DR5 facilitates informed decision-making in dataset refinement, ensuring that only the most valuable entries are retained for AI model training purposes.

Complementing Reinhardt et al. [3], approach, Baresi et al. [2] present the ACQUA methodology, a comprehensive framework aimed at predicting and improving ticket data quality through advanced statistical analyses. ACQUA employs a 15-step process to assess initial data quality directly from issue descriptions, employing metrics such as text length, content relevance, and syntactic complexity. This deductive approach enables organizations to preemptively filter out low-quality data entries, thereby optimizing the dataset's suitability for AI model training.

The ACQUA methodology's emphasis on initial data quality assessment aligns with best practices in data-driven AI model development, ensuring that only high-quality inputs are integrated into the training process. By leveraging statistical metrics and deductive reasoning, ACQUA provides a robust framework for enhancing data quality within incident management datasets, setting a precedent for rigorous data governance practices in AI-driven ITSM environments.

However, while these methodologies offer comprehensive frameworks for enhancing AI capabilities in incident management, their reliance on extensive data science expertise and resources may limit their accessibility to smaller organizations. The challenge remains in adapting and simplifying these methodologies to suit the resource constraints of small to medium-sized enterprises, thereby democratizing access to advanced AI-driven solutions in IT incident management.

f. The Gap In The Existing Research In AI-Incident Management

The Despite advancements by Baresi et al. [2] and Reinhardt et al. [3], significant gaps persist in accessible methodologies for improving data quality in AI-driven incident management. Existing approaches often require extensive data science expertise and resources, limiting their applicability to smaller enterprises with constrained budgets. Heinrich et al. [4] underscore the correlation between data quality and AI performance, yet accessible strategies for enhancing data quality remain underexplored in current literature.

This research aims to address this gap by identifying alternative methods to improve data quality in AI training datasets, specifically tailored for small to medium-sized businesses. By developing more accessible methodologies that prioritize simplicity and effectiveness, this study seeks to empower organizations with limited resources to enhance AI-driven incident management capabilities. By bridging this knowledge gap, organizations can leverage AI more effectively to optimize service delivery, reduce operational costs, and enhance overall customer satisfaction in IT service management.

To address the research questions concerning dataset enhancement methodologies in AI-driven incident management and their adaptation for small and medium-sized enterprises (SMEs) with limited outsourcing resources, a quantitative research approach was undertaken. This methodology aimed to evaluate and enhance the data quality of ticketing system datasets to improve AI classification accuracy, thereby enhancing customer experience through automation and streamlined processes.

Data Collection

The dataset utilized in this research was sourced from the ticketing system backlog of an IT service provider operating within the nautical tourism sector. Comprising over ten thousand entries, each representing a unique IT service ticket, the dataset included initial text data triggering each ticket and their manual classifications. Stakeholders from the company confirmed the dataset's low quality, attributing this to several factors. Firstly, the diverse nature of their client base-comprising personnel from various nationalities working onboard ships-contributed to incomplete, inaccurate, and often misspelled initial ticket descriptions. Secondly, resource constraints within the service desk, including limited personnel and time allocated to incident management (IM) functions, further exacerbated data quality issues. This situation led to a dataset plagued by misclassified tickets and contamination from irrelevant or improperly generated entries, hindering previous AI implementation attempts aimed at improving IM predictive capabilities and workload reduction.

The AI Model, Classifiers, and Evaluation

In this study, the orange data mining tool was employed for data processing, AI model creation, and performance evaluation of various data quality enhancements. While not directly integrable into operational ticketing applications, Orange facilitated a structured approach to conceptualizing and comparing different methodological steps in isolation. Key functionalities of Orange, such as preprocessing capabilities, bag-of-words analysis, and diverse classifier options, were instrumental. Of particular significance was its ability to generate performance metrics like confusion matrices, crucial for assessing the effectiveness of AI models trained on datasets processed with different enhancement methodologies.

The Data-Quality Enhancement Methodologies

Drawing from prior research on AI-driven incident management, two primary methodologies for enhancing data quality were adapted and tested within the constraints of limited outsourcing resources:

Reinhardt et al.'s Methodology: Reinhardt et al. [3] proposed a methodology centered on scoring each data entry based on predefined data quality dimensions. Entries with low scores, indicative of poor quality based on metrics like accuracy and relevance, were filtered out to create an enhanced dataset. This approach emphasized rigorous data analysis and model comparison to demonstrate performance improvements between the native and enhanced AI models.

Baresi et al.'s Methodology: In contrast, Baresi et al. [2] focused on predictive methods to preemptively assess data quality using statistical metrics derived from initial issue descriptions. This approach involved scoring text data early in the preprocessing stage to optimize predictive model performance, aiming to filter out low-quality entries before formal model training.

Given resource constraints prohibiting extensive data analysis typical of these methodologies, this study adapted a practical approach leveraging domain expertise from the nautical tourism sector. Expert knowledge was utilized to subjectively preselect and filter out entries deemed low-scoring in terms of data quality. This initial filtering step was followed by the application of various classifiers and preprocessing techniques to develop AI models using the minimally viable dataset version.

Subsequently, efforts were directed at optimizing the enhanced model further, focusing on achieving operational AI models capable of reliably classifying incident tickets in real-time scenarios. Evaluation criteria included comparing the performance metrics of AI models trained on the enhanced dataset against those trained on the native dataset with minimal preprocessing.

This section presents the process of developing AI models and their performance outcomes, specifically assessing the impact of data quality enhancement methodologies on prediction accuracy within the context of AI-driven enhancements in IT incident management aimed at improving customer experience through automation and streamlined processes.

       A) Establishing the Base Model

1. The Base Model

The initial step involved creating a minimally viable AI model using the orange data mining tool with the native dataset. Figure 3 illustrates the layout of the base model, structured to preprocess and analyze text data extracted from IT service tickets:

Figure 3: Base Model Layout Using The Native, UN Enhanced, Data Set.

Starting with the "Native" Corpus widget, representing the unaltered dataset, the Preprocess Text and Bag of Words widgets were employed to standardize and transform text features into semantic representations. This preprocessing step was crucial for extracting meaningful insights from the often unstructured and varied text data present in IT service tickets. Utilizing a Word Cloud visualization allowed for the identification and exclusion of non-informative words such as greetings and salutations, thereby enhancing the relevance of the processed data. The Select Columns widget facilitated the designation of issue categories as the prediction variable, essential for training the model to classify incoming tickets accurately. Subsequently, the Data Sampler widget partitioned the dataset into distinct training and test sets, enabling robust performance evaluation through the Predictions and Confusion Matrix widgets. See Figure 4 for the performance metrics of the base model. The main metric of importance is Classification Accuracy (CA). The performance metrics also showed that the most viable classifier for this research was the Artificial Neural Network classifier, and therefore it was used in all further models.

Figure 4: Performance Evaluation and Methodology Selection.

In evaluating the base model, the Bag of Words feature creation method was chosen over Document Embedding due to superior performance metrics across various classifiers tested, prominently the Artificial Neural Network. The models utilizing Document Embedding exhibited lower performance metrics, attributed to the dataset's inherent complexities such as misspellings and synonyms, which undermined the efficacy of semantic clustering techniques.

1. The Enhanced Models

Building upon the base model framework, various data quality enhancement methodologies were implemented and evaluated against the native dataset to measure their individual and combined effects on model performance. Each enhancement sought to address specific challenges in data quality and prediction accuracy within the context of IT incident management.

Enhanced I: Email Signature Removal

Enhanced I focused on improving data cleanliness by removing email signatures from text data, aiming to eliminate irrelevant information that could distort predictive outcomes.

Figure 5: Email Signature Skim Formula Used In Enhanced II.

Figure 5 illustrates the formula used to skim email signatures; a process not native to the orange data mining tool but crucial for enhancing the relevance of text features. However, despite the intention to streamline text features and enhance prediction accuracy, Enhanced I resulted in a slight decrease in Classification Accuracy (CA) compared to the base model. This unexpected outcome suggested that email signatures might contain contextual clues linking ticket requesters to specific issue categories, inadvertently impacting the model's predictive efficacy.

Enhanced II: Semantic Category Combination

Enhanced II leveraged insights from the base model's confusion matrix to merge low-frequency and semantically similar issue categories. For example, categories such as "Servers" were combined with more prevalent categories like "Internet," refining the granularity of issue categorization. While this approach yielded a marginal improvement in model performance, its impact underscored the importance of semantic clarity in enhancing predictive accuracy.

Enhanced III: Expert-Guided Data Filtering

Enhanced III introduced a critical dimension to data quality enhancement by leveraging domain-specific expertise to filter out low-quality tickets. This manual duration process, informed by expert knowledge of IT service issues, significantly enhanced prediction accuracy by removing noise and irrelevant data points from the dataset. The successful application of expert knowledge validated its pivotal role in optimizing AI model performance in real-world operational contexts.

Enhanced IV: Combined Enhancements

Building on the successes of Enhanced II and III, Enhanced IV integrated these methodologies to achieve a synergistic enhancement approach. By combining refined category combinations with expert-filtered data, Enhanced IV demonstrated a cumulative increase in Classification Accuracy (CA) by nearly five percent compared to the native dataset model. The removal of low-quality entries and the consolidation of semantically related categories proved effective in further optimizing model predictions and enhancing overall operational efficiency in incident management [9-10].

Performance Evaluation Summary

Table 1: Provides A Comprehensive Summary Of The Performance Metrics Across Different Enhancement Methodologies Using The Neural Network Classifier.

These metrics highlight Enhanced III and IV as standout performers, demonstrating substantial improvements in Classification Accuracy (CA) compared to the baseline model. The findings underscore the efficacy of targeted data quality enhancements in optimizing AI-driven incident management processes and improving overall customer experience.

C. Additional Insights and Limitations

Beyond category prediction, attempts were made to extend AI automation capabilities to predict variables such as site location, assigned agent, and specific sub-categories within issue classifications. However, these efforts encountered significant challenges due to inherent data limitations, including insufficient textual information to reliably predict such variables. Future enhancements may focus on refining predictive variables and leveraging advanced AI techniques to overcome these challenges and further enhance operational efficiencies in IT incident management [11-14].

A. heoretical Contribution

The findings from this research demonstrated that significant improvements in AI prediction performance can be achieved even with an initially low-quality dataset by implementing targeted data quality enhancement methodologies. Enhancing the dataset through systematic changes, such as combining categories based on semantic meanings and filtering low-quality entries with domain-specific expertise, proved instrumental in enhancing the AI’s predictive capabilities. These results corroborate prior studies, such as Heinrich et al., which underscore the positive correlation between data quality improvements and AI performance.

The study revealed nuanced insights into methodology efficacy. While semantic category combination yielded modest performance gains, attempts to remove supposedly redundant data, such as email signatures, occasionally resulted in performance decreases. This discrepancy highlights the complex interplay between feature relevance and model performance, where seemingly extraneous data can still contribute to classification accuracy, particularly under the Neural Network classifier utilized here.

Moreover, the research addresses practical gaps in the literature by offering insights tailored to small and medium-sized businesses (SMBs) with limited outsourcing resources. By demonstrating the feasibility of enhancing AI classification in incident management despite data constraints, this study provides actionable strategies for SMB service managers seeking to implement AI-driven solutions effectively.

B. Practical Contribution

Despite incremental gains in prediction accuracy, the practical implications of these enhancements are significant, especially considering the volume of service tickets processed. SMB service managers can leverage accessible AI tools to implement the methodologies discussed in this research, thereby customizing AI models specific to their operational needs even with suboptimal datasets. Outsourcing AI development to external experts remains an alternative for achieving higher prediction performance, albeit at potentially higher costs, necessitating a trade-off analysis based on business priorities.

Furthermore, proactive measures to ensure data quality, such as implementing standardized ticket formats and data validation protocols, emerge as crucial strategies. These systematic changes not only enhance data completeness, consistency, and accuracy but also lay the foundation for sustained AI model effectiveness. However, the timeline for accumulating a sufficiently sized, high-quality dataset should be considered, as it may require years for SMBs with lower ticket volumes.

Effective AI implementation in incident management hinges on continuous data quality improvements post-implementation. Failure to integrate systematic changes could undermine AI’s ability to classify new, low-quality tickets accurately, emphasizing the ongoing need for data governance and quality assurance within SMB operations.

C. Future Research

Looking ahead, future research should delve deeper into systematic changes required to ensure sustained data quality improvements in ticket influxes. Specifically, exploring the impacts of semantic category combinations and other enhancement methodologies within different operational contexts will refine AI implementation strategies tailored to SMB environments. Understanding how process design within incident management frameworks influences AI effectiveness remains a critical yet underexplored area.

Additionally, extending the scope beyond email data to encompass other ticket submission methods, such as phone calls or online forms, is essential. SMBs often receive tickets through diverse channels, necessitating research into optimizing AI for multi-format data integration. Comparative studies across these formats will elucidate best practices for maximizing AI’s utility in diverse operational settings.

By addressing these avenues, future research can provide comprehensive guidance for SMBs seeking to harness AI-driven enhancements effectively in IT incident management, ensuring sustained improvements in customer experience and operational efficiency.

1. Galup SD, Dattero R, Quan JJ, Conger S. An overview of IT service management. Communications of the ACM. 2009; 52: 1.
2. Baresi L, Quattrocchi G, Tamburri DA, Van Den Heuvel W. Automated Quality Assessment of Incident Tickets for Smart Service Continuity. In Lecture notes in computer science. 2020; 492-499.
3. Reinhard P, Li MM, Dickhaut E, Peters C, Leimeister JM. Empowering Recommender Systems in ITSM: A Pipeline Reference Model for AI-Based Textual Data Quality Enrichment. In Lecture notes in computer science. 2023; 279-293.
4. Heinrich B, Hopf M, Lohninger D, Schiller A, Szubartowicz M. Data quality in recommender systems: The impact of completeness of item content data on prediction accuracy of recommender systems. Electronic Markets. 2019; 31: 389-409.
5. Batini C, Barone D, Mastrella M, Maurino A, Ruffini C. A framework and a methodology for data quality assessment and monitoring. In ICIQ. 2007; 333-346.
6. Cai L, Zhu Y. The Challenges of Data Quality and Data Quality Assessment in the Big Data Era. Data Science Journal. 2015; 14: 2.
7. Conger S, Winniford M, Erickson-Harris L. Service management in operations. Paper presented at the Fourteenth Americas Conference on Information Systems Toronto ON. Canada. 2008; 14-17.
8. Wang RY, Strong DM. Beyond Accuracy: What Data Quality Means to Data Consumers? J Management Info Sys. 1996; 12: 5-33.
9. Agarwal S, Aggarwal V, Akula AR, Dasgupta GB, Sridhara G. (2017). Automatic problem extraction and analysis from unstructured text in IT tickets. IBM J Res Dev. 2017; 61: 4:41-4:52.
10. International Organization for Standardization. ISO/IEC 20000-1:2005 Information technology - Service management - Part 1: Service management system requirements. ISO. 2005.
11. Koehler J, et al. (n.d.). Towards Intelligent Process Support for Customer Service Desks: Extracting Descriptions from Noisy and Multi-lingual Texts.
12. Marcuzzo M, Zangari A, Schiavinato M, Giudice L, Gasparetto A, Albarelli A. A multi-level approach for hierarchical Ticket Classification. In Proceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022). 2022; 201-214.
13. Revina A, Buza K, Meister VG. IT ticket classification: The simpler the better. IEEE Access. 2020; 8: 193380-193395.
14. Zicari P, Folino G, Guarascio M, Pontieri L. Discovering accurate deep learning based predictive models for automatic customer support ticket classification. In Proceedings of the 36th Annual ACM Symposium on Applied Computing. 2021.