Textual process discovery is a method of process discovery that takes process descriptions as input to construct process models automatically. Textual process discovery is at the core of the services provided by DCR Solutions, a business process management company, which employs a fine-tuned BERT model to identify and annotate process elements in process descriptions, allowing for the creation of declarative process models.
In a previous study, DCR Solutions' BERT model's process element annotation capabilities were tested with promising results. The recent release of OpenAI's widely lauded ChatGPT presents an opportunity to investigate the affordances of a state-of-the-art, pre-trained model against an established, fine-tuned model. This comparison examines ChatGPT's potential as a cost-effective, accessible, and user-friendly alternative to existing process management tools.
This paper presents a within-subjects designed experiment, comparing the process ele-ment annotation capabilities of ChatGPT and DCR Solutions' fine-tuned BERT model. The results are based on a comparison of the precision, recall, and F1-scores achieved by the two models. The results show that ChatGPT outperforms the BERT model, scoring significantly higher in precision and Fl-score, suggesting it to be a more viable model for process element annotation and textual process discovery.
The study furthers the understanding of ChatGPT's potential as a process element annotation tool, and its potential applications in related fields, by discussing the implications of fine-tuning, temperature parameters, and prompt engineering techniques.
