Computer Science Diagram Understanding with Topology Parsing

Wang, S; Zhang, L; Luo, X; Yang, Y; Hu, X; Qin, T; Liu, J

Computer Science Diagram Understanding with Topology Parsing

Wang, S Zhang, L Luo, X Yang, Y

Hu, X Qin, T Liu, J

Permalink

Publisher:: Association for Computing Machinery (ACM)
Publication Type:: Journal Article
Citation:: ACM Transactions on Knowledge Discovery from Data, 2022, 16, (6), pp. 1-20
Issue Date:: 2022-07-30

Closed Access

	Filename	Description	Size
	Computer Science Diagram Understanding with Topology.pdf	Published version	3.49 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Wang, S
dc.contributor.author	Zhang, L
dc.contributor.author	Luo, X
dc.contributor.author	Yang, Y https://orcid.org/0000-0002-0512-880X
dc.contributor.author	Hu, X
dc.contributor.author	Qin, T
dc.contributor.author	Liu, J
dc.date.accessioned	2023-04-13T02:46:49Z
dc.date.available	2023-04-13T02:46:49Z
dc.date.issued	2022-07-30
dc.identifier.citation	ACM Transactions on Knowledge Discovery from Data, 2022, 16, (6), pp. 1-20
dc.identifier.issn	1556-4681
dc.identifier.issn	1556-472X
dc.identifier.uri	http://hdl.handle.net/10453/169730
dc.description.abstract	Diagram is a special form of visual expression for representing complex concepts, logic, and knowledge, which widely appears in educational scenes such as textbooks, blogs, and encyclopedias. Current research on diagrams preliminarily focuses on natural disciplines such as Biology and Geography, whose expressions are still similar to natural images. In this article, we construct the first novel geometric type of diagrams dataset in Computer Science field, which has more abstract expressions and complex logical relations. The dataset has exhaustive annotations of objects and relations for about 1,300 diagrams and 3,500 question-answer pairs. We introduce the tasks of diagram classification (DC) and diagram question answering (DQA) based on the new dataset, and propose the Diagram Paring Net (DPN) that focuses on analyzing the topological structure and text information of diagrams. We use DPN-based models to solve DC and DQA tasks, and compare the performances to well-known natural images classification models and visual question answering models. Our experiments show the effectiveness of the proposed DPN-based models on diagram understanding tasks, also indicate that our dataset is more complex compared to previous natural image understanding datasets. The presented dataset opens new challenges for research in diagram understanding, and the DPN method provides a novel perspective for studying such data. Our dataset can be available from https://github.com/WayneWong97/CSDia.
dc.language	en
dc.publisher	Association for Computing Machinery (ACM)
dc.relation.ispartof	ACM Transactions on Knowledge Discovery from Data
dc.relation.isbasedon	10.1145/3522689
dc.rights	info:eu-repo/semantics/closedAccess
dc.subject	0801 Artificial Intelligence and Image Processing, 0806 Information Systems
dc.subject.classification	Artificial Intelligence & Image Processing
dc.title	Computer Science Diagram Understanding with Topology Parsing
dc.type	Journal Article
utslib.citation.volume	16
utslib.for	0801 Artificial Intelligence and Image Processing
utslib.for	0806 Information Systems
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
utslib.copyright.status	closed_access	*
dc.date.updated	2023-04-13T02:46:36Z
pubs.issue	6
pubs.publication-status	Published
pubs.volume	16
utslib.citation.issue	6

Abstract:

Diagram is a special form of visual expression for representing complex concepts, logic, and knowledge, which widely appears in educational scenes such as textbooks, blogs, and encyclopedias. Current research on diagrams preliminarily focuses on natural disciplines such as Biology and Geography, whose expressions are still similar to natural images. In this article, we construct the first novel geometric type of diagrams dataset in Computer Science field, which has more abstract expressions and complex logical relations. The dataset has exhaustive annotations of objects and relations for about 1,300 diagrams and 3,500 question-answer pairs. We introduce the tasks of diagram classification (DC) and diagram question answering (DQA) based on the new dataset, and propose the Diagram Paring Net (DPN) that focuses on analyzing the topological structure and text information of diagrams. We use DPN-based models to solve DC and DQA tasks, and compare the performances to well-known natural images classification models and visual question answering models. Our experiments show the effectiveness of the proposed DPN-based models on diagram understanding tasks, also indicate that our dataset is more complex compared to previous natural image understanding datasets. The presented dataset opens new challenges for research in diagram understanding, and the DPN method provides a novel perspective for studying such data. Our dataset can be available from https://github.com/WayneWong97/CSDia.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/169730