Draft:SmartXML

SmartXML
Stable release	1.0 / May 2024 (2024-05)
Written in	Red
Operating system	Cross-platform
Type	XML processing, data transformation
Website	redata.dev/smartxml

SmartXML is an advanced XML processing application developed in the Red programming language. It offers innovative solutions for working with XML data, enabling users to handle complex hierarchical data structures, classify documents, and transform data into formats suitable for databases or applications.

Application use a virtual DOM-like representation, that allow to processes XML files without requiring an XSD schema that helps to extract, classify, and transform data. It addresses challenges outlined in XPath and XPointer: Locating Content in XML Documents by John Simpson.

SmartXML supports integration with PostgreSQL, and implements proprietary parsing rules to prevent vulnerabilities such as XPath injection attacks.

Features

• Schema Independence: Builds a virtual DOM-like representation of XML data, enabling transformations into tabular or JSON formats without relying on predefined XSD schemas.
• Document Classification: Automatically classifies documents based on content, even without a fixed schema.
• Field Extraction Configuration: Allows users to flexibly configure the required fields for data extraction.
• Hierarchical Data Preservation: Generates SQL or JSON from XML, preserving hierarchical relationships for seamless database integration.
• Database Compatibility: Supports both relational databases (e.g., PostgreSQL) and NoSQL databases for data loading.
• Data Preprocessing with Built-In Grammars: Utilizes built-in grammars and lightweight natural language processing techniques for data cleansing and preprocessing.
• Batch Processing Mode: Efficiently handles large-scale data transformations.
• Secure Parsing Rules: Implements proprietary parsing rules to prevent vulnerabilities such as XPath injection attacks.

How SmartXML Works

SmartXML's processing pipeline includes the following steps:

1. Virtual DOM Representation: Parses XML data and converts it into a virtual DOM-like structure, enabling efficient manipulation and transformation.
2. Data Transformation: Transforms data into SQL or JSON formats while preserving complex hierarchical relationships.
3. Custom Extraction: Allows users to configure extraction rules, defining specific fields or nodes to process, making the tool adaptable to various datasets.
4. Database Integration: Loads transformed data into relational or NoSQL databases, supporting operations such as updates, inserts, and schema patches.

Advantages Over Alternatives

Comparison of SmartXML with Other XML Processing Tools

Feature	SmartXML	Alternatives
Schema Independence	Virtual DOM allows no-schema processing	Typically requires XSD
Document Classification	Automatically classifies and preprocesses documents	Rarely supported
Preserves Hierarchies	Generates SQL/JSON while maintaining hierarchical relationships	Flattening may lose structure
Built-in Grammars (Tiny NLP)	Built-in support for preprocessing with lightweight NLP grammars	Requires external libraries
Security	Proprietary parsing rules prevent XPath injection vulnerabilities	Relies on standard XPath handling

Practical Applications

• Database Integration. Organizations managing large XML datasets can use SmartXML to integrate data into their databases (such as PostgreSQL), ensuring hierarchical relationships remain intact.

• Document Classification. Companies processing diverse XML documents can classify them dynamically, even when schemas vary or are entirely absent.

• Patch Management. SmartXML enables the generation of database schema patches directly from XML, simplifying system updates and migrations.

See Also

• XML transformation
• Query languages
• Parsing expression grammar
• XPath