AST AI Agents - Crawl Data & AI Data Extraction | AI Data Collection Case Study by Relipa

Compared to common tools and extensions that rely on screenshot capture and AI reading of displayed content, AST AI Agents was built around multi-source data collection with direct API connections to e-commerce platforms. Integrating directly with the Rakuten API and Amazon API allows the system to proactively collect data on a set schedule and keep the database updated with the latest information, so product data is maintained over time rather than only reflecting what is displayed at a single moment. Pulling data directly from marketplace APIs also lets the system retrieve more product information than approaches that only read what is visible on screen. Catalog PDF support is included as well, letting users upload documents for automatic extraction, though this capability alone is not the main differentiator compared to other tools. 

Screenshot-based collection tools can only capture what is currently displayed on screen, limiting both the amount of information retrieved and the ability to track changes over time. To solve this, Relipa built direct API integrations with Rakuten and Amazon, allowing the system to pull richer product data than screen-reading approaches and proactively refresh that data on a set schedule. Rakuten roughly every 3 hours, Amazon through a daily batch – rather than only reflecting a single moment in time. 

Product data collected from different sources rarely shares the same structure. Relipa addressed this by building an AI extraction layer that converts inconsistently formatted product information into structured data fields, with the schema configurable by product category – for example, computers, measuring instruments, or furniture – so each category can use the fields relevant to it. 

Catalog PDFs vary widely in complexity, from simple text layouts to scanned pages with no extractable text. To handle this, Relipa designed the system to select a processing method based on document complexity: simpler content is processed with faster, lightweight methods, while complex or scanned pages are handled using AI to read the image content directly. 

Reprocessing an entire dataset because of a single failure is costly and slow. To avoid this, Relipa built a mechanism that splits data into smaller units and reprocesses only the segments that failed, rather than rerunning the full pipeline – improving both processing efficiency and resilience. 

The same product can appear across multiple pages or listings with slightly different information. Relipa implemented logic to recognize when data refers to the same product across sources, merge it into a single complete record, and remove duplicate entries – resulting in a cleaner, more reliable dataset. 

Not every image collected during crawling is relevant to a product. To reduce unnecessary processing, Relipa added an AI-based image classification step that identifies which images are actually product-related before they move further down the pipeline. 

To ensure the extracted data can be trusted, Relipa built a quality evaluation mechanism that compares processing results against reference data, providing a basis for identifying issues and continuously improving extraction accuracy. 

Keeping a multi-source, continuously running pipeline visible to the team was a key operational need. Relipa built a dashboard that surfaces crawl status, processing progress, and data statistics, giving the team clear visibility into how the system is performing at any given time.