Automate the rules before automating the upload
A supplier feed can arrive on time and still damage the catalogue. Prices may use the wrong tax basis, stock can refer to a warehouse that does not fulfil your orders and missing rows can mean several different things. Automating the transfer without defining those rules simply makes the mistakes happen faster.
At Midoriweb, we start with the meaning of the data. Which fields should the supplier control? Which decisions belong to your business? What should happen when a feed is incomplete, late or implausible? Once those questions have clear answers, the integration can make repeatable decisions and expose the exceptions that need attention.
This guide sets out a practical design for supplier feed automation. It complements our product import troubleshooting guide, which helps diagnose an import that already fails. Here, the aim is to build a process that remains understandable as suppliers, products and trading rules change.
Give every field an owner
Start with a mapping worksheet. For each destination field, record the supplier field, data type, transformation, scope and owner. A supplier's product code may map to a supplier reference rather than your Magento SKU. A cost price may inform a pricing rule without becoming the storefront selling price directly.
Agree whether the supplier may overwrite titles, descriptions, images and category assignments that your team has edited. A useful default is not universal; it depends on the catalogue and commercial agreement. What matters is that ownership is explicit and that the automation does not erase editorial work by accident.
Stock needs particular care. A supplier quantity is not automatically the quantity you can promise to customers. Consider allocation, dispatch lead time, other sales channels and how your Magento inventory setup represents sources. Adobe documents source-item imports separately from general product data; verify the mechanism against your installed version and configuration.
| Field | Define before automation |
|---|---|
| Supplier code | Stable relationship to your own SKU and variants. |
| Cost | Currency, tax basis and effective date; separate from selling price. |
| Quantity | Source, availability rules and maximum acceptable age. |
| Description | Whether supplier updates may overwrite editorial changes. |
| Missing row | Snapshot or delta; deletion, unchanged or review. |
Worked example: a supplier changes its file
Imagine a fictional supplier that sends a complete stock file every morning. Your mapping expects a stable supplier code and an integer quantity. One morning, a new export contains blank quantities for several products and omits a substantial part of the normal catalogue.
Treating blanks as zero might unnecessarily remove products from sale. Treating blanks as unlimited availability would be worse. Assuming every missing row means “discontinued” could deactivate an entire category. The correct response depends on whether the file is a complete snapshot, a set of changes or an incomplete export.
In this example, the agreed rule is to quarantine the suspicious batch and alert the responsible team. The last accepted data remains identifiable, and a separate stale-stock rule determines what customers can be offered while the issue is resolved. The system records the reason for rejection instead of reporting a successful import merely because the file could be opened.
Distinguish a complete snapshot from a delta
A complete snapshot claims to describe the whole agreed dataset at a point in time. A delta describes changes since another point. Absence has a different meaning in each. A product missing from a delta usually says nothing about its continued existence; a product missing from a complete snapshot may still require confirmation before removal.
Record a batch reference and the supplier's effective time where available. Define how out-of-order files are handled. A delayed file should not silently overwrite newer accepted stock or prices just because it arrived last.
For deltas, establish how missed events are recovered. Periodic reconciliation against an authoritative complete dataset can reveal gaps, but its frequency and behaviour need agreement. The integration should explain whether it is current, waiting for a batch or operating on stale information.
- 01ReceiveIdentify batch and effective time
- 02ValidateCheck format and business rules
- 03ApplyAccept valid data; isolate exceptions
- 04ReconcileCheck catalogue and storefront result
Validate meaning as well as format
File validation should check expected columns, encoding, identifiers and data types. Business validation should also check the values and relationships that matter to trading. A valid decimal can still be the wrong currency. A valid SKU can still be mapped to the wrong product.
Use agreed thresholds to identify unexpected changes, such as an unusually large catalogue reduction or a major price movement. These are review rules, not universal numbers to copy from another store. A seasonal supplier may legitimately change large parts of its range; a stable catalogue may justify much tighter checks.
Decide whether an invalid row blocks the complete batch or is isolated while valid rows proceed. Either can be appropriate. If rows are processed independently, the result must show accepted, rejected and unchanged records clearly. Staff need to know whether the catalogue represents a complete update or a partial one.
Keep commercial pricing separate from source cost
If the feed provides cost, define how selling prices are calculated and approved. Record currency, tax basis, rounding, effective dates and any product-specific exceptions. A percentage rule that ignores these details can produce a price that looks plausible while undermining the intended margin.
Decide how promotions and manually agreed prices interact with supplier updates. A routine feed should not erase a planned campaign unless that behaviour is intentional. Test the product page, basket and completed order calculation under the relevant customer and store-view conditions.
Our price synchronisation guide goes deeper into tracing a mismatch. For an automation project, the important preparation is to identify the authoritative value and the rules that transform it before it reaches the customer.
Make retries and reconciliation visible
Give each accepted batch a durable identity and record what it changed. If a process stops part-way through, the recovery mechanism should know which operations completed. Reprocessing should not create duplicate products or repeatedly apply a transformation to an already transformed value.
Keep the original input according to an agreed retention policy and record the mapping version used. This lets a developer distinguish a supplier change from a transformation defect. Avoid retaining unnecessary personal information or placing credentials in feed URLs, logs or support messages.
Reconcile the business result, not just the job status. Compare representative product values, source quantities and storefront availability against the accepted data. A green scheduler entry means a process ran; it does not prove that the customer sees the correct price or delivery promise.
Launch one controlled feed before widening coverage
Start with a representative subset that includes normal products and important exceptions: variants, discontinued items, missing data and products with manual overrides. Agree the expected result before running the import. Verify it in an appropriate test environment and retain the evidence.
For the live transition, identify the last accepted manual update and the first automated batch. Define who monitors the initial runs, how suspicious batches are stopped and how incorrect changes would be reversed. Restoring a file is not enough if another process has already used its values; rollback needs to consider the downstream effect.
Our ecommerce integrations service covers supplier feeds as part of the wider operational flow. Bring a representative file, the supplier's field definitions and examples of the decisions your team currently makes by hand. That gives us a much stronger starting point than “please automate this CSV”.
Technical references
Use the documentation that matches your installed version and deployment.