Ole Germann Kristensen

Systems, Integration & Architecture

Tag: Architecture

  • Introducing the integration platform

    Introducing the integration platform

    When a warehouse update fails, someone has to decide whether to retry it, correct it, or wait for another system. That decision gets awkward when the only documentation is a scheduled task on a server and the person who wrote it is on holiday. This is the work I want an integration platform to make easier.

    For the fictional seafood company in this series, I would start with one flow: an accepted order in the ERP needs a stock reservation in the warehouse. It is a useful starting point because it has an identifiable business result, a failure that people care about, and a fairly clear owner on each side.

    Give the flow a contract and an owner

    I would write down what the ERP is promising when it emits OrderAccepted.v1. Has the customer been validated? Are quantities final? Can the order still change? The warehouse team needs those answers before it can implement a consumer. A JSON schema can check that quantity is a number. The teams still have to agree whether that number represents kilograms, boxes, or individual items.

    The event needs a stable eventId, an orderId, a schema version and a source revision. I would keep customer names and addresses out of this event unless the warehouse genuinely needs them. The integration repository should include an example message, schema, producing team, consuming team and recovery instructions. An owner should be a maintained team contact that operations can actually reach.

    {
  "eventId": "9f26c80e-5b91-438b-86c9-6f27a13f2195",
  "type": "OrderAccepted.v1",
  "orderId": "SO-1042",
  "revision": 3,
  "lines": [{ "itemId": "SALMON-BOX", "quantity": 12, "unit": "box" }]
}

    A small implementation I would start with

    In a Microsoft environment, one possible implementation is an ERP adapter, Azure Service Bus and a .NET worker that calls the warehouse API. Azure API Management can sit in front of HTTP endpoints where authentication policy, quotas and API publication are needed. It has a different job from the message broker. Adding it to an asynchronous flow needs a specific reason.

    If we can change the source application, I would write the order change and an outbox record in the same SQL transaction. A background relay sends committed outbox records to Service Bus. If the ERP is a packaged application, I would first investigate its supported event or change interface. Writing an outbox alongside a database we do not control requires care; an adapter cannot make two unrelated commits atomic just by calling them consecutively.

    The warehouse consumer receives a message under a lock, validates it and calls the reservation endpoint using a stable request identity. It completes the message after the warehouse has confirmed the result. A crash between the reservation and message completion causes a repeat delivery, so the warehouse endpoint needs to recognize the same reservation request. That agreement belongs in the contract.

    graph LR
 ERP[ERP transaction and outbox] --> RELAY[Outbox relay]
 RELAY --> BUS[Service Bus]
 BUS --> WORKER[Reservation worker]
 WORKER --> WH[Warehouse API]
 WORKER --> OPS[Outcome and error telemetry]

    Make uncertain outcomes visible

    Test the uncertain outcome

    A timeout is the case I would test early. The warehouse may have created the reservation just before the connection dropped. Retrying with a new identifier could reserve the stock twice. I would reuse the original identifier and, where supported, query the reservation status before another attempt. If the target supports neither idempotency nor status lookup, the recovery procedure needs a reconciliation step and probably an operator.

    Validation failures go to a controlled error path with a useful reason. Temporary connection failures get bounded retries with increasing delays. Repeated failures need an alert and an owner. A dead-letter queue is only useful if someone can explain why a message is there and what has to happen before it is safe to replay.

    What operations should be able to see

    For this flow, I want to search by orderId and see source acceptance, publication, receipt, reservation outcome and final completion. Each attempt should record duration, eventId and a technical error code. Full payload logging would expose more customer data than most investigations need.

    I would measure the age of the oldest unprocessed message as well as queue depth. The dashboard also needs the last successful reservation, dead-letter count and a link to the runbook. An example service target might be reservation confirmation within two minutes during warehouse operating hours; that target has to be agreed with the people using the result.

    Fifty messages arriving seconds ago and fifty messages stuck since yesterday need different responses.

    Build a repeatable way to add the next flow

    Once this path works, I would package its deployment, logging fields, health checks and contract tests so the next integration can reuse them. Bicep or Terraform can describe the broker resources and access assignments. The worker and schema live in version control, and a pipeline deploys the same artifact through test and production.

    The platform team can own that common setup while the warehouse team owns reservation rules. Keeping those responsibilities explicit helps avoid a central integration team becoming the accidental owner of every business decision. Some flows can remain simple scheduled transfers if their timing and recovery requirements allow it.

    Before calling the first flow ready, I would stop the worker during processing, send the same event several times, make the target time out and replay a corrected failed message. The expected warehouse state should be written down for each test. Those results give the next team something concrete to copy.

    For the broker behaviour behind this design, see Microsoft’s documentation on duplicate detection. Broker deduplication has a configured history window; the reservation endpoint still needs its own repeat-request handling.

  • Why Point to Point Integrations Break at Scale

    Why Point to Point Integrations Break at Scale

    A direct integration is often a reasonable first choice. If one system needs a small daily extract from another, a scheduled transfer can be easy to build and easy to explain. The trouble starts when more processes begin depending on that transfer and its original assumptions stay buried in the script.

    Consider a fictional seafood processor with an ERP, warehouse, production system, transport portal and reporting database. Sales uses warehouse stock to promise deliveries. Production updates the available lots. Quality approval determines which lots can ship. A stock export now carries several decisions that matter to people outside the warehouse team.

    Follow one stock update

    Suppose a PowerShell task exports stock every fifteen minutes and uploads a CSV to the ERP. At 18:00 on Friday the export fails. Both applications remain available, and the scheduler may even report success if the script caught an exception without returning a failing exit code. Sales keeps reading the last imported quantities.

    I would give that transfer a manifest containing an exportId, source timestamp, row count and checksum. The receiving job should record when it finished validating and applying the export.

    A successful file upload tells us about transport; we also need the import outcome before treating the stock as current.

    For a full snapshot, I would load into staging, validate the rows and publish the new snapshot atomically where the database design allows it. Replacing half the stock table before discovering a malformed row leaves readers with a mixture of two exports. A staging table also makes it easier to inspect rejected data without disturbing the last accepted snapshot.

    Make freshness part of the interface

    Freshness is part of the contract

    The stock response should carry an asOf timestamp. If the business agrees that stock older than thirty minutes is unsuitable for a delivery promise, the consuming application can flag it and ask for confirmation. Thirty minutes is an example threshold, not a universal integration rule. The warehouse and sales teams need to decide what delay they can tolerate.

    I would alert on the age of the last successfully applied export, with separate alerts for transfer failure and validation failure. This catches a job that never starts as well as one that crashes. Counting exceptions alone misses both a disabled scheduler and a script that keeps producing yesterday’s file.

    Changes expose the real dependencies

    A new warehouse version might change availableStock from individual units to cartons. The CSV still parses and all HTTP calls still return success. A contract test should catch that by checking representative items against agreed units and values. The same applies to cancelled orders, deleted items and time zones.

    With incremental extracts, I would document the cursor and deletion behaviour. A query based only on LastModified greater than the previous run time can miss records when timestamps collide or commits become visible late. A supported change feed is preferable where available. Otherwise the extraction design needs a stable ordering key, an overlap strategy and deduplication, plus a way to reconcile the full source periodically.

    This is where several independent scripts become expensive. Each one can implement a slightly different definition of stock, retry policy and cursor. A warehouse change then requires finding and testing every copy. A shared, versioned contract gives consumers a common interpretation even if several runtimes continue to carry the data.

    Count the dependencies you actually have

    For n systems, n × (n − 1) / 2 is the number of possible undirected pairs. Ten systems gives 45 pairs. That is an upper-bound illustration for a fully connected graph; it is not a prediction that every estate has 45 integrations. Direction, multiple contracts between the same systems and differing business rules complicate the count further.

    I would inventory the real flows instead. For each one, record the source, consumer, contract, schedule, credentials owner, last successful run and recovery method. Then ask which source changes require coordinated releases. That list is more useful for planning than a dramatic connection-count diagram.

    Improve the existing path before replacing it

    My first change would be to put the export code and its configuration template in version control, give it a maintained service identity and make failed runs fail visibly. Then I would add staging, freshness monitoring and a replay procedure using the original exportId. These changes can improve a scheduled transfer without a platform migration.

    A durable broker becomes useful when several consumers need independent retries, when the source must keep working during a target outage, or when the delay needs to be much shorter than the schedule. Each subscriber then needs its own progress and error handling. Moving the same undocumented payload onto a topic leaves the contract problem in place.

    I would start the migration with the flow causing the most operational pain. Disable its target during a test, restore it and check the recovered stock against the source. Repeat an export and verify that quantities stay correct. That gives a concrete basis for deciding which shared platform capabilities the next integration actually needs.