Integration Credentials and Call Handling

This internal reference explains, for Tai staff, how the Tai TMS stores integration credentials per organization, how it automatically disables an integration after repeated authentication failures (and how that count is reset), how it logs every outbound vendor call, how timeouts work, how the public-API rate-limit sync to the edge behaves, and where integration code lives across the platform.

Overview

This article is for Tai staff supporting integrations. It covers the things that come up constantly when diagnosing "why did this vendor stop working" or "where are these credentials coming from": credentials live on a per-organization integration record rather than in global configuration; repeated sign-in failures within a day flip an integration into an authentication-error state that halts further calls; every outbound call to a carrier or vendor is written to an activity log; calls have default timeouts that some integrations let you override; public-API rate limits are mirrored to an edge component that currently observes rather than blocks; and integration code is organized into a few distinct "homes" in the platform, which matters when you're trying to find or reason about a given integration.

How it works

Credentials are per-organization, on the integration source record. Each organization's connection to a given carrier or vendor is represented by an integration source record. That record holds the credentials and settings for that specific organization-plus-vendor combination — account number, license key, username/password, tokens, the vendor type, capability toggles, and additional per-integration settings (see the integration source settings reference). When the Tai TMS needs to call a vendor for a particular organization, it reads the credentials from that organization's integration source record. It does not read per-tenant credentials from global configuration.

A small, deliberate exception exists: values that are genuinely global and tenant-independent come from application-level settings rather than from an organization's integration record. Examples are a shared service base URL or platform-wide API key for certain services, the document-OCR service address and access key, the sales-tax rate service token, and the edge rate-limit component's address and token (see below). The rule of thumb is: anything specific to an organization's account with a vendor lives on that organization's integration source; only truly global, shared values live in application settings.

Repeated authentication failures auto-disable the integration. The platform keeps an authentication-failure count on each integration source. When calls to a vendor fail authentication (the vendor rejects the credentials), that count is incremented, and once it passes a threshold the integration's state is flipped to an authentication-error state and the Tai TMS stops making calls on that integration until it is reset. There are two counting paths with different thresholds:

Bulk path — used when a whole batch of sources fails together (rating lane calls and the Trucker Tools driver-track pull). Each failure adds one, and the source flips to the authentication-error state when it had already recorded four failures, so the flip lands on the fifth recorded failure.
Per-source path — used by most vendor integrations that handle their own failures (for example EPay, CargoChief, MyCarrierPackets, Parade, TriumphPay, RoadSyncPay, Greenscreens, HubTrans, driver tracking, and MacroPoint carrier sync). The source flips when its count exceeds the allowed failure count, which defaults to 5 (so the flip lands on the sixth failure) and can be changed per source with an allowedFailureCount setting on the integration source.
Immediate flips — some load-board style integrations skip counting: a failed sign-in or a missing required credential (for example a load board with no license key) can put the source straight into the authentication-error state without waiting for repeated failures.

The failure count is reset every day, but the error state is sticky. A scheduled job runs daily (04:00 UTC, in every deployed environment) and sets the failure count back to zero on every integration source that is not already in the authentication-error state. So in practice a source flips only when it accumulates more failures than the threshold within a single day, and isolated failures spread across days never add up. Sources already in the authentication-error state are left alone: once in that state, the integration stays there and stops calling until something resets it, even if the underlying credential problem is later fixed. A "this vendor suddenly stopped calling" report very often traces back to this state.

How to reset it. Opening the integration source in the integration source screen and saving it (even with no other change) sets the failure count back to zero and clears the authentication-error state, so the integration resumes calling. Some vendors also clear the state themselves after a successful token refresh (the DAT token refresh does this). Fixing the credential on its own does not do it.

Every outbound vendor call is logged as activity. Each call the Tai TMS makes to a carrier or vendor is recorded in an activity log, tagged with the vendor and the kind of activity (rating, dispatch, tracking, webhook, carrier alert, and so on), along with the request, the response, and any error. This log is the primary tool for reconstructing what the Tai TMS actually sent to a vendor and what came back. Different integrations write this log slightly differently — some log the call explicitly at the moment it happens, while others accumulate the request/response during the call and write the log entry only when the operation finishes (so an operation that never finishes cleanly can lose its log entry) — but the net effect is that outbound calls are auditable.

Timeouts. Outbound calls use a default timeout of 15 seconds. Rating calls also use 15 seconds. Dispatch, imaging, and tracking calls use 30 seconds by default, and each can be overridden on the integration source with a numeric DispatchTimeout, ImagingTimeout, or TrackingTimeout setting (in seconds); a blank or non-numeric value is ignored and the default applies.

Error handling is consistent across vendors. The shared plumbing classifies vendor responses into a consistent set of outcomes: authentication failure (the vendor answers unauthorized or forbidden), server-down/transient (a server-error response), throttling/rate-limited (the response text mentions throttling or a limit), "safe to ignore" (for example a not-found page the caller can swallow), and "lane/point not serviced" for rating. Some of these are intentionally treated as non-fatal so the caller can skip or retry rather than hard-failing the whole operation. User-visible validation problems (for example a missing credential) are raised as customer-facing messages that name the missing item.

Integration code lives in a few distinct "homes." When locating integration behavior, it helps to know there are three separate places integration code lives, and a single vendor can appear in more than one:

Thin transport clients — a low-level layer whose only job is to turn a request into an outbound web call and turn the response back into data. These hold essentially no business logic and (with narrow exceptions) don't touch the database; credential lookup and persistence happen a layer up. Today this layer has clients for four vendors: a fuel-pricing service, a shared-inbox tool, a document-OCR service, and a dock/appointment-scheduling service, plus data definitions only for a factoring/carrier-payment vendor (its actual client lives in the vendor-orchestration layer). The fuel client writes an activity-log row for every call; the OCR client is the only one that reads a feature flag.
Shared capability infrastructure — a layer that provides the common plumbing every integration reuses (the base web-call pattern, the error/exception classification, request logging, and timeouts), plus certain capabilities grouped by function rather than by vendor (for example, the rating pipeline, load posting, and the client for the standalone shipment-locations service).
Vendor orchestration — the large layer organized folder-per-vendor (well over 60 folders), where each vendor's service pulls that organization's credentials, calls either a transport client or issues the web call itself, maps the vendor's data to Tai's own models, logs the call, and persists the results.

Knowing which home a given integration lives in tells you where to look: transport concerns in the first, shared plumbing/rating/load-posting in the second, and per-vendor business logic in the third.

Document OCR switches between two services. The OCR client sends an uploaded PDF to a document-extraction service and returns structured data. A feature flag chooses which service is used: with the flag off it talks to the original third-party OCR service; with it on it talks to the Tai-run OCR service, with a different address, access key, and header. Both come from global application settings, not an integration source. Failures are turned into customer-facing messages prefixed with the service name: duplicate document, authentication error, not found, file too large, a generic error with the status, or an unreadable response.

Not every integration is a carrier or rating vendor. The vendor-orchestration home also contains integrations to platform infrastructure and utilities.

Edge rate-limiting sync (Cloudflare). It does two jobs: purging the CDN cache, and mirroring each accounting organization's public-API rate limits (and its per-API-key limits) out to a Cloudflare edge Worker, then pulling the observed request and over-limit counts back in for reporting. Things that make it different from a normal per-organization integration:
- Its credentials are global, not per-organization. The Worker's base URL comes from application settings, and the bearer token comes from Azure Key Vault (read once and cached, and it fails safe to "no token" if Key Vault is unreachable). There is no per-organization integration source record for it.
- It only runs in production. Every call short-circuits outside production and reports success without contacting the edge at all, and the whole sync is additionally gated behind a feature flag. The rate-limit snapshots themselves are built in the data-services layer; the integration code only does the input/output against the Worker. It is driven entirely by scheduled background jobs, not by anything a broker does interactively.
- It currently observes; it does not block. The Worker counts requests per organization and per API key for per-minute, per-day, and per-month windows (fixed windows: the minute, the UTC day, the UTC month) and compares them with the pushed limits, but it always forwards the request unchanged and never answers "too many requests." It adds a response header marking that it is in observe mode and, when a limit was exceeded, a header naming the scope and period. The counters it keeps include "blocked by key limit" and "blocked by organization limit" tallies; these mean "exceeded," not "rejected." An over-limit caller is therefore not actually shielded or throttled today.
- Only some public-API areas are counted. A fixed allow-list of path prefixes is instrumented — shipping (v2 and v3), accounting, broker, carrier(s), customer v2, general ledger, load board, location, organization, sales rep, shipment commissions, shipment stops, staff, authenticate, and the Highway carrier-confirmation callback. Driver tracking, the webhook-demo routes, and the original v1 rate-quote routes are not counted. A request with an unknown or missing API key is forwarded without being recorded, and a recording failure never blocks the request.
- Limits are chosen from fixed tiers. Per-minute limits come from 100 to 1,000 in steps of 100; per-day from 25,000, 50,000, 100,000, 150,000, 250,000, or 500,000; per-month from 500,000, 1,000,000, 2,000,000, 3,000,000, 5,000,000, or 10,000,000; any window can be left unlimited. A Tai super-user edits limits per organization and per API key in the LSP area; the admin area has a read-only monitoring screen that charts requests and "blocked" counts for 24 hours, 48 hours, or 7 days. Both read the same limit data.
- Changes take a short while to reach the edge. The Worker caches which organization owns an API key for about a minute (and caches misses for about 15 seconds), and its counters are saved on a short delay, so a freshly pushed limit or key can take around a minute to take effect and a crash can lose a few seconds of counts.
Canadian sales-tax rates. A rate lookup for Canadian provincial taxes (GST, HST, PST) used for tax charge lines. Its token is a global application setting. Live results are cached for 24 hours per province. If the province code is blank or the token is missing, built-in standard rates are returned without being cached (a blank code gets Ontario's rates); if the live call fails, the built-in rates are returned and that fallback is cached for 24 hours too. The built-in table covers all thirteen provinces and territories and was captured on 2026-09-09.

Highway carrier-alert monitoring. Highway (the carrier onboarding/compliance vendor) also feeds carrier-monitoring alerts — insurance, authority, and identity changes — back into the Tai TMS. A scheduled sync pulls a page of alerts from Highway, matches each one to a broker carrier by MC#/DOT# (creating the master carrier record if it is missing), and updates that carrier's compliance state so what the broker sees in the Tai TMS reflects Highway. Successful alerts are marked processed on Highway (that call is skipped in local debug builds so testing doesn't mutate Highway's state), a private carrier activity-log entry is written per carrier, and the last-processed alert id is remembered on the carrier so the next run resumes after it. Alerts that fail to apply have their ids persisted on the integration source so the next run skips them rather than retrying forever, and the failure is written to the integration activity log tagged as a carrier-alert call. An authentication failure during the sync feeds the same auth-failure back-off as any other Highway call. (Alert ids here are 64-bit — do not assume they fit a 32-bit integer.)

Inbound remit-to updates from integrations. An incoming remit-to from Highway or MyCarrierPackets is written to the carrier whatever the carrier's remit-to source is. Highway applies it on a sync that is allowed to update carrier details (an overwrite sync, or the first Highway sync of that carrier), and replaces only the remit-to address, and only when the street address differs. MyCarrierPackets overwrites the remit-to email and address on a sync that is allowed to overwrite. When the carrier factoring-company management feature is on, an integration that assigns a factoring company also sets the carrier's remit-to source to "factoring company": Highway when an overwrite sync's onboarding packet carries factoring payment terms, MyCarrierPackets when an overwrite sync's carrier data names a factoring company, and RMIS whenever it handles the carrier's pay-to factoring company. With the feature on, when MyCarrierPackets updates a factoring company the Tai TMS already holds, every carrier whose remit-to source is that factoring company has its remit-to name, address, phone, fax, mobile, and email refreshed from the factoring company. Also with the feature on, onboarding (including Parade and MyCarrierPackets) creates an empty remit-to instead of copying the carrier master's name and address; with it off, onboarding copies the master's details.

Other vendor behaviors that changed recently.

MacroPoint callbacks. Schedule callbacks raise a "shipment delayed" alert when the vendor text says the load is more than 3 hours behind (and, when the alert-codes feature is on, also for "cannot make it" and "past appointment time" codes). Temperature callbacks set exactly one of in/above/below threshold. A possible-fraud order status raises a possible-fraud alert, and the not-tracking/invalid-input statuses raise a load-track exception alert, each only when its own feature is on. With the stop-type/geofence feature on, middle stops in the tracking order are typed pickup or drop-off with the geofence radius; with it off they are all sent as pickups without a geofence.
NSure insurance. The insured party is the shipment's end customer (the bill-to organization), and the reference, BOL number, and invoice number sent are all the shipment id. A successful declaration cancel removes the stored certificate and charge-line reference numbers.
TriumphPay. A bill with a negative total is sent as a credit note rather than an invoice. When a shipment's charge lines span more than one carrier, one load per carrier is sent, keyed by shipment id plus carrier, with that carrier's lines; otherwise a single load is sent for the linehaul carrier.
EPay. A carrier that has a factoring company recorded is no longer auto-activated as an EPay payee. The hourly document-page pull runs only for sources that opt in with the DocumentRetrieval setting (see the settings reference); the former global kill switch is gone.
DAT Convoy load posting. Each batch is validated first. Shipments that fail validation are not posted, and any that were already posted are cancelled on DAT with a broker-issue reason; the rest are posted as available.
FarEye and Shippeo. Origin and destination coordinates come from the shipment-locations service. Shippeo is sent only shipments modified in the last 45 days, past Dispatched through Delivered, on a linked organization, that have a location in the last 15 minutes.
Reference-number mirror. Vendor paths that add or remove shipment reference numbers also write the matching master reference numbers to the new custom reference table, so the two stay in step. This is skipped unless the dual-write feature is on.

Business rules & limits

Credentials resolve from the organization's integration source, not global config. Reading a per-organization credential from application settings (or vice versa) is a common mistake and results in the wrong or missing credentials being used silently. Organization-specific values belong on the integration source record; only truly global values belong in application settings.
The authentication-failure threshold depends on the code path. Bulk-marked sources flip on the fifth recorded failure; per-source integrations flip after more failures than the allowed count (default 5, so on the sixth, adjustable per source); some load-board logins flip immediately. Because the count is zeroed daily at 04:00 UTC for sources not already in error, the threshold effectively applies to failures within one day.
The authentication-error state is sticky. An integration in the authentication-error state will not resume calling on its own; it must be reset by editing and saving the integration source (or by a vendor-specific token refresh that clears it). Fixing the credential alone does not bring it back, and the daily counter reset does not touch it.
Every outbound call is logged, including full request and response content. Because the activity log captures full request and response bodies, those log entries can contain sensitive material (credentials or personal data present in a payload). Treat integration activity logs as sensitive.
Default timeouts are 15 seconds, or 30 for dispatch, imaging, and tracking. The latter three can be overridden per integration source; rating cannot.
Error handling distinguishes "safe to skip" from real failures. The shared plumbing classifies vendor responses into authentication failure, server-down/transient, throttling, "safe to ignore," and "lane/point not serviced." Some of these are intentionally treated as non-fatal so the caller can safely skip or retry rather than hard-failing the whole operation.
A vendor can exist in more than one home. The same vendor may have a thin transport client in one layer and its orchestration/business logic in another; don't assume a single location holds everything for a given vendor.
A few integrations are configured globally, not per organization. The edge rate-limiting sync (Cloudflare), document OCR, and the Canadian sales-tax lookup are the notable ones. For the rate-limit sync the Worker URL comes from application settings and its token from Azure Key Vault, with no per-organization integration source record, and it runs in production only behind a feature flag, so "nothing is syncing" in a non-production environment is expected.
Public-API rate limits are observed, not enforced, today. Counts and over-limit flags are recorded for a fixed set of API areas, but over-limit requests are still served. Do not tell a customer their traffic was blocked by a rate limit.
Highway feeds carrier-compliance alerts, not just onboarding. Beyond onboarding-packet sync, Highway pushes carrier-monitoring alerts (insurance/authority/identity) that update a broker carrier's compliance state. Its bookkeeping (last-processed alert id, failed alert ids) lives on the carrier and integration-source records; a stuck or repeating alert usually traces to those saved ids. The "mark processed on Highway" step is intentionally skipped in local debug builds.

Frequently asked questions

Where does the Tai TMS get the credentials for a given organization's carrier or vendor integration?

From that organization's integration source record. Each organization-plus-vendor combination has its own record holding the account number, license key, username/password, tokens, vendor type, capability toggles, and extra settings. The Tai TMS reads credentials from that per-organization record when it calls the vendor. Only genuinely global, tenant-independent values (like a shared base URL, platform-wide key, the OCR service address and key, or the sales-tax service token) come from application settings instead.

Why did a vendor integration suddenly stop making calls?

The most common cause is the authentication-failure back-off. The Tai TMS counts authentication failures on each integration source and, once the count passes the threshold, flips that integration into an authentication-error state and stops calling until it is reset. The threshold is the fifth recorded failure for bulk-marked sources and, for most other integrations, more than the allowed failure count (default 5, adjustable per source). Some load-board logins flip on the first failure or when a required credential (such as a license key) is missing. Check the integration's state and its authentication-failure count first.

If I fix the bad password, will the integration start working again automatically?

No. The authentication-error state is sticky — once an integration is in it, it stops calling and stays that way until the state is reset, even if you correct the underlying credential. Open the integration source and save it: saving sets the failure count to zero and clears the authentication-error state. The nightly counter reset does not help here, because it skips sources already in the error state.

Do a few failed logins over a couple of weeks eventually disable an integration?

No. A scheduled job zeroes the authentication-failure count every day at 04:00 UTC for every integration source that is not already in the authentication-error state, so only failures that pile up within a single day can trip the threshold.

How can I see exactly what the Tai TMS sent to a vendor and what came back?

Use the integration activity log. Every outbound carrier/vendor call is recorded there, tagged with the vendor and the activity type (rating, dispatch, tracking, webhook, carrier alert, etc.), together with the request, the response, and any error. It's the authoritative record of the actual exchange. Note that these entries can include full request/response bodies, so they may contain sensitive data.

A vendor's credentials look correct but calls still fail — where else should I look?

Confirm you're looking at the right integration source (per-organization) rather than a global setting, verify the integration isn't already sitting in the authentication-error state from earlier failures, check whether a required source setting is missing or mis-cased (see the integration source settings reference), and check the activity log for the actual vendor response. Also confirm which "home" the integration lives in, because the transport client, shared plumbing, and vendor orchestration are separate layers and the problem may be in a different one than you expect.

How long do outbound vendor calls wait before timing out?

Fifteen seconds by default. Dispatch, imaging, and tracking calls wait 30 seconds by default, and each of those can be changed on the integration source with a numeric DispatchTimeout, ImagingTimeout, or TrackingTimeout setting in seconds. Rating calls use 15 seconds and have no per-source override.

Why is the same vendor referenced in more than one part of the platform?

Because integration code is split across three homes — thin transport clients, shared capability infrastructure, and vendor orchestration — and a single vendor can appear in more than one. For example, a vendor might have a thin transport client in one layer while its credential lookup, mapping, and persistence live in the vendor-orchestration layer. When investigating, identify which home holds the behavior you care about.

Are integration activity logs safe to share?

Treat them as sensitive. The activity log captures full request and response content, which can include credentials or personal data embedded in a payload. Share only what's needed for diagnosis, and handle the log entries according to data-handling policy rather than pasting them freely.

Where are the public-API rate limits enforced, and where is that integration's credential stored?

They are tracked at the edge by a Cloudflare Worker, which the Tai TMS keeps in sync so it knows each organization's and API key's per-minute/per-day/per-month limits. Today the Worker only observes: it records usage, flags over-limit requests with a response header, and keeps "blocked" counters, but it still forwards every request and never rejects one. That sync is one of the few integrations configured globally rather than per organization: the Worker URL comes from application settings and the token from Azure Key Vault — there is no per-organization integration source record for it. It runs in production only and is feature-flag gated, so don't expect it to do anything in beta/demo/local. The limit snapshots are assembled in the data-services layer; the integration itself only talks to the Worker.

A customer says they were blocked by the API rate limit — is that possible?

Not by the edge rate-limit component as it works today. It records and flags usage above a limit (the admin chart labels these "blocked by key limit" or "blocked by organization limit"), but it always passes the request through to the Tai TMS. If a customer saw a rejection, look for another cause such as an authentication failure, a retired operation answering "not found," or a problem outside the rate limiter.

What limit values can be set for an organization or API key?

Per-minute limits of 100 to 1,000 (in steps of 100), per-day limits of 25,000, 50,000, 100,000, 150,000, 250,000, or 500,000, and per-month limits of 500,000, 1,000,000, 2,000,000, 3,000,000, 5,000,000, or 10,000,000; any window can be unlimited. Only a Tai super-user can edit them, in the LSP area, and the admin area has a read-only chart of requests and over-limit counts.

A carrier's compliance state in the Tai TMS looks wrong or stale — could an integration be involved?

Yes. Highway feeds carrier-monitoring alerts (insurance/authority/identity) into the Tai TMS, and a scheduled sync matches each alert to a broker carrier by MC#/DOT# and updates that carrier's compliance state. Check the integration activity log for carrier-alert entries, and check the saved bookkeeping — the last-processed alert id on the carrier and any failed alert ids stored on the integration source (failed ids are deliberately skipped on subsequent runs). Remember the "mark processed on Highway" step is skipped in local debug builds, so local behavior differs from production.

A Highway or MyCarrierPackets remit-to update did not change the carrier's remit-to — why?

The carrier's remit-to source does not block either integration; both write an incoming remit-to regardless of it. Highway writes a remit-to only on a sync that is allowed to update carrier details (an overwrite sync, or the carrier's first Highway sync), changes only the address and not the name, and skips the change when the street address already matches. MyCarrierPackets writes the remit-to email and address only on a sync that is allowed to overwrite and only when its carrier data includes remit-to information. Check which kind of sync ran and what the vendor sent in the integration activity log.