Background Jobs and Automation Internals

This internal reference explains, for Tai staff, how the Tai TMS runs background work: a database-backed job framework with separate queues, a zero-automatic-retry policy, an explicit and environment-gated list of recurring jobs, the provider workers and backfill jobs that do the actual work, and the separately deployed job host.

Overview

This article is for Tai staff who operate, deploy, or debug background processing in the Tai TMS. Background work — EDI in and out, load posting, carrier onboarding/compliance sync, tracking pulls, accounting and payment sync, email and document generation, report delivery, workflow-automation schedules, data cleanup, and one-off migrations and backfills — runs on a job framework backed by the SQL database. Four facts shape everything about how these jobs behave: work is split across separate queues; the global automatic-retry count is zero, so jobs must be safe to re-run; the recurring-job schedule is an explicit, hand-maintained, environment-gated list rather than something auto-discovered; and the job host is deployed separately from the main web application.

How it works

A SQL-backed job framework with separate queues. The job framework stores its state in the SQL database (not an in-memory or Redis queue for jobs). Work is divided across three named queues, each served by its own dedicated worker pool started at host startup:

default — general background work; the largest worker pool (roughly ten workers per processor core; one in local debug).
EDI — EDI processing work, kept on its own queue (about two workers per core; one in local debug) so heavy EDI traffic doesn't crowd out everything else.
email — email sending and closely related output work (such as document/text generation), with a large fixed pool (around fifty workers; one in local debug) so mail throughput stays high.

A job runs on the EDI or email queue only if it's explicitly marked for that queue; anything unmarked runs on the default queue. (The queue list is still exactly these three — the jobs added recently, described below, are unmarked and therefore all run on the default queue.)

Three kinds of jobs. There are recurring jobs (scheduled on a cron-like timetable), fire-and-forget jobs (enqueued once to run as soon as a worker is free), and delayed jobs (enqueued to run once after a set wait — for example a workflow-automation rule with a time-delay trigger, or the follow-up check after a carrier tender window). A very common pattern is a "batch/parent" recurring job that figures out what needs processing and then enqueues one child job per unit of work (per organization, per shipment, or per integration source), fanning the work out across workers.

Zero automatic retries — jobs must be re-runnable. The global policy is zero automatic retries. If a job fails, the framework does not retry it automatically. This is deliberate: every job is expected to be safe to run again and to recover on its next scheduled run rather than relying on an automatic retry. A specific job can opt back into retries by declaring its own retry policy, but that is the exception, not the default. Current examples: PDF generation, email sending, and text-message sending retry up to five times, five minutes apart; inbound EDI carrier processing retries up to five times, two minutes apart; the inbound EDI document processors retry once after five minutes; the master reference number backfill workers retry up to three times (1, 5, and 15 minutes apart); Parade and DAT onboarding and a few other vendor syncs retry up to two or three times. The practical consequence for everything else: jobs must be idempotent and tolerant of partial completion, because a transient failure otherwise simply drops that run until the next scheduled tick.

Every failed job is reported to telemetry. Sitting next to the zero-retry policy is a second framework-wide filter: whenever a job lands in the failed state, it records the exception to Application Insights, tagged with the job's id, type, method, and retry count. This is what powers the operational alerting on background-job error rates (the "higher errors on Hangfire" alert). It combines cleanly with the zero-retry policy: because a failed job is not retried, it usually fails — and reports — exactly once, so an error spike in telemetry maps directly to real failed runs rather than to retry noise.

Idempotency and overlap control. Because there are no automatic retries and batches fan out work, two mechanisms keep things safe. First, a job can be enqueued with a stable caller-supplied identifier; if a job with that identifier is already waiting, processing, awaiting, or scheduled, a duplicate enqueue is skipped rather than creating a second copy. Callers that need cross-run deduplication persist that identifier and reuse it (carrier onboarding syncs keep it as a setting on the integration source). Second, heavier jobs are marked to disallow concurrent execution, so the same job can't overlap itself if a prior run is still going; the master reference number backfill shares one named lock across its daily run, its manual repair, and its continuations.

Recurring jobs are an explicit, environment-gated list. There is no auto-discovery: a job class does nothing on a schedule until a line is explicitly added to the recurring-registration list. That registration is environment-gated — production, demo, and beta can register the same job with different schedules, some environments skip certain jobs entirely, and some jobs are registered as "never auto-fire" so they exist only to be triggered manually from the job dashboard (used for data migrations, backfills, and correction jobs). Recurring jobs are also not registered at all in local debug builds — with one exception: the remit-to backfill (manual-only) is registered in every build, so it is visible on a local dashboard. Local behavior therefore differs sharply from a deployed environment.

Provider workers. The concrete workers are grouped by provider/capability. Representative examples:

Tracking updates — a worker processes received driver-track / location updates for a shipment from a tracking provider (for example, the MacroPoint location-update worker). A Trucker Tools driver-track pull merges driver-track rows and writes the locations (every 15 minutes in production, every 3 hours in beta), and a MacroPoint push job (every 16 minutes in production, manual-only in beta) sends the latest geocoded location for dispatched, not-yet-delivered linehaul shipments whose last location is under 15 minutes old, in chunks of 20.
Carrier onboarding sync — a batch/child pair syncs carrier onboarding and compliance data per integration source from an onboarding provider (for example, the RMIS onboarding delta sync), using a persisted job identifier so a run isn't duplicated.
Factoring / payment application — a batch/child pair enqueues accounts-payable/receivable payment application per factoring integration source (for example, the Denim payment-application worker, daily at 06:00 UTC).
Load posting — a job posts, reposts, and removes available loads across the connected load boards.
Shipment-change processing — a worker drains the shipment-change queue every minute and enqueues the appropriate follow-up work.
Workflow automation — a scheduler job runs every 15 minutes in production only (it is not registered in demo or beta). It picks active workflow-automation schedules owned by active organizations that are due (monthly on the chosen day, weekly on chosen weekdays at a time of day, recurring on chosen weekdays within a daily window and no more often than every 15 minutes, each in its own time zone), stamps the last-ran time, and enqueues them in chunks of 50. Each scheduled workflow looks at up to 500 matching shipments, newest first. A person can also run a scheduled workflow on demand, but not again within 30 minutes of the previous on-demand run.
Integration authentication counter reset — a daily job at 04:00 UTC zeroes the authentication-failure count on every integration source that is not already in the authentication-error state (see the integrations internal reference). It runs in every deployed environment.
Archive and cleanup — jobs move older rows to the archive database (shipment activity logs, every 5 minutes) and prune caches, activity/trace/API logs, outgoing emails, and the framework's own failed and deleted job records on a schedule. The data-cleanup processor is registered roughly thirty times, once per cleanup method, and several of those are manual-only.
Shipment status housekeeping — a nightly job (06:00 UTC in production, manual-only in beta, absent in demo) marks transit legs complete for up to 5,000 delivered shipments more than 12 months old whose bills and invoices are fully paid, then deletes last-known-location rows for shipments that are not in transit or out for delivery and were last modified more than one year ago. It re-queues itself with a run counter and stops once the UTC hour passes 10 or after 250 runs.
EPay document pulls — an hourly job in production (manual-only in beta; on beta it only considers sources flagged as testing) pulls document pages only for EPay sources that opted in with a document-retrieval setting. The former global kill switch was removed, so the opt-in is now unconditional.
Monitoring/alerting — jobs that watch operational health (for example, alerting when the AI quoting assistant's response time crosses a threshold) and post to the team's alerting channel.

There are also EDI, accounting, invoice, document-generation, report-delivery, and bulk-operation workers among the scheduled and fanned-out jobs.

Jobs reach shipment locations only through the shipment-locations service. The old in-process location tables are gone. Location jobs call the standalone shipment-locations service over HTTP (bulk writes and deletes go in chunks of 2,000, the default call timeout is 30 seconds), so there is no database transaction across the two stores. In the driver-track pull, the driver-track, stop, and reference-number saves happen before the location call and the shipment activity logs happen after it, and a failure of the location call throws after the earlier rows were already saved — with zero retries, that run is dropped until the next tick. Shipment deletion deletes a shipment's location rows through the service, and the demo-data refresh re-creates a copied shipment's location history shifted by seven days. The former location sync, last-location sync, and location archive jobs have been retired and are no longer registered.

Recently added and manual backfill jobs (all hand-registered on the default queue). A few newer job families are worth knowing when reconciling what is scheduled:

Edge rate-limit sync (Cloudflare), production only. A set of jobs keeps the edge rate-limiter in step with each organization's public-API limits and pulls back its counts: a count pull runs every 5 minutes, a full sync of all organizations runs once a day at 08:00 UTC, and a weekly cleanup (Sundays at 09:00 UTC) removes disabled organizations from the edge. There are also single-organization sync and single-organization delete jobs used on demand. These are registered only in the production branch (they no-op elsewhere) and are additionally feature-flag gated. The full sync reports failure if any organization failed.
EDI automation backfill, all deployed environments. A backfill job runs every 5 minutes, a reconcile job runs once a day at 08:00 UTC, and a job that reports unmatched shipment-to-EDI-file references runs at 08:30 UTC. Together they backfill EDI files and shipment-automation log entries into the main TMS database. These are registered on the unconditional tail of the schedule, so they run in every deployed environment.
Document automation detail sync, all deployed environments. Every 5 minutes a job copies the document-automation queue from the logging database to the main TMS database, whether or not the document-automation double-write setting is on. It first applies the changes a trigger on the logging table has recorded, 500 at a time for up to about two minutes, re-copying each changed document and removing main-database copies whose logging row was deleted. It then compares the two databases in id ranges of 100,000, newest range first, and copies missing documents for up to about two minutes or 20,000 documents, whichever comes first. The job cannot overlap itself, and its result message gives the counts and how many changes or ranges are left for the next run.
Master reference number backfill, daily. A batch/parent job copies master reference numbers (the built-in reference types) from the legacy reference table into the custom reference table, for the shipment level only. It runs daily at 07:05 UTC in every deployed environment; a manual repair variant is registered as never-auto-fire. Its guards, in order: a positive chunk size, the off-hours window (05:00 through 08:59 UTC — always open in debug builds), the dual-write feature on, for the repair variant only the "read from custom table" feature off (repair is refused once the custom table is the read source), and master types being seeded. The backfill only inserts missing rows and never overwrites an existing custom row; the repair also overwrites differing values and deletes rows whose legacy value is blank or that have no legacy row. Shipment ids are bucketed in chunks of 10,000, at most 200 chunks are queued per run, and if more remain it queues a continuation from the last queued shipment, so it is resumable. Workers retry up to three times and share one named lock with the parent.
Remit-to backfill, manual. A one-time-style job walks all broker carriers in batches (batch size clamped between 10,000 and 50,000, default 10,000), first setting each carrier's remit-to source (factoring company if the carrier has a factoring organization, otherwise user-entered if its remit-to address already has any name or address text, otherwise carrier name), then copying the matching name and address into the remit-to for carrier-name, DBA, and factoring-company sources (user-entered remit-tos are left alone). It re-queues itself while a batch reads a full page and stops when a batch reads fewer rows than the batch size.
Customer reference setting backfill, manual. Merges customer reference number settings in chunks of 500 organizations, at most 40 chunks per run, only when the customer reference settings feature is on; its result states which organization the next run should resume from.
Other manual-only jobs. Several migrations and corrections are registered as never-auto-fire, including the workflow-automation customer-filter data correction, the pass-through charge-line status correction (report and fix), the carrier-rules migration, the master reference default-data migration, and a duplicate broker-carrier attachment cleanup.

API tracking logs per-shipment failures. In the API-tracking batch, each shipment's tracking update runs in its own error handler. A failure used to be swallowed silently; it is now logged with the shipment id while the batch continues with the next shipment. A carrier that sends no dates no longer produces a location dated year 1; the location date falls back to the update date, then the actual delivery date, then now.

The job host is deployed separately. The background-job host is a separate deployable from the main web application — it runs as its own hosted service, deployed on its own. Its dashboard is available in deployed environments behind sign-in, but the only check is that the user is signed in (there is no role check); in local debug it has no authorization at all. At startup the host (and the legacy service) formats a date once before any worker thread starts, to avoid a rare race in date formatting that could otherwise leave every date string in the process with a doubled time-zone offset. Separately again, an older, standalone background service handles a small fixed set of processors (EDI inbound, EDI outbound, and a payment processor) on simple timer loops; it is a different deployable from the main job host and is not part of the same framework. When operating or deploying, keep these straight: the main job host, the legacy standalone service, and the web application are three different things.

Business rules & limits

Jobs are SQL-backed and split across default, EDI, and email queues. Queue assignment is explicit; unmarked jobs land on the default queue. Heavy EDI and email work is isolated on its own queues so it doesn't starve general processing.
Global automatic retries are zero. A failed job is not retried automatically. Jobs must be safe to re-run and expected to recover on the next scheduled run. A job may opt into its own retry policy (as PDF, email, text-message, inbound EDI, and the master reference backfill workers do), but that is an exception.
Every failed job is reported to Application Insights. A framework-wide telemetry filter records each failed run's exception (with the job id, type, method, and retry count). Combined with zero retries, a failing job typically fails and reports exactly once, and this feeds the background-job error-rate alerting.
Jobs must be idempotent. Because of zero retries plus batch fan-out, jobs must tolerate being run again and tolerate partial completion. Deduplication (stable job identifiers) and no-overlap marking on heavy jobs exist to support this.
Bounded, resumable runs. Recurring and backfill jobs bound the set a run processes and cap it so the next run resumes (200 chunks for the master reference backfill, 40 chunks for the customer reference settings backfill, a single page of carriers for the remit-to backfill, 500 shipments per workflow run, 5,000 shipments per status-housekeeping pass). Writes use bulk operations rather than one save per row.
Recurring jobs are not auto-discovered. A new recurring job only runs if it is explicitly registered. Writing the job class is not enough.
Recurring registration is environment-specific. The same job can have different schedules in production, demo, and beta; some are skipped in some environments (the workflow scheduler, for example, is production only); some are registered to never auto-fire and exist only for manual triggering; and none of the recurring jobs register in a local debug build (the remit-to backfill is the one exception). Never assume a job's schedule is identical across environments.
The master reference number backfill is guarded. It silently does nothing (returning a message) outside the 05:00–08:59 UTC window, when the dual-write feature is off, when no master types are seeded, and — for the repair variant — once reads come from the custom table. It never overwrites an existing custom row when backfilling.
The job host is a separate deployment from the web app. The background-job host deploys on its own, apart from the main web application. A separate legacy standalone service handles a small fixed set of processors on timers and is yet another deployable — don't conflate the three.
A dropped run waits for the next tick. With no automatic retry, a transient failure means that unit of work isn't reprocessed until the job's next scheduled run (or until it's re-enqueued), so timing expectations should account for the cron interval, not an immediate retry.
Location writes are not transactional. Jobs write shipment locations to a separate service over HTTP; rows saved before a failed location call stay saved.

Frequently asked questions

What backs the background-job framework — is it Redis?

No. The Tai TMS background-job framework stores its job state in the SQL database. (Redis is used elsewhere in the platform for other purposes, but jobs are SQL-backed.) Work is divided across three named queues — default, EDI, and email — each served by its own worker pool.

Why didn't a failed job retry on its own?

Because the global automatic-retry count is zero. The Tai TMS deliberately does not retry failed jobs automatically; each job is expected to be safe to re-run and to recover on its next scheduled run. Individual jobs can opt into their own retry policy (PDF, email, text-message, inbound EDI, and the master reference backfill workers do), but by default a failed run is simply dropped until the next scheduled tick or a manual re-enqueue.

What does "jobs must be idempotent" mean in practice here?

Because there are no automatic retries and batch jobs fan out one child per unit of work, a job may end up running again or running after a partial failure. Idempotent means running it twice produces the same correct result without duplicating side effects. The framework supports this with stable, caller-supplied job identifiers that skip duplicate enqueues, and with no-overlap marking on heavier jobs so a job can't run concurrently with itself.

I added a recurring job class but it never runs — why?

There is no auto-discovery. A recurring job only runs if it is explicitly added to the recurring-registration list, and that registration is environment-gated. Confirm the job is registered, that it's registered for the environment you're testing, and remember that recurring jobs are not registered at all in local debug builds — so it must be verified in a deployed environment, not locally.

Why does a job run on a different schedule in beta than in production?

Because recurring-job registration is environment-specific. Production, demo, and beta can register the same job with different schedules, some environments skip a job entirely, and some jobs are registered to never auto-fire (existing only for manual triggering). Differences in schedule between environments are expected and intentional, not a bug.

What are the provider workers and what do they do?

They're the concrete background workers grouped by provider/capability. Examples include a tracking worker that processes location/driver-track updates for a shipment (such as the MacroPoint updates), a carrier-onboarding sync (such as the RMIS delta sync) that runs per integration source, a factoring payment-application worker (such as Denim) that applies AP/AR payments per factoring source, a load-posting job that posts/reposts/removes loads on the boards, plus workflow-automation scheduling, archive, cleanup, EDI, accounting, document-generation, and monitoring/alerting jobs.

Why didn't my scheduled workflow-automation rule run in beta or demo?

The workflow scheduler job is registered only in production, where it runs every 15 minutes. In beta and demo, scheduled workflows are not auto-run; an on-demand run from the schedule list still works, though not more than once every 30 minutes. Each run considers up to 500 matching shipments, newest first.

Is the job host part of the main web application deployment?

No. The background-job host is a separate deployable that runs on its own, apart from the main web application. Additionally, an older standalone background service handles a small fixed set of processors (EDI inbound, EDI outbound, and a payment processor) on timer loops and is a third, independent deployable. When deploying or troubleshooting, keep the web app, the main job host, and the legacy service distinct.

How are heavy EDI or email jobs kept from slowing everything else down?

They run on their own queues. The Tai TMS separates work into default, EDI, and email queues, each with its own worker pool, so a surge of EDI processing or a large batch of outbound email is handled by that queue's workers rather than crowding out general background work on the default queue. There are still exactly three queues; recently added job families (edge rate-limit sync, EDI automation backfill, document automation detail sync, the reference-number and remit-to backfills) are unmarked and run on the default queue.

If a background job fails, how do we find out?

Every failed job is reported to Application Insights by a framework-wide telemetry filter, tagged with the job's id, type, method, and retry count. That is the source of the background-job error-rate alerting. Because the global retry count is zero, a failed job normally fails and reports once, so a telemetry spike corresponds to real failed runs rather than retry churn.

What are the newer scheduled and manual jobs I might not recognize?

The edge rate-limit sync (Cloudflare) keeps each organization's public-API limits in step with the edge and pulls back counts — production only and feature-flag gated (a count pull every 5 minutes, a daily full sync at 08:00 UTC, a weekly disabled-organization cleanup, plus on-demand single-org sync/delete). The EDI automation backfill runs in every deployed environment (a backfill every 5 minutes, a daily reconcile at 08:00 UTC, and a daily unmatched-reference report at 08:30 UTC). The document automation detail sync also runs every 5 minutes in every deployed environment, copying document-automation records from the logging database to the main database. The master reference number backfill runs daily at 07:05 UTC inside its off-hours window and has a manual repair variant. The remit-to backfill and the customer reference setting backfill are manual-only. A daily 04:00 UTC job resets integration authentication-failure counts, and the nightly shipment-status housekeeping job also trims old last-known-location rows. The former shipment-location sync and archive jobs no longer exist.

The master reference number backfill returned a message and did nothing — is it broken?

Not necessarily. It deliberately skips work outside the 05:00–08:59 UTC window, when the dual-write feature is off, when no master reference types are seeded, and (for the manual repair) when reads already come from the custom table. Check the returned message in the job history; it names the guard that stopped it. In local debug builds the time-window guard is always open.

Who can open the job dashboard?

In deployed environments the dashboard is served behind sign-in, and any signed-in user is allowed — there is no admin-role check — so access is controlled by who can sign in to the host at all. In local debug builds it has no authorization.