Batches and tracking
Group jobs and run a callback job when they finish, and record the progress of one job.
A batch groups jobs. When the jobs finish, the server pushes a callback job. Use a batch for work in many parts, such as an import of many files, followed by one final step, such as an email to the user.
Job tracking records the state and the progress of one job, so an app can show it to a user.
Make a batch
- Create the batch with a
successcallback, acompletecallback, or both. The server returns the batch id (bid). - Push each job with
custom.bidset to that id. - Commit the batch. This closes the definition.
client = Anway.client
bid = client.batch_new(
success: Anway::Job.new('import.done', [import.id]),
complete: Anway::Job.new('import.finished', [import.id]),
description: "import #{import.id}"
)
import.files.each do |file|
client.push(Anway::Job.new('import.file', [file.id], custom: { 'bid' => bid }))
end
client.batch_commit(bid)success, err := anway.NewJob("import.done", []any{importID})
if err != nil {
return err
}
bid, err := client.BatchNew(ctx, anway.Batch{Success: success, Description: "import"})
if err != nil {
return err
}
for _, fileID := range fileIDs {
job, err := anway.NewJob("import.file", []any{fileID})
if err != nil {
return err
}
if err := client.Push(ctx, job.SetCustom("bid", bid)); err != nil {
return err
}
}
return client.BatchCommit(ctx, bid)const bid = await client.batchNew({
success: new Anway.Job('import.done', [importId]),
description: 'import',
});
for (const fileId of fileIds) {
await client.push(new Anway.Job('import.file', [fileId], { custom: { bid } }));
}
await client.batchCommit(bid);After the commit, a push to the batch fails. batch_open (Ruby), BatchOpen (Go), and batchOpen (Node.js) open a committed batch again, for new jobs and child batches, until the next commit. Open fails after the success callback fired.
A batch can have a parent: give parent_bid (Ruby), ParentBID (Go), or parentBid (Node.js) when you create it. The parent then waits for the child batch.
Callbacks
| Callback | Fires when |
|---|---|
complete | All jobs ran one time, with ACK or FAIL, and all child batches are complete. |
success | All jobs have ACK, and the success callback of all child batches has ACK. |
successalways comes aftercomplete.- An empty batch fires its callbacks at the commit.
- A callback is a normal job. It runs and retries as other jobs do. The server adds
custom._bidandcustom._cbto it. - A callback counts as finished only at its
ACK. A missing callback counts as finished at once.
Batch status
status = Anway.client.batch_status(bid)
status['pending']status, err := client.BatchStatus(ctx, bid)
if err != nil {
return err
}
log.Println(status.Total, status.Pending, status.Failed)const status = await client.batchStatus(bid);
console.warn(status.total, status.pending, status.failed);| Field | Meaning |
|---|---|
bid | The batch id. |
parent_bid | The parent batch, if any. |
description | The description that you gave. |
created_at | When the batch was made. |
total | The jobs in the batch. |
pending | The jobs without an ACK. |
failed | The pending jobs that failed at least one time. |
complete_st | The complete callback: blank before it fires, 1 after it fires, 2 after it finishes. |
success_st | The success callback, with the same values. |
Limits
- A job that goes to the Dead set, or that a
mutate, a kill, or an expiry removes, stays pending. Thesuccesscallback then does not fire. - A batch with no commit expires after 30 minutes. A batch that you opened again has no expiry.
- A batch expires 24 hours after its
successcallback finishes. A batch that did not finish does not expire.
Job tracking
Push a job with track: true. The server then records its state, and the worker can add a percent and a description.
class ExportJob
include Anway::Worker
anway_options event: 'report.export', track: true
def perform(report_id)
track(percent: 40, desc: 'rows written')
# ...
end
end
jid = ExportJob.perform_async(7)
Anway.client.track_get(jid)
# => {"jid" => "...", "state" => "working", "percent" => 40, "desc" => "rows written", ...}job, err := anway.NewJob("report.export", []any{7}, anway.Options{Track: true})
if err != nil {
return err
}
if err := client.Push(ctx, job); err != nil {
return err
}
err = client.TrackSet(ctx, anway.TrackUpdate{JID: job.JID, Percent: anway.Int(40), Desc: "rows written"})
if err != nil {
return err
}
track, err := client.TrackGet(ctx, job.JID)
if err != nil {
return err
}
log.Println(track.State, *track.Percent, track.Desc)const job = new Anway.Job('report.export', [7], { track: true });
await client.push(job);
await client.trackSet({ jid: job.jid, percent: 40, desc: 'rows written' });
const track = await client.trackGet(job.jid);
console.warn(track.state, track.percent, track.desc);- States:
unknown,enqueued,working,success,failed, anddead. - Set: a track set changes
percentanddesc.reserve_untilextends the reservation of the running job. It cannot make the reservation end sooner. - Expiry: a track expires 30 minutes after its last state change, or 30 minutes after the reservation ends. A get then returns
unknown.