esedark
Menu
Back to notes

logs / alerts / monitoring

Application monitoring: logs, alerts and implementation

A dashboard can be green while orders stop progressing. I start with the workflow the business needs to protect, then define the evidence, alert and response it requires.

Application monitoring should help a team recognise interrupted work and decide what to do next. Before choosing tools, I map one critical workflow, its dependencies and the person responsible for recovery. Centralised logs, service metrics and external checks provide a starting point; their coverage depends on what you instrument and test.

Start with a business workflow and its failure modes

For a hypothetical order system, an HTTP response confirms receipt but does not prove the warehouse received the order. Track accepted, pending, completed and failed work separately. Agree how long an order may remain pending and who investigates when that limit is exceeded. The acceptable delay comes from the operation, not a copied alert threshold.

Combine externally visible checks with internal signals. A working homepage does not establish checkout health, and a running worker does not establish successful processing. The Google SRE monitoring chapter explains this distinction and the role of latency, traffic, errors and saturation.

Choose signals that lead to a decision

FlowUseful signalDecision to prepare
Public applicationCritical transaction result, errors and latencyInvestigate a release or dependency; identify affected users
Background workOldest pending job, completion rate and final failuresInspect the backlog before scaling or retrying
Host and storageAvailable capacity, growth and restart eventsRestore headroom without deleting needed evidence
Telemetry pipelineMissing expected samples or failed delivery checksDistinguish lost visibility from a healthy service

Define the measurement window, expected activity and recovery condition for each signal. No traffic may be normal overnight; missing telemetry is a different state. Keep request and job identifiers in logs or traces rather than creating an unbounded metric series for every execution.

Implementation details belong in the relevant subsystem: Laravel queues in production, process supervision and Nginx configuration for Laravel. This guide focuses on how those components form an operational monitoring scope.

Specify the alert and the response together

Separate an urgent interruption from a ticket that can wait for working hours. For every alert, record the affected workflow, owner, coverage hours, evidence link, first investigation step and escalation if nobody acknowledges it. Group repeated notifications from the same incident and define when recovery should close the alert.

A runbook is a short response procedure. In the order example it could ask the operator to inspect queue age and the last successful delivery, check the downstream system, then choose whether to pause intake or resume processing. Replaying orders requires duplicate protection; monitoring alone does not make a retry safe.

Prometheus alerting guidance also recommends symptom-based alerts and testing the monitoring path itself. A configured notification channel is not evidence that an operator can receive and act on a real alert.

Keep enough evidence without collecting everything

Use structured events with timestamp, environment, service, release version, correlation identifier and outcome. Preserve selected diagnostic evidence with controlled access. Exclude credentials and unnecessary payload contents; define retention and deletion by data category. Review who can read, export and change the monitoring configuration, alongside infrastructure access controls.

The Adslyfy monitoring case provides documented examples of execution states, attempts and captured evidence. Its screenshots demonstrate operational visibility, not a measured reduction in incidents or a general monitoring service-level guarantee.

How to accept a monitoring implementation

I would agree the following checks before implementation. These are proposed acceptance criteria, not tests performed on a client system. Use a controlled environment or an explicitly agreed exercise that cannot affect customer transactions.

  1. Interrupted workflow: simulate a stalled job and verify that the agreed signal identifies unfinished work even while the web endpoint remains available.
  2. Actionable notification: confirm receipt by the designated owner, the evidence link and the escalation path within the agreed coverage period.
  3. Noise and recovery: repeat the same fault, check grouping and verify that recovery closes the incident without concealing a separate failure.
  4. Loss of visibility: interrupt telemetry or its delivery path and verify that missing data is surfaced instead of reported as healthy.
  5. Evidence and access: follow one job across components, confirm restricted access and check that secrets are absent from logs and exports.
  6. Handover: ask the responsible operator to follow the runbook and record any missing permissions, unclear steps or unsupported recovery actions.

Keep a record of the scenario, expected result, observed result, owner and unresolved gaps. An untested dependency stays outside demonstrated coverage. Passing these checks does not guarantee that every future failure will be detected.

Separate implementation costs from ongoing operation

The initial scope includes workflow mapping, instrumentation, dashboards, notification routing, runbooks and a handover exercise. Ongoing costs include telemetry ingestion, storage, retention, queries, tool subscriptions, rule maintenance and response time. More environments, services and data volume increase the work; an existing stack may only need better rules and ownership.

Ask proposals to identify included workflows, excluded dependencies, account ownership and maintenance responsibilities. Use the software proposal comparison checklist to separate delivery from recurring support. Monitoring installation and incident response are different scopes; coverage hours and response commitments must be agreed explicitly.

Scope the first improvement

If incidents still reach you through customer messages, I can review the workflow, current signals and response gaps through my CTO and infrastructure services. The first deliverable can be a prioritised monitoring scope with owners and acceptance checks, followed by implementation where needed.

Tell me about your monitoring problem with the services involved, one sanitised incident, the current tools and who handles alerts. I can use that context to define a practical starting point and clarify what ongoing support would require.