Client & logging

What is this

dlboard.client.dlboard_logger.DLBoardLogger (dlboard/client/dlboard_logger.py) is a pytorch_lightning.loggers.Logger — drop it into a Trainer like any other PL logger. Under the hood it talks to a running dlboard server over REST via BasicDlboardAPI (dlboard/client/_rest_api.py), and spawns two daemon subprocesses that batch metrics and artifacts off multiprocessing queues and flush them on an interval, so training is never blocked on network I/O.

When you’d need this

Every training script that wants runs to show up in dlboard uses this. This page is also where to look if you want to log a new kind of artifact (images are the only built-in kind today) — see below.

How it works

from dlboard.client.dlboard_logger import DLBoardLogger

logger = DLBoardLogger(project_id=project.id, server_url="http://localhost:8050")
trainer = pl.Trainer(logger=logger)

If you don’t already have an experiment, omit experiment_id and one is created for you. Metrics logged via PL’s normal self.log(...) and hyperparameters via save_hyperparameters() are picked up automatically. See examples/lightning_quickstart.py for a complete, runnable example (MNIST MLP), and examples/lightning_advanced.py for a heavier one (nested hyperparameters, checkpoints, mask and box previews, Lightning’s monitoring callbacks) that doubles as a stress test for the UI.

Hyperparameters. save_hyperparameters() (or logger.log_hyperparams(...)) takes anything nested: pydantic models, pydantic and stdlib dataclasses, a Namespace, plain dicts and ordinary init args. They’re flattened to parent/child keys (optim/schedule/warmup_steps), which the experiment page’s hyperparameter columns and run compare group on the same /. Pydantic serializes them, so a model’s aliases and field serializers apply; a list is stored as one JSON string, and scalars are stored exactly as given.

Checkpoints. DLBoardLogger(..., log_model="all") uploads every checkpoint a ModelCheckpoint saves, even one save_top_k deletes moments later (each is copied before it’s queued). log_model=True uploads only the ones still kept, when training finishes. Each is a File (below) of kind checkpoint, keyed checkpoints/<file name>, tagged with its score and Lightning’s latest/best/best_k, and stamped with the global step its callback saved at (with log_model=True, the step of that callback’s last save). Any number of ModelCheckpoint callbacks work. On the experiment page they are one list, split into Latest, Best and Top-k tabs by those tags, paged and sortable by step and score.

Authenticating. If the server signs people in (see Auth), the script has to sign in too:

• Create an API token on your Account page.
• Set it as DLBOARD_API_KEY wherever the script runs, or pass api_key= to BasicDlboardAPI.

Before it starts, the logger checks the key with the server’s /api/v1/whoami and fails right away if it’s missing or wrong. Without that check, bad credentials would only show up later, as metrics that were never stored. dlboard serve local needs no key.

Guard your script’s entry point. The two shipping processes are started with multiprocessing’s spawn method, which re-imports your training script’s __main__ module from scratch in each one. A script that builds the logger (or starts training) at module level, with no if __name__ == "__main__": guard, re-runs that code in every shipping process too — usually visible as training appearing to restart, or the logger itself being constructed more than once. This is the same requirement Python’s own multiprocessing docs describe for any spawn-based code, not something specific to dlboard.

How a run ends. Lightning finalizes the logger when a stage finishes: "success" records the run as finished and "failed" as failed, with the time it ended (stamped by your machine, like the run’s start, so the duration never mixes two clocks). The runs table shows it as “running”, “3m 23s”, “2d 3h” or “✕ 1h 2m” next to the run. A later stage (trainer.test after fit) moves the end time later. A script that exits without Lightning finalizing the logger (an exception outside the trainer, say) is recorded as unknown – ended, but not known how. A process that is killed outright (kill -9, the OOM killer) can’t say anything, so its run stays running. A server that predates this simply doesn’t record any of it.

Multi-GPU (DDP). Every rank builds its own logger and Lightning calls all of them, so only global rank 0 logs: a non-zero rank makes no server calls, starts no shipping processes and drops whatever it’s asked to log, which means one run per job rather than one per GPU. To log into a run that already exists instead of creating one — resuming it, or sharing it between processes you launch separately — pass run_id= or set DLBOARD_RUN_ID; the logger checks it exists up front and uses its experiment. logger.run_id is how you read the id of a run the logger created, to hand on to such a process.

If the server (or your network) is down. Logging never blocks training: log_metrics/ log_artifact queue onto a bounded buffer the shipping processes drain in the background, retrying a failure with exponential backoff rather than hammering a server that’s still down. If the buffer fills faster than it can drain, the oldest-queued item is dropped (not blocked on), with a rate-limited warning while it keeps happening and a final count at finalize(). Size the buffers with DLBoardLoggerSettings (metrics_q_size, artifact_q_size, …) if your logging rate is high enough that this matters; DLBOARD_CONNECT_TIMEOUT_S/DLBOARD_READ_TIMEOUT_S (defaults 10s/60s) cap how long any one request waits before it’s treated as failed.

Histograms. logger.log_histograms({"grad_norm/layer1": tensor}, step=global_step) logs the distribution of any array, tensor or list of numbers at a step – per-layer gradient norms, weights, activations, per-sample losses – instead of averaging it into one scalar. The values are binned in your training process, at the call, in one pass, and only the bucket edges and counts are queued and sent: the server never sees (or computes over) the raw values, so a huge tensor or a fine binning slows down only the script that chose it. A histogram has at most 1024 buckets (DLBOARD_MAX_HISTOGRAM_BUCKETS, set on the server and, for a finer binning, in your training environment too); a binning that needs more raises at the call rather than being rejected later.

How values are bucketed is the binning= argument:

• ExponentialBinning(buckets_per_decade=10, zero_threshold=1e-12) (the default) has a fixed number of buckets per power of ten, mirrored for negative values, with one bucket around zero. It suits values spread over many orders of magnitude. Its edges depend only on those two settings, never on the data, so every step and run lines up bucket for bucket, and a stretch with no values costs one empty bucket.
• ExplicitBinning(edges=[...]) is buckets exactly where you say. Values outside the edges are not dropped: an extra bucket reaching the smallest or largest value seen holds them.

NaN and infinite values are counted aside, not binned. Pass a dlboard.models.Histogram you already binned (its edges and counts) instead of values to send it as it is. Histograms ride the same queue and shipping process as metrics, so the backpressure notes below apply to them too. A server older than the version that added histograms ignores them, so upgrade the server first.

Logging artifacts. Call logger.log_artifact([...]) with anything implementing AnyArtifact (dlboard/models/_artifact.py): a key, tags, a step, and a to_artifact(local_temp, run_id, experiment_id) -> tuple[NewArtifact, Path | AnyUrl] method. Returning a Path (written under local_temp) uploads that file; returning an AnyUrl instead registers it as a link — no bytes move, the server just checks the ref is one its ArtifactStore can actually serve (see Storage) and records it. tags are shown as a caption next to the artifact and take strings, numbers or booleans: numbers are formatted the same way everywhere (floats to four significant digits, 0.1410), so log tags={"iou": iou} rather than formatting it yourself. They are sent and stored as the numbers they are. A number with a unit is a Quantity: tags={"took": Quantity(value=5400, unit=TimeUnit.SECONDS)} says what the number was measured in (time: ns to d; size: B to TB, decimal), and nothing about how it should look. The chart decides that: by default the largest unit that fits (1.500 h, 2.500 GB), or the tag_time_unit / tag_size_unit it is set to (5,400,000 ms), with tag_precision significant digits (default 4); the chart’s tag_display table (in its editor, a row per tag name) overrides those for a particular tag. Plain numbers are written out with separators (12,346) rather than in exponent form. There are four built-in kinds:

• dlboard.client.artifacts.image.Image — uploads. Wraps a torch.Tensor/np.ndarray (validated CHW uint8 via dlbype), used like:

  from dlboard.client.artifacts import image
  
  logger.log_artifact([image.Image(key="sample", image=tensor, step=global_step)])
  

  It is encoded as a lossless PNG by default, so masks, label images and annotated overlays stay pixel-exact. Pass format="jpg" for a much smaller (lossy) file when it is a large photograph.

• dlboard.client.artifacts.figure.Figure — uploads a matplotlib figure as an image. You rarely build one yourself: logger.log_figure mirrors mlflow.log_figure, with the one difference that dlboard needs a step:

  # mlflow: mlflow.log_figure(fig, "loss_curve.png")
  logger.log_figure(fig, "loss_curve.png", step=global_step)  # same name works as-is
  logger.log_figure(fig, "loss_curve", step=global_step)  # or leave the extension off
  

  The format is a lossless PNG by default. If the key ends in another supported extension (.png, .jpg, .jpeg, .svg) that format is used instead, and a format in save_kwargs overrides both (an unsupported one is an error). Any other extension just falls back to PNG. The key itself is used exactly as given. The figure is rendered right at the call, so it’s safe to plt.close(fig) straight after. tags= is optional, and save_kwargs= goes to fig.savefig (dpi, bbox_inches, …). Any object with a matplotlib-style savefig(fname, **kwargs) works, and dlboard doesn’t depend on matplotlib itself.

• dlboard.client.artifacts.file.File — uploads a file that already exists on disk, as-is, tagged with a FileKind saying what it is (FileKind.CHECKPOINT). log_model builds these for you; the file must stay put until the logger has flushed:

  from dlboard.client.artifacts.file import File
  from dlboard.models import FileKind
  
  logger.log_artifact([File(key="weights", path=path, kind=FileKind.CHECKPOINT, step=global_step)])
  

• dlboard.client.artifacts.link.Link — links. For a blob a training job already wrote somewhere dlboard’s ArtifactStore can serve (e.g. the same S3 bucket), without shipping the bytes through dlboard a second time:

  from pydantic import AnyUrl
  from dlboard.client.artifacts import link
  
  logger.log_artifact([link.Link(key="checkpoint", ref=AnyUrl("s3://my-bucket/ckpt.pt"), step=global_step)])
  

Adding a new artifact kind. Note this is a client-side protocol, not a PluginProtocol plugin — there’s no plug(), and it’s never passed into app(). Implement AnyArtifact (the four members above) for whatever you’re logging — a plot, a model checkpoint, a text blob — and pass instances of it to log_artifact the same way. The server stores it as an opaque blob plus whatever metadata to_artifact put in the NewArtifact; if you also want a dedicated way to render that kind of artifact, pair it with a chart plugin — see image_series.py for how the built-in image chart type reads back what Image.to_artifact wrote.